使用 SQL/LINQ 删除基于多个属性的重复实体

Question

我有两个实体，Etf 和 DataPoint。每个Etf 有多个DataPoints。请参阅下面的结构

public class Etf
{
    public Etf() 
    { 
        DataPoints = new HashSet<DataPoint>(); 
    }

    public string Id { get; set; }
    public ICollection<DataPoint> DataPoints { get; private set; }
}

public class DataPoint { 
    public string EtfId { get; set; }
    public Etf Etf { get; set; }
    public DateTime Date { get; set; }

}

碰巧将具有相同EtfId 和Date 的多个数据点插入到数据库中。我想根据这两个字段删除重复项。

在 SQL 中，我试过这个：

WITH CTE AS 
    (SELECT Id, ROW_NUMBER() OVER 
       (PARTITION BY EtfId, Date ORDER BY EtfId, Date DESC) 
       AS ROW_NUMBER FROM DataPoints) 
DELETE FROM CTE WHERE ROW_NUMBER > 1;

这给了我一个错误，即 CTE 不是可更新的表（可能是因为它同时连接到 Etf 和 DataPoint）。然后，我根据this 的答案在 LINQ 中尝试了以下内容：

// Get the duplicate indices
var idxs = await _context.DataPoints
                    .GroupBy(g => new {g.EtfId, g.Date})
                    .OrderBy(g => g.Key.EtfId)
                    .ThenByDescending(g => g.Key.Date)
                    .SelectMany(g =>
                        g.Select((i, idx) => new { i.Id, Idx = idx }))
                    .Where(g => g.Idx > 0)
                    .Select(g => g.Id)
                    .ToListAsync(cancellationToken);

// Get the duplicate entities from indices
var duplicates = await _context.DataPoints
                    .Where(x => idxs.Contains(x.Id))
                    .ToListAsync(cancellationToken);

// Remove them
_context.DataPoints.RemoveRange(duplicates);

但是，这种方法在Select((i, idx) => ..) 语句中给了我一个System.InvalidOperationException，说它可能是来自 EF Core 的错误或限制。

我可以使用更好的（或只是有效的）方法吗？

Answer 1

我设法用 SQL 找到了一个可怕的解决方案：

await _context.Database
                .ExecuteSqlRawAsync("DELETE FROM DataPoints WHERE Id IN " +
                                    "(SELECT Id FROM " +
                                    "(WITH c AS (SELECT Id, ROW_NUMBER() " +
                                    "OVER (PARTITION BY EtfId, Date ORDER BY EtfId, Date DESC) AS rn FROM DataPoints)" +
                                    "SELECT * FROM c WHERE rn > 1) AS t);");

请不要将这篇文章告诉任何人。

Answer 2

... 这可能是 c#/linq 中更糟糕的解决方案。它将删除除重复数据点之一之外的所有数据点，即 Id = 2 和 5 的数据点。

void Main()
{
    var datapoints = new List<Datapoint>();
    datapoints.Add(new Datapoint() { Id = 1, EtfId = "A", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Add(new Datapoint() { Id = 2, EtfId = "A", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Add(new Datapoint() { Id = 3, EtfId = "A", EtfDate = new DateTime(2020, 6, 2) });
    datapoints.Add(new Datapoint() { Id = 4, EtfId = "B", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Add(new Datapoint() { Id = 5, EtfId = "B", EtfDate = new DateTime(2020, 6, 1) });
    var duplicates = datapoints
        .GroupBy(g => new { g.EtfId, g.EtfDate })
        .Select(g => new { g.Key.EtfId, g.Key.EtfDate, Count = g.Count() })
        .Where(g => g.Count > 1);
    var datapointsToremove = new List<int>();
    foreach (var d in duplicates)
    {
        var removeDataPoints = datapoints
            .Where(x => x.EtfId == d.EtfId && x.EtfDate == d.EtfDate).Skip(1)
            .Select(x => x.Id);
        foreach (var rd in removeDataPoints)
        {
            datapointsToremove.Add(rd);
        };
    };
    foreach (var dtr in datapointsToremove)
    {
        datapoints.Remove(datapoints.FirstOrDefault(d => d.Id == dtr));
    }
}
public class Datapoint
{
    public int Id { get; set; }
    public string EtfId { get; set; }
    public DateTime EtfDate { get; set; }
}

然后是这个灵感来自 var distinctItems = items.GroupBy(x => x.Id).Select(y => y.First());见Remove duplicates in the list using linq

void Main()
{
    var datapoints = new List<Datapoint>();
    datapoints.Add(new Datapoint() { Id = 1, EtfId = "A", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Add(new Datapoint() { Id = 2, EtfId = "A", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Add(new Datapoint() { Id = 3, EtfId = "A", EtfDate = new DateTime(2020, 6, 2) });
    datapoints.Add(new Datapoint() { Id = 4, EtfId = "B", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Add(new Datapoint() { Id = 5, EtfId = "B", EtfDate = new DateTime(2020, 6, 1) });
    datapoints.Dump();
    var distinctItems = datapoints.GroupBy(x => new { x.EtfId, x.EtfDate}).Select(y => y.First());
}
class Datapoint
{
    public int Id  {get;set;}
    public string EtfId {get;set;}
    public DateTime EtfDate  {get;set;} 
}

...这是一个更优雅的解决方案，但无法消除数据库中的重复项。