JSON 中的 23,908 个对象我正在尝试将值添加到属性，然后再将它们添加到 SQL 数据库

Question

我正在创建一个接收字符串化 JSON 的 WEB API，并且我正在进行 JSON 转换，在将其添加到 SQL 数据库之前，我使用 for 循环提供了两个属性值，但它需要 5 分钟以上它终于到达了 db.SaveChanges()。 JSON 有大约 23,908 个对象。我想问社区是否有更好的实现，然后我正在做什么来加快每个对象的添加？

//This handles the stringify conversion and the adding values to the properties before adding it to SQL database 
public void SaveCSV(string file, string fileName)
{
  var csv = JsonConvert.DeserializeObject<List<SecurityFile>>(file);
  using (ApplicationDbContext db = ApplicationDbContext.Create())
  {
   //For loop that adds the values to each object in JSON
   for (var i = 0; i < csv.Count(); i++)
   {
      csv[i].DateSubmitted = DateTime.Now;
      csv[i].FileName = fileName;
      db.SecurityFiles.Add(csv[i]);
   }
     //Saves it to SQL Database
     db.SaveChanges();
   }

//Here is my Class
public class SecurityFile
{
  [Key]
  public int ID { get; set; }
  [JsonProperty("Plugin ID")]
  public string PluginId { get; set; }
  [JsonProperty("CVE")]
  public string CVE { get; set; }
  [JsonProperty("Risk")]
  public string Risk { get; set; }
  [JsonProperty("Host")]
  public string Host { get; set; }
  [JsonProperty("Protocol")]
  public string Protocol { get; set; }
  [JsonProperty("Port")]
  public string Port { get; set; }
  [JsonProperty("Name")]
  public string Name { get; set; }
  [JsonProperty("Synopsis")]
  public string Synopsis { get; set; }
  [JsonProperty("Description")]
  public string Description { get; set; }
  [JsonProperty("Solution")]
  public string Solution { get; set; }
  [JsonProperty("See Also")]
  public string SeeAlso { get; set; }
  [JsonProperty("FileName")]
  public string FileName{ get; set; }
  [JsonProperty("DateSubmitted")]
  public  DateTime DateSubmitted { get; set; }
}

Answer 1

有几个优化选项。 我认为如果您使用并行化流程，您将受益 C# Linq TPL 然后一次性添加所有实体。

using System.Threading.Tasks;

public void SaveCSV(string file, string fileName)
{
    var csv = JsonConvert.DeserializeObject<List<SecurityFile>>(file);
    using (ApplicationDbContext db = ApplicationDbContext.Create())
    {
        var now = DateTime.Now;

        Parallel.Foreach(csv, item => {
            item.DateSubmitted = now;
            item.FileName = fileName;
        })

        //Attach the Entities all at once
        db.SecurityFiles.AddRange(csv);

        //Saves it to SQL Database
        db.SaveChanges();
    }
}

Answer 2

20'000 个对象不是很多。
您应该能够在 5 秒 700 毫秒内完成。

你的路很慢，因为你这样做了

foreach(jsonObject)
   connection.open()
   cmd.execute() // Individual insert-statement
   connection.close()

打开和关闭连接的成本很高，单独执行每个命令也是如此。

你应该这样做

int batchSize = 100;
System.Text.StringBuilder sb = new System.Text.StringBuilder();
connection.Open();
transaction.Begin();
for(int i = 0; i < Count;++i)
{ 
     sb.Append("SQL-Insert");
     if(i%batchSize == 0 && i != 0)
     {
          execute(sb.ToString())
          sb.Length = 0;
     }
}
execute(sb.ToString())
transaction.commit();
// TODO: Try/Catch + Rollback
connection.Close();

更简单的方法是使用SELECT * FROM table_name WHERE (1=2) 获取要插入对象的表 然后将条目添加到数据表中，然后调用 DataAdaper.Update()
如果您的表有主键，您甚至可以使用 SqlCommandBuilder 自动生成插入命令。

C#

public static void InsertUpdateDataTable(string strTableName, System.Data.DataTable dt)
{
    string strSQL = string.Format("SELECT * FROM [{0}] WHERE 1 = 2 ", strTableName.Replace("]", "]]"));

    using (System.Data.SqlClient.SqlDataAdapter daInsertUpdate = new System.Data.SqlClient.SqlDataAdapter(strSQL, getConnectionString())) {
        SqlCommandBuilder cmdBuilder = new SqlCommandBuilder(daInsertUpdate);
        daInsertUpdate.InsertCommand = cmdBuilder.GetInsertCommand();
        daInsertUpdate.UpdateCommand = cmdBuilder.GetUpdateCommand();

        daInsertUpdate.Update(dt);
    }

}

VB.NET：

Public Shared Sub InsertUpdateDataTable(strTableName As String, dt As System.Data.DataTable)
    Dim strSQL As String = String.Format("SELECT * FROM [{0}] WHERE 1 = 2 ", strTableName.Replace("]", "]]"))

    Using daInsertUpdate As New System.Data.SqlClient.SqlDataAdapter(strSQL, getConnectionString())
        Dim cmdBuilder As New SqlCommandBuilder(daInsertUpdate)
        daInsertUpdate.InsertCommand = cmdBuilder.GetInsertCommand()
        daInsertUpdate.UpdateCommand = cmdBuilder.GetUpdateCommand()

        daInsertUpdate.Update(dt)
    End Using

End Sub

您可以在 DataAdaper 上设置批量大小，它也会为您生成 SQL。

并且以与数据库无关的方式：

private static System.Data.Common.DbProviderFactory m_factory = System.Data.Common.DbProviderFactories.GetFactory(typeof(System.Data.SqlClient.SqlClientFactory).Namespace);


public static void InsertUpdateDataTable(string strTableName, System.Data.DataTable dt)
{
    if (dt == null)
        throw new System.ArgumentNullException("DataTable dt may not be NULL.");

    // https://msdn.microsoft.com/en-us/library/aadf8fk2(v=vs.110).aspx
    using (System.Data.Common.DbDataAdapter daInsertUpdate = m_factory.CreateDataAdapter())
    {

        using (System.Data.Common.DbConnection conn = m_factory.CreateConnection())
        {
            conn.ConnectionString = getConnectionString();

            daInsertUpdate.SelectCommand = conn.CreateCommand();
            daInsertUpdate.SelectCommand.CommandText = string.Format("SELECT * FROM [{0}] WHERE 1 = 2 ", strTableName.Replace("]", "]]"));

            using (System.Data.Common.DbCommandBuilder cmdBuilder = m_factory.CreateCommandBuilder())
            {
                cmdBuilder.DataAdapter = daInsertUpdate;
                daInsertUpdate.InsertCommand = cmdBuilder.GetInsertCommand();
                daInsertUpdate.UpdateCommand = cmdBuilder.GetUpdateCommand();
            } // End Using cmdBuilder

            daInsertUpdate.Update(dt);
        } // End Using conn
    } // End Using daInsertUpdate 

    System.Console.WriteLine(dt);
}

并使用事务（假设 ls 是数字列表）：

private static System.Data.Common.DbProviderFactory m_factory = System.Data.Common.DbProviderFactories.GetFactory(typeof(System.Data.SqlClient.SqlClientFactory).Namespace);


public static string getConnectionString()
{
    System.Data.SqlClient.SqlConnectionStringBuilder csb = new System.Data.SqlClient.SqlConnectionStringBuilder();

    csb.DataSource = System.Environment.MachineName;
    csb.InitialCatalog = "TestDb";
    csb.IntegratedSecurity = true;

    return csb.ConnectionString;
}


public static System.Data.Common.DbConnection GetConnection()
{
    var con = m_factory.CreateConnection();
    con.ConnectionString = getConnectionString();

    return con;
}


public static int BatchedInsert(System.Collections.IList ls)
{
    int iAffected = 0;
    int batchSize = 100; // Each batch corresponds to a single round-trip to the DB.

    using (System.Data.IDbConnection idbConn = GetConnection())
    {

        lock (idbConn)
        {

            using (System.Data.IDbCommand cmd = idbConn.CreateCommand())
            {

                lock (cmd)
                {
                    if (cmd.Connection.State != System.Data.ConnectionState.Open)
                        cmd.Connection.Open();

                    using (System.Data.IDbTransaction idbtTrans = idbConn.BeginTransaction())
                    {

                        try
                        {
                            cmd.Transaction = idbtTrans;


                            System.Text.StringBuilder sb = new System.Text.StringBuilder();

                            for (int i = 0; i < ls.Count; ++i)
                            {
                                sb.Append("INSERT INTO T_TransactionInsertTest(TestValue) VALUES ( ");
                                sb.Append(ls[i].ToString());
                                sb.AppendLine(");");

                                if (i % batchSize == 0 && i != 0)
                                {
                                    cmd.CommandText = sb.ToString();
                                    iAffected += cmd.ExecuteNonQuery();
                                    sb.Length = 0;
                                }
                            }

                            if (sb.Length != 0)
                            {
                                cmd.CommandText = sb.ToString();
                                iAffected += cmd.ExecuteNonQuery();
                            }

                            idbtTrans.Commit();
                        } // End Try
                        catch (System.Data.Common.DbException ex)
                        {
                            if (idbtTrans != null)
                                idbtTrans.Rollback();

                            iAffected = -1;

                            //if (Log(ex))
                            throw;
                        } // End catch
                        finally
                        {
                            if (cmd.Connection.State != System.Data.ConnectionState.Closed)
                                cmd.Connection.Close();
                        } // End Finally

                    } // End Using idbtTrans

                } // End lock cmd

            } // End Using cmd 

        } // End lock idbConn

    } // End Using idbConn

    return iAffected;
} // End Function BatchedInsert 

Answer 3

我建议使用一个 SQL 查询插入所有行，而不是在循环中插入每一行。

您可以插入所有项目tempTable，然后创建一个插入连接语句并在循环外执行它。

看看这个问题bulk-record-update-with-sql

或者您可以执行批量插入语句：

using (ApplicationDbContext db = ApplicationDbContext.Create())
{
   db.Database.ExecuteSqlCommand(@" BULK
    INSERT SecurityFiles
    FROM 'your file path'
    WITH
    (
    FIRSTROW
    = 1,
    FIELDTERMINATOR = ';',
    ROWTERMINATOR = '\n',
    )");
}