将 1000000 个文档插入 RavenDB答案

【问题标题】：Insert 1000000 documents into RavenDB将 1000000 个文档插入 RavenDB
【发布时间】：2012-10-13 13:40:06
【问题描述】：

我想在 RavenDB 中插入 1000000 个文档。

class Program
{
        private static string serverName;
        private static string databaseName;

        private static DocumentStore documentstore;
        private static IDocumentSession _session;

        static void Main(string[] args)
        {

            Console.WriteLine("Start...");

            serverName = ConfigurationManager.AppSettings["ServerName"];
            databaseName = ConfigurationManager.AppSettings["Database"];

            documentstore = new DocumentStore { Url = serverName };
            documentstore.Initialize();

            Console.WriteLine("Initial Databse...");

            _session = documentstore.OpenSession(databaseName);

            for (int i = 0; i < 1000000; i++)
            {
                var person = new Person()    
                {
                    Fname = "Meysam" + i,
                    Lname = " Savameri" + i,
                    Bdate = DateTime.Now,
                    Salary = 6001 + i,
                    Address = "BITS provides one foreground and three background priority levels that" +
                              "you can use to prioritize transBfer jobs. Higher priority jobs preempt"+
                              "lower priority jobs. Jobs at the same priority level share transfer time,"+
                              "which prevents a large job from blocking small jobs in the transfer"+
                              "queue. Lower priority jobs do not receive transfer time until all the "+
                              "higher priority jobs are complete or in an error state. Background"+
                              "transfers are optimal because BITS uses idle network bandwidth to"+
                              "transfer the files. BITS increases or decreases the rate at which files "+
                              "are transferred based on the amount of idle network bandwidth that is"+
                              "available. If a network application begins to consume more bandwidth,"+
                              "BITS decreases its transfer rate to preserve the user's interactive"+
                              "experience. BITS supports multiple foreground jobs and one background"+
                              "transfer job at the same time.",
                    Email = "Meysam" + i + "@hotmail.com",
                };

                _session.Store(person);

                Console.ForegroundColor = ConsoleColor.Green;
                Console.WriteLine("Count:" + i);
                Console.ForegroundColor = ConsoleColor.White;
            }

            Console.WriteLine("Commit...");

            _session.SaveChanges();
            documentstore.Dispose();

            _session.Dispose();

            Console.WriteLine("Complete...");
            Console.ReadLine();
        }
    }

但会话不保存更改，我收到错误：

在 mscorlib.dll 中发生“System.OutOfMemoryException”类型的未处理异常

【问题讨论】：

您不了解OutOfMemory的哪一部分？！？！？您需要分批加载文档（例如，一次 1'000 个）

标签： ravendb

【解决方案1】：

document session 旨在处理少量请求。相反，尝试分批插入 1024。之后，处理会话并创建一个新会话。你得到一个OutOfMemoryException 的原因是因为文档会话缓存了所有组成对象以提供一个unit of work，这就是为什么你应该在插入一个批处理后处理会话。

一个巧妙的方法是使用Batch linq extension：

foreach (var batch in Enumerable.Range(1, 1000000)
 .Select(i => new Person { /* set properties */ })
 .Batch(1024))
{
 using (var session = documentstore.OpenSession())
 {
   foreach (var person in batch)
   {
     session.Store(person);
   }
   session.SaveChanges();
 }
}

Enumerable.Range 和 Batch 的实现都是惰性的，不会将所有对象都保存在内存中。

【讨论】：

感谢您指出来自 morelinq 的 .Batch() 扩展。这是一个非常有用的技术！

【解决方案2】：

RavenDB 也有一个 bulk API 可以做类似的事情而不需要额外的 LINQ 扩展：

using (var bulkInsert = store.BulkInsert())
{
    for (int i = 0; i < 1000 * 1000; i++)
    {
        bulkInsert.Store(new User
            {
                Name = "Users #" + i
            });
    }
}

注意.SaveChanges() 不会被调用，并且会在达到批量大小时调用（如果需要，在BulkInsert() 中定义），或者在处理bulkInsert 时调用。

【讨论】：