使用 XmlWriter 从 LINQ 创建大型文档到 SQL / LINQPad 抛出内存不足异常

Question

我正在尝试在 LINQPad 脚本中导出数据并不断收到内存不足异常。我觉得脚本正在执行所有“可流式”操作，所以不确定我为什么会得到这个。

代码的主循环如下所示。几点说明：

1) 第一个查询返回大约 60K 行 profileDB.Profiles.Where(p => p.Group.gName == groupName).Select( d => d.pAuthID )

2) 每个pAuthID 的第二个查询返回数据库中的行，其中一个字段是存储在字符串字段中的 Xml 数据块。它不是那么大......肯定是pAuthID 行可以有多达50 行的FolderItems。查询是profileDB.FolderItems.Where(f => f.Profile.pAuthID == p && ( folderTypes[0] == "*" || folderTypes.Contains(f.fiEntryType) ) ).OrderBy(f => f.fiEntryDate)

3) 处理开始时，我只在结果窗格中写入一行。

4) 脚本运行时间长，输出文件在600-700MB左右时抛出异常。我知道，但要求我们将所有数据转储到 Xml 中。

5) WriteFolderItems 函数/循环将粘贴在主循环下方。

6) 我在每个 xDataDef 元素之后调用XmlWriter.Flush。

using (var xw = XmlWriter.Create(fileName, new XmlWriterSettings { Indent = false } ) )
{
    xw.WriteStartElement( "xDataDefs" );

    foreach( var p in profileDB.Profiles.Where(p => p.Group.gName == groupName).Select( d => d.pAuthID ) )
    {
        if ( totalRows == 0 ) // first one...
        {
            string.Format( "Writing results to {0}...", fileName ).Dump( "Progress" );
        }

        totalRows++;

        var folderItems = profileDB.FolderItems.Where(f => f.Profile.pAuthID == p && ( folderTypes[0] == "*" || folderTypes.Contains(f.fiEntryType) ) ).OrderBy(f => f.fiEntryDate);

        if ( folderItems.Any() )
        {
            xw.WriteStartElement("xDataDef");
                xw.WriteAttributeString("id-auth", p);

                xw.WriteStartElement("FolderItems");
                    WriteFolderItems(profileDB, datalockerConnectionString, xw, folderItems, documentsDirectory, calcDocumentFolder, exportFileData);
                xw.WriteEndElement();

                xw.WriteEndElement();

            xw.Flush();
        }
    }

    xw.WriteEndElement();
}

WriteFolderItems 也有循环代码，如下所示。几点说明：

1) 我希望 foreach( var f in folderItems ) 能够流式传输

2) 对于某些作为缓存文档的 Xml blob 的 FolderItem 行，我需要对数据库运行 ~ 1-5 次查询以获取一些附加信息以粘贴到 Xml 导出中：var docInfo = profileDB.Documents.Where( d => d.docfiKey == f.fiKey && d.docFilename == fileName ).FirstOrDefault();

3) 我在每个 FolderItem 行之后调用 XmlWriter.Flush。

public void WriteFolderItems( BTR.Evolution.Data.DataContexts.Legacy.xDS.DataContext profileDB, string datalockerConnectionString, XmlWriter xw, IEnumerable<BTR.Evolution.Data.DataContexts.Legacy.xDS.FolderItem> folderItems, string documentsOutputDirectory, string calcDocumentFolder, bool exportFileData )
{
    foreach( var f in folderItems )
    {
        // The Xml blob string
        var calculation = XElement.Parse( f.fiItem );

        // If it contains 'cached-document' elements, need to download the actual document from DataLocker database
        foreach( var document in calculation.Elements( "Data" ).Elements( "TabDef" ).Elements( "cache-documents" ).Elements( "cached-document" ) )
        {
            var fileName = (string)document.Attribute( "name" );

            // Get author/token to be used during import
            var docInfo = profileDB.Documents.Where( d => d.docfiKey == f.fiKey && d.docFilename == fileName ).FirstOrDefault();

            if ( docInfo != null )
            {
                document.Add( new XElement( "author", docInfo.docUploadAuthID ) );
                document.Add( new XElement( "token", docInfo.docDataLockerToken ) );
            }

            // Export associated document from DataLocker connection...XmlWriter is not affected, simply saves document to local hard drive
            if ( exportFileData && DataLockerExtensions.ByConnection( datalockerConnectionString ).Exists( calcDocumentFolder, (string)document.Attribute( "name" ), null ) )
            {
                using ( var fs = new FileStream( Path.Combine( documentsOutputDirectory, fileName.Replace( "/", "__" ) ), FileMode.Create ) )
                {
                    string contentType;
                    using ( var ds = DataLockerExtensions.ByConnection( datalockerConnectionString ).Get( calcDocumentFolder, (string)document.Attribute( "name" ), null, out contentType ) )
                    {
                        ds.CopyTo( fs );
                    }
                }
            }
        }

        // Write the calculation to the XwlWriter
        xw.WriteStartElement( "FolderItem" );
            xw.WriteElementString( "Key", f.fiKey.ToString() );
            xw.WriteElementString( "EntryDate", XmlConvert.ToString( f.fiEntryDate.Value, XmlDateTimeSerializationMode.Local ) );
            xw.WriteElementString( "ItemType", f.fiEntryType );
            xw.WriteElementString( "Author", f.fiAuthor );
            xw.WriteElementString( "Comment", f.fiComment );
            xw.WriteStartElement( "Item" );
                calculation.WriteTo( xw );
            xw.WriteEndElement();
        xw.WriteEndElement();
        xw.Flush();
    }
}

Answer 1

确保禁用更改跟踪，否则 EF 或 L2S 更改跟踪器将保留对每个已加载实体的引用。