LINQ 搜索中的 mscorlib.dll 中出现了“System.OutOfMemoryException”类型的未处理异常

Question

使用来自 MSDN 的 this 文章，我正在尝试搜索目录中的文件。问题是，每次我执行程序时，我都会得到：

“在 mscorlib.dll 中发生了 'System.OutOfMemoryException' 类型的未处理异常”。

我尝试了其他一些选项，例如StreamReader，但我无法让它工作。这些文件很大。其中一些的范围高达 1.5-2GB，每天可能有 5 个或更多文件。

此代码失败：

private static string GetFileText(string name)
{
    var fileContents = string.Empty;
    // If the file has been deleted since we took  
    // the snapshot, ignore it and return the empty string. 
    if (File.Exists(name))
    {
        fileContents = File.ReadAllText(name);
    }
    return fileContents;
}

有什么想法会发生什么或如何让它在没有内存错误的情况下读取？

完整代码（以防你不想打开 MSDN 文章）

class QueryContents {
public static void Main()
{
    // Modify this path as necessary. 
    string startFolder = @"c:\program files\Microsoft Visual Studio 9.0\";

    // Take a snapshot of the file system.
    System.IO.DirectoryInfo dir = new System.IO.DirectoryInfo(startFolder);

    // This method assumes that the application has discovery permissions 
    // for all folders under the specified path.
    IEnumerable<System.IO.FileInfo> fileList = dir.GetFiles("*.*", System.IO.SearchOption.AllDirectories);

    string searchTerm = @"Visual Studio";

    // Search the contents of each file. 
    // A regular expression created with the RegEx class 
    // could be used instead of the Contains method. 
    // queryMatchingFiles is an IEnumerable<string>. 
    var queryMatchingFiles =
        from file in fileList
        where file.Extension == ".htm" 
        let fileText = GetFileText(file.FullName)
        where fileText.Contains(searchTerm)
        select file.FullName;

    // Execute the query.
    Console.WriteLine("The term \"{0}\" was found in:", searchTerm);
    foreach (string filename in queryMatchingFiles)
    {
        Console.WriteLine(filename);
    }

    // Keep the console window open in debug mode.
    Console.WriteLine("Press any key to exit");
    Console.ReadKey();
}

// Read the contents of the file. 
static string GetFileText(string name)
{
    string fileContents = String.Empty;

    // If the file has been deleted since we took  
    // the snapshot, ignore it and return the empty string. 
    if (System.IO.File.Exists(name))
    {
        fileContents = System.IO.File.ReadAllText(name);
    }
    return fileContents;
}

}

Answer 1

您遇到的问题是基于尝试同时加载多个 GB 的文本。如果它们是文本文件，您可以流式传输它们并一次只比较一行。

var queryMatchingFiles =
    from file in fileList
    where file.Extension == ".htm" 
    let fileLines = File.ReadLines(file.FullName) // lazy IEnumerable<string>
    where fileLines.Any(line => line.Contains(searchTerm))
    select file.FullName;

Answer 2

我建议您遇到内存不足错误，因为查询的编写方式我相信您需要将每个文件的整个文本加载到内存中，并且在整个文件之前不能释放任何对象套装已加载。您不能在 GetFileText 函数中检查搜索词，然后只返回真或假吗？

如果你这样做了，文件文本至少在函数结束时超出范围，GC 可以恢复内存。如果您正在处理大文件/数量，实际上最好将其重写为流式功能，然后如果您遇到搜索词并且您不需要整个文件一直在内存中，您可以提前退出阅读。

Previous question on finding a term in an HTML file using a stream