一次读取文件的 x 行 C#

Question

我想一次读取和处理 10+ 行 GB 文件，但是直到最后都没有找到解决方案来吐出 10 行。

我最后一次尝试是：

        int n = 10;
        foreach (var line in File.ReadLines("path")
            .AsParallel().WithDegreeOfParallelism(n))
        {
            System.Console.WriteLine(line);
            Thread.Sleep(1000);
        }

我见过使用缓冲区大小的解决方案，但我想读取整行。

Answer 1

此方法会从您的文件中创建“页面”行。

public static IEnumerable<string[]> ReadFileAsLinesSets(string fileName, int setLen = 10)
{
    using (var reader = new StreamReader(fileName))
        while (!reader.EndOfStream)
        {
            var set = new List<string>();
            for (var i = 0; i < setLen && !reader.EndOfStream; i++)
            {
                set.Add(reader.ReadLine());
            }
            yield return set.ToArray();
        }
}

...更有趣的版本...

class Example
{
    static void Main(string[] args)
    {
        "YourFile.txt".ReadAsLines()
                      .AsPaged(10)
                      .Select(a=>a.ToArray()) //required or else you will get random data since "WrappedEnumerator" is not thread safe
                      .AsParallel()
                      .WithDegreeOfParallelism(10)
                      .ForAll(a =>
        {
            //Do your work here.
            Console.WriteLine(a.Aggregate(new StringBuilder(), 
                                          (sb, v) => sb.AppendFormat("{0:000000} ", v), 
                                          sb => sb.ToString()));
        });
    }
}

public static class ToolsEx
{

    public static IEnumerable<IEnumerable<T>> AsPaged<T>(this IEnumerable<T> items,
                                                              int pageLength = 10)
    {
        using (var enumerator = new WrappedEnumerator<T>(items.GetEnumerator()))
            while (!enumerator.IsDone)
                yield return enumerator.GetNextPage(pageLength);
    }

    public static IEnumerable<T> GetNextPage<T>(this IEnumerator<T> enumerator,
                                                     int pageLength = 10)
    {
        for (var i = 0; i < pageLength && enumerator.MoveNext(); i++)
            yield return enumerator.Current;
    }

    public static IEnumerable<string> ReadAsLines(this string fileName)
    {
        using (var reader = new StreamReader(fileName))
            while (!reader.EndOfStream)
                yield return reader.ReadLine();
    }
}

internal class WrappedEnumerator<T> : IEnumerator<T>
{
    public WrappedEnumerator(IEnumerator<T> enumerator)
    {
        this.InnerEnumerator = enumerator;
        this.IsDone = false;
    }

    public IEnumerator<T> InnerEnumerator { get; private set; }
    public bool IsDone { get; private set; }

    public T Current { get { return this.InnerEnumerator.Current; } }
    object System.Collections.IEnumerator.Current { get { return this.Current; } }

    public void Dispose()
    {
        this.InnerEnumerator.Dispose();
        this.IsDone = true;
    }

    public bool MoveNext()
    {
        var next = this.InnerEnumerator.MoveNext();
        this.IsDone = !next;
        return next;
    }

    public void Reset()
    {
        this.IsDone = false;
        this.InnerEnumerator.Reset();
    }
}

Answer 2

默认行为是一口气读完所有行，如果你想读的少一点，你需要更深入地研究它是如何读的，然后得到一个StreamReader，然后你就可以控制阅读了进程

        using (StreamReader sr = new StreamReader(path)) 
        {
            while (sr.Peek() >= 0) 
            {
                Console.WriteLine(sr.ReadLine());
            }
        }

它还有一个ReadLineAsync 方法会返回一个任务

如果您将这些任务包含在 ConcurrentBag 中，您可以非常轻松地保持处理一次运行 10 行。

var bag =new ConCurrentBag<Task>();
using (StreamReader sr = new StreamReader(path))
{
    while(sr.Peek() >=0)
    {
        if(bag.Count < 10)
        {
            Task processing = sr.ReadLineAsync().ContinueWith( (read) => {
                string s = read.Result;//EDIT Removed await to reflect Scots comment
                //process line
            });
            bag.Add(processing);
        }
        else
        {
            Task.WaitAny(bag.ToArray())
            //remove competed tasks from bag
        }
    }
}

请注意，此代码仅供参考，请勿按原样使用；

如果你想要的只是最后十行，那么你可以通过这里的解决方案得到它 How to read a text file reversely with iterator in C#