逐行读取文本文件，并提供准确的偏移/位置报告答案

【问题标题】：Reading text files line by line, with exact offset/position reporting逐行读取文本文件，并提供准确的偏移/位置报告
【发布时间】：2010-04-07 16:25:21
【问题描述】：

我的简单要求：读取一个巨大的（>一百万）行测试文件（对于此示例，假设它是某种 CSV）并保留对该行开头的引用以便将来更快地查找（读取一行，从 X 开始）。

我首先尝试了简单的方法，使用StreamWriter 并访问底层BaseStream.Position。不幸的是，这并没有按我的预期工作：

给定一个包含以下内容的文件

Foo
Bar
Baz
Bla
Fasel

还有这个非常简单的代码

using (var sr = new StreamReader(@"C:\Temp\LineTest.txt")) {
  string line;
  long pos = sr.BaseStream.Position;
  while ((line = sr.ReadLine()) != null) {
    Console.Write("{0:d3} ", pos);
    Console.WriteLine(line);
    pos = sr.BaseStream.Position;
  }
}

输出是：

000 Foo
025 Bar
025 Baz
025 Bla
025 Fasel

我可以想象流正在尝试提供帮助/高效，并且可能在需要新数据时读取（大）块。对我来说这很糟糕..

最后的问题是：有什么方法可以在逐行读取文件时获取（字节，字符）偏移量，而不使用基本的流并手动弄乱 \r \n \r\n 和字符串编码等？没什么大不了的，真的，我只是不喜欢构建可能已经存在的东西..

【问题讨论】：

如果您反映 System.IO.Stream 类，则允许的最小缓冲区为 128 字节......不确定这是否有帮助，但是当我尝试这个时，在更长的文件上，那就是我能得到的最短位置。

标签： c# text-files offset

【解决方案1】：

您可以创建一个TextReader 包装器，它会跟踪基础TextReader 中的当前位置：

public class TrackingTextReader : TextReader
{
    private TextReader _baseReader;
    private int _position;

    public TrackingTextReader(TextReader baseReader)
    {
        _baseReader = baseReader;
    }

    public override int Read()
    {
        _position++;
        return _baseReader.Read();
    }

    public override int Peek()
    {
        return _baseReader.Peek();
    }

    public int Position
    {
        get { return _position; }
    }
}

然后您可以按如下方式使用它：

string text = @"Foo
Bar
Baz
Bla
Fasel";

using (var reader = new StringReader(text))
using (var trackingReader = new TrackingTextReader(reader))
{
    string line;
    while ((line = trackingReader.ReadLine()) != null)
    {
        Console.WriteLine("{0:d3} {1}", trackingReader.Position, line);
    }
}

【讨论】：

似乎有效。不知何故，现在看起来很明显.. 非常感谢。
只要你想要字符位置而不是字节位置，这个解决方案就可以了。如果底层文件有一个字节顺序标记 (BOM) 它将偏移，或者如果它使用多字节字符，则字符和字节之间的 1:1 对应关系不再成立。
同意，仅适用于单字节编码字符，例如ASCII。例如，如果您的基础文件是 Unicode，则每个字符将采用 2 或 4 字节编码。上面的实现是在一个字符流上工作，而不是一个字节流，所以你会得到不会映射到实际字节位置的字符偏移量，因为每个字符可以是 2 或 4 个字节。例如，第二个字符位置将报告为索引 1，但字节位置实际上是索引 2 或 4。如果有 BOM（字节顺序标记），这将再次向真正的底层字节位置添加额外的字节。

【解决方案2】：

在搜索、测试和做一些疯狂的事情之后，有我的代码要解决（我目前在我的产品中使用这个代码）。

public sealed class TextFileReader : IDisposable
{

    FileStream _fileStream = null;
    BinaryReader _binReader = null;
    StreamReader _streamReader = null;
    List<string> _lines = null;
    long _length = -1;

    /// <summary>
    /// Initializes a new instance of the <see cref="TextFileReader"/> class with default encoding (UTF8).
    /// </summary>
    /// <param name="filePath">The path to text file.</param>
    public TextFileReader(string filePath) : this(filePath, Encoding.UTF8) { }

    /// <summary>
    /// Initializes a new instance of the <see cref="TextFileReader"/> class.
    /// </summary>
    /// <param name="filePath">The path to text file.</param>
    /// <param name="encoding">The encoding of text file.</param>
    public TextFileReader(string filePath, Encoding encoding)
    {
        if (!File.Exists(filePath))
            throw new FileNotFoundException("File (" + filePath + ") is not found.");

        _fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read);
        _length = _fileStream.Length;
        _binReader = new BinaryReader(_fileStream, encoding);
    }

    /// <summary>
    /// Reads a line of characters from the current stream at the current position and returns the data as a string.
    /// </summary>
    /// <returns>The next line from the input stream, or null if the end of the input stream is reached</returns>
    public string ReadLine()
    {
        if (_binReader.PeekChar() == -1)
            return null;

        string line = "";
        int nextChar = _binReader.Read();
        while (nextChar != -1)
        {
            char current = (char)nextChar;
            if (current.Equals('\n'))
                break;
            else if (current.Equals('\r'))
            {
                int pickChar = _binReader.PeekChar();
                if (pickChar != -1 && ((char)pickChar).Equals('\n'))
                    nextChar = _binReader.Read();
                break;
            }
            else
                line += current;
            nextChar = _binReader.Read();
        }
        return line;
    }

    /// <summary>
    /// Reads some lines of characters from the current stream at the current position and returns the data as a collection of string.
    /// </summary>
    /// <param name="totalLines">The total number of lines to read (set as 0 to read from current position to end of file).</param>
    /// <returns>The next lines from the input stream, or empty collectoin if the end of the input stream is reached</returns>
    public List<string> ReadLines(int totalLines)
    {
        if (totalLines < 1 && this.Position == 0)
            return this.ReadAllLines();

        _lines = new List<string>();
        int counter = 0;
        string line = this.ReadLine();
        while (line != null)
        {
            _lines.Add(line);
            counter++;
            if (totalLines > 0 && counter >= totalLines)
                break;
            line = this.ReadLine();
        }
        return _lines;
    }

    /// <summary>
    /// Reads all lines of characters from the current stream (from the begin to end) and returns the data as a collection of string.
    /// </summary>
    /// <returns>The next lines from the input stream, or empty collectoin if the end of the input stream is reached</returns>
    public List<string> ReadAllLines()
    {
        if (_streamReader == null)
            _streamReader = new StreamReader(_fileStream);
        _streamReader.BaseStream.Seek(0, SeekOrigin.Begin);
        _lines = new List<string>();
        string line = _streamReader.ReadLine();
        while (line != null)
        {
            _lines.Add(line);
            line = _streamReader.ReadLine();
        }
        return _lines;
    }

    /// <summary>
    /// Gets the length of text file (in bytes).
    /// </summary>
    public long Length
    {
        get { return _length; }
    }

    /// <summary>
    /// Gets or sets the current reading position.
    /// </summary>
    public long Position
    {
        get
        {
            if (_binReader == null)
                return -1;
            else
                return _binReader.BaseStream.Position;
        }
        set
        {
            if (_binReader == null)
                return;
            else if (value >= this.Length)
                this.SetPosition(this.Length);
            else
                this.SetPosition(value);
        }
    }

    void SetPosition(long position)
    {
        _binReader.BaseStream.Seek(position, SeekOrigin.Begin);
    }

    /// <summary>
    /// Gets the lines after reading.
    /// </summary>
    public List<string> Lines
    {
        get
        {
            return _lines;
        }
    }

    /// <summary>
    /// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
    /// </summary>
    public void Dispose()
    {
        if (_binReader != null)
            _binReader.Close();
        if (_streamReader != null)
        {
            _streamReader.Close();
            _streamReader.Dispose();
        }
        if (_fileStream != null)
        {
            _fileStream.Close();
            _fileStream.Dispose();
        }
    }

    ~TextFileReader()
    {
        this.Dispose();
    }
}

【讨论】：

【解决方案3】：

这是一个非常棘手的问题。在互联网上对不同的解决方案进行了漫长而疲惫的列举（包括来自这个线程的解决方案，谢谢！）我必须创建自己的自行车。

我有以下要求：

性能 - 读取速度必须非常快，因此一次读取一个字符或使用反射是不可接受的，因此需要缓冲
流式传输 - 文件可能很大，因此将其完全读入内存是不可接受的
Tailing - file tailing 应该可用
长行 - 行可以很长，所以缓冲区不能被限制

稳定 - 单字节错误在使用过程中立即可见。对我来说不幸的是，我发现的几个实现都存在稳定性问题

public class OffsetStreamReader
{
    private const int InitialBufferSize = 4096;    
    private readonly char _bom;
    private readonly byte _end;
    private readonly Encoding _encoding;
    private readonly Stream _stream;
    private readonly bool _tail;

    private byte[] _buffer;
    private int _processedInBuffer;
    private int _informationInBuffer;

    public OffsetStreamReader(Stream stream, bool tail)
    {
        _buffer = new byte[InitialBufferSize];
        _processedInBuffer = InitialBufferSize;

        if (stream == null || !stream.CanRead)
            throw new ArgumentException("stream");

        _stream = stream;
        _tail = tail;
        _encoding = Encoding.UTF8;

        _bom = '\uFEFF';
        _end = _encoding.GetBytes(new [] {'\n'})[0];
    }

    public long Offset { get; private set; }

    public string ReadLine()
    {
        // Underlying stream closed
        if (!_stream.CanRead)
            return null;

        // EOF
        if (_processedInBuffer == _informationInBuffer)
        {
            if (_tail)
            {
                _processedInBuffer = _buffer.Length;
                _informationInBuffer = 0;
                ReadBuffer();
            }

            return null;
        }

        var lineEnd = Search(_buffer, _end, _processedInBuffer);
        var haveEnd = true;

        // File ended but no finalizing newline character
        if (lineEnd.HasValue == false && _informationInBuffer + _processedInBuffer < _buffer.Length)
        {
            if (_tail)
                return null;
            else
            {
                lineEnd = _informationInBuffer;
                haveEnd = false;
            }
        }

        // No end in current buffer
        if (!lineEnd.HasValue)
        {
            ReadBuffer();
            if (_informationInBuffer != 0)
                return ReadLine();

            return null;
        }

        var arr = new byte[lineEnd.Value - _processedInBuffer];
        Array.Copy(_buffer, _processedInBuffer, arr, 0, arr.Length);

        Offset = Offset + lineEnd.Value - _processedInBuffer + (haveEnd ? 1 : 0);
        _processedInBuffer = lineEnd.Value + (haveEnd ? 1 : 0);

        return _encoding.GetString(arr).TrimStart(_bom).TrimEnd('\r', '\n');
    }

    private void ReadBuffer()
    {
        var notProcessedPartLength = _buffer.Length - _processedInBuffer;

        // Extend buffer to be able to fit whole line to the buffer
        // Was     [NOT_PROCESSED]
        // Become  [NOT_PROCESSED        ]
        if (notProcessedPartLength == _buffer.Length)
        {
            var extendedBuffer = new byte[_buffer.Length + _buffer.Length/2];
            Array.Copy(_buffer, extendedBuffer, _buffer.Length);
            _buffer = extendedBuffer;
        }

        // Copy not processed information to the begining
        // Was    [PROCESSED NOT_PROCESSED]
        // Become [NOT_PROCESSED          ]
        Array.Copy(_buffer, (long) _processedInBuffer, _buffer, 0, notProcessedPartLength);

        // Read more information to the empty part of buffer
        // Was    [ NOT_PROCESSED                   ]
        // Become [ NOT_PROCESSED NEW_NOT_PROCESSED ]
        _informationInBuffer = notProcessedPartLength + _stream.Read(_buffer, notProcessedPartLength, _buffer.Length - notProcessedPartLength);

        _processedInBuffer = 0;
    }

    private int? Search(byte[] buffer, byte byteToSearch, int bufferOffset)
    {
        for (int i = bufferOffset; i < buffer.Length - 1; i++)
        {
            if (buffer[i] == byteToSearch)
                return i;
        }
        return null;
    }
}

【讨论】：

我有一个日志文件，当使用 offsetreader 读取时会导致它进入无限循环...
你能以某种方式分享那个文件吗？

【解决方案4】：

虽然 Thomas Levesque 的解决方案效果很好，但这是我的解决方案。它使用反射，所以它会更慢，但它与编码无关。另外，我还添加了 Seek 扩展。

/// <summary>Useful <see cref="StreamReader"/> extentions.</summary>
public static class StreamReaderExtentions
{
    /// <summary>Gets the position within the <see cref="StreamReader.BaseStream"/> of the <see cref="StreamReader"/>.</summary>
    /// <remarks><para>This method is quite slow. It uses reflection to access private <see cref="StreamReader"/> fields. Don't use it too often.</para></remarks>
    /// <param name="streamReader">Source <see cref="StreamReader"/>.</param>
    /// <exception cref="ArgumentNullException">Occurs when passed <see cref="StreamReader"/> is null.</exception>
    /// <returns>The current position of this stream.</returns>
    public static long GetPosition(this StreamReader streamReader)
    {
        if (streamReader == null)
            throw new ArgumentNullException("streamReader");

        var charBuffer = (char[])streamReader.GetType().InvokeMember("charBuffer", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null);
        var charPos = (int)streamReader.GetType().InvokeMember("charPos", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null);
        var charLen = (int)streamReader.GetType().InvokeMember("charLen", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null);

        var offsetLength = streamReader.CurrentEncoding.GetByteCount(charBuffer, charPos, charLen - charPos);

        return streamReader.BaseStream.Position - offsetLength;
    }

    /// <summary>Sets the position within the <see cref="StreamReader.BaseStream"/> of the <see cref="StreamReader"/>.</summary>
    /// <remarks>
    /// <para><see cref="StreamReader.BaseStream"/> should be seekable.</para>
    /// <para>This method is quite slow. It uses reflection and flushes the charBuffer of the <see cref="StreamReader.BaseStream"/>. Don't use it too often.</para>
    /// </remarks>
    /// <param name="streamReader">Source <see cref="StreamReader"/>.</param>
    /// <param name="position">The point relative to origin from which to begin seeking.</param>
    /// <param name="origin">Specifies the beginning, the end, or the current position as a reference point for origin, using a value of type <see cref="SeekOrigin"/>. </param>
    /// <exception cref="ArgumentNullException">Occurs when passed <see cref="StreamReader"/> is null.</exception>
    /// <exception cref="ArgumentException">Occurs when <see cref="StreamReader.BaseStream"/> is not seekable.</exception>
    /// <returns>The new position in the stream. This position can be different to the <see cref="position"/> because of the preamble.</returns>
    public static long Seek(this StreamReader streamReader, long position, SeekOrigin origin)
    {
        if (streamReader == null)
            throw new ArgumentNullException("streamReader");

        if (!streamReader.BaseStream.CanSeek)
            throw new ArgumentException("Underlying stream should be seekable.", "streamReader");

        var preamble = (byte[])streamReader.GetType().InvokeMember("_preamble", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null);
        if (preamble.Length > 0 && position < preamble.Length) // preamble or BOM must be skipped
            position += preamble.Length;

        var newPosition = streamReader.BaseStream.Seek(position, origin); // seek
        streamReader.DiscardBufferedData(); // this updates the buffer

        return newPosition;
    }
}

【讨论】：

【解决方案5】：

这行得通吗：

using (var sr = new StreamReader(@"C:\Temp\LineTest.txt")) {
  string line;
  long pos = 0;
  while ((line = sr.ReadLine()) != null) {
    Console.Write("{0:d3} ", pos);
    Console.WriteLine(line);
    pos += line.Length;
  }
}

【讨论】：

不幸的是不是，因为我必须接受不同类型的换行符（想想这个 \n、\r\n、\r），并且数字会出现偏差。如果我坚持使用 consistent 换行符分隔符（在实践中很可能混合使用）并且如果我先对其进行探测以了解实际偏移量，这可能会起作用。所以 - 我试图避免走那条路。
@Benjamin：该死 - 我刚刚发布了一个类似的答案，它明确依赖于一致的换行符分隔符......
那么我认为你最好使用 StreamReader.Read() 手动完成。
@Jon：呵呵。正如我所说：这可能是方式，而不是使用普通的 Stream - 如果这是唯一的两个选项，我必须掷骰子并忍受后果：要么一致的分隔符（坏对于在多个平台上处理的文件，在不良编辑器中复制/粘贴等）或 Stream 的东西（无聊的低级行解析和字符串编码混乱，大量样板代码看似低回报）
那没有多大帮助。我必须放弃整个StreamReader。甚至Read() 也会导致对底层流的块读取，并将BaseStream.Position 移动到我的示例中的25。 一个字符之后。