使用 UTF8 编码将字节转换为字符串是否安全？

Question

今天看到一个问题，代码是这样的：

var accumulator = "";
var buffer = new byte[8192];
while (true)
{
    var readed = stream.Read(buffer, 0, buffer.Length);
    accumulator += Encoding.UTF8.GetString(buffer, 0, readed);
    if (readed < buffer.Length)
        break;
}
var result = Encoding.UTF8.GetBytes(accumulator);

我知道这段代码效率低下，但它安全吗？是否有一些字节序列会破坏结果？

Answer 1

代码明显坏了；如果这是作为答案提出的，那么您应该提请作者注意该错误。

显然，UTF-8 序列可以超过一个字节。如果有一个多字节序列从当前缓冲区的末尾开始，并在下一个缓冲区的开头继续，那么每个缓冲区到字符串的转换都会出错。

Answer 2

安全的方法是使用有状态的 UTF8 解码器，可以从Encoding.UTF8.GetDecoder() 获得。

有状态解码器将在内部保留对应于不完整多字节序列的字节。下次你给它更多字节时，它会完成序列并返回它从序列中解码的字符。

以下是如何使用它的示例。在我的实现中，我使用了一个char[] 缓冲区，该缓冲区的大小使得我们总是有足够的空间来存储X 个字节的完整转换。这样，我们只需执行两次内存分配即可读取整个流。

public static string ReadStringFromStream( Stream stream )
{
    // --- Byte-oriented state ---
    // A nice big buffer for us to use to read from the stream.
    byte[] byteBuffer = new byte[8192];

    // --- Char-oriented state ---
    // Gets a stateful UTF8 decoder that holds onto unused bytes when multi-byte sequences
    // are split across multiple byte buffers.
    var decoder = Encoding.UTF8.GetDecoder();

    // Initialize a char buffer, and make it large enough that it will be able to fit
    // a full reads-worth of data from the byte buffer without needing to be resized.
    char[] charBuffer = new char[Encoding.UTF8.GetMaxCharCount( byteBuffer.Length )];

    // --- Output ---
    StringBuilder stringBuilder = new StringBuilder();

    // --- Working state ---
    int bytesRead;
    int charsConverted;
    bool lastRead = false;

    do
    {
        // Read a chunk of bytes from our stream.
        bytesRead = stream.Read( byteBuffer, 0, byteBuffer.Length );

        // If we read 0 bytes, we hit the end of stream.
        // We're going to tell the converter to flush, and then we're going to stop.
        lastRead = ( bytesRead == 0 );

        // Convert the bytes into characters, flushing if this is our last conversion.
        charsConverted = decoder.GetChars( 
            byteBuffer, 
            0, 
            bytesRead, 
            charBuffer, 
            0, 
            lastRead 
        );

        // Build up a string in a character buffer.
        stringBuilder.Append( charBuffer, 0, charsConverted );
    }
    while( lastRead == false );

    return stringBuilder.ToString();
}