复制流的最后 16 个字节以外的所有字节？早期检测流结束？

Question

这与 C# 相关。我们有一种情况，我们需要将整个源流复制到目标流中除了最后 16 个字节。

编辑：流的范围可达 40GB，因此不能进行一些静态字节 [] 分配（例如：.ToArray()）

查看MSDN documentation，似乎只有当返回值为0时，我们才能可靠确定流的结束。0和the requested size之间的返回值可以暗示字节是“当前不可用”（真正是什么意思？）

目前它复制每个字节如下。 inStream 和 outStream 是通用的 - 可以是内存、磁盘或网络流（实际上还有更多）。

public static void StreamCopy(Stream inStream, Stream outStream)
{
    var buffer = new byte[8*1024];
    var last16Bytes = new byte[16];
    int bytesRead;
    while ((bytesRead = inStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        outStream.Write(buffer, 0, bytesRead);
    }
    // Issues:
    // 1. We already wrote the last 16 bytes into 
    //    outStream (possibly over the n/w)
    // 2. last16Bytes = ? (inStream may not necessarily support rewinding)
}

什么是可靠的方法来确保除了最后 16 个之外的所有内容都被复制？我可以考虑在 inStream 上使用 Position 和 Length 但在 MSDN 上有一个陷阱，上面写着

如果派生自 Stream 的类不支持查找，则对 Length、SetLength、Position 和 Seek 的调用将引发 NotSupportedException。 .

Answer 1

1. 从输入流中读取 1 到 n 个字节。¹

2. 将字节追加到circular buffer。²

3. 将循环缓冲区中的前 max(0, b - 16) 个字节写入输出流，其中 b 是循环缓冲区。

4. 从循环缓冲区中删除刚刚写入的字节。

5. 转到步骤 1。

¹这就是Read 方法所做的——如果您调用int n = Read(buffer, 0, 500);，它将读取1 到500 个字节到buffer 并返回读取的字节数。如果Read 返回 0，则您已到达流的末尾。

²为了获得最佳性能，您可以将字节直接从输入流读取到循环缓冲区中。这有点棘手，因为您必须处理缓冲区底层数组内的环绕。

Answer 2

以下解决方案快速且经过测试。希望它有用。它使用了您已经想到的双缓冲想法。 编辑：简化循环，删除将第一次迭代与其余迭代分开的条件。

public static void StreamCopy(Stream inStream, Stream outStream) {
     // Define the size of the chunk to copy during each iteration (1 KiB)
     const int blockSize = 1024;
     const int bytesToOmit = 16;

     const int buffSize = blockSize + bytesToOmit;

     // Generate working buffers
     byte[] buffer1 = new byte[buffSize];
     byte[] buffer2 = new byte[buffSize];

     // Initialize first iteration
     byte[] curBuffer = buffer1;
     byte[] prevBuffer = null;

     int bytesRead;

     // Attempt to fully fill the buffer
     bytesRead = inStream.Read(curBuffer, 0, buffSize);
     if( bytesRead == buffSize ) {
        // We succesfully retrieved a whole buffer, we will output
        // only [blockSize] bytes, to avoid writing to the last
        // bytes in the buffer in case the remaining 16 bytes happen to 
        // be the last ones
        outStream.Write(curBuffer, 0, blockSize);
     } else {
        // We couldn't retrieve the whole buffer
        int bytesToWrite = bytesRead - bytesToOmit;
        if( bytesToWrite > 0 ) {
           outStream.Write(curBuffer, 0, bytesToWrite);
        }
        // There's no more data to process
        return;
     }

     curBuffer = buffer2;
     prevBuffer = buffer1;

     while( true ) {
        // Attempt again to fully fill the buffer
        bytesRead = inStream.Read(curBuffer, 0, buffSize);
        if( bytesRead == buffSize ) {
           // We retrieved the whole buffer, output first the last 16 
           // bytes of the previous buffer, and output just [blockSize]
           // bytes from the current buffer
           outStream.Write(prevBuffer, blockSize, bytesToOmit);
           outStream.Write(curBuffer, 0, blockSize);
        } else {
           // We could not retrieve a complete buffer 
           if( bytesRead <= bytesToOmit ) {
              // The bytes to output come solely from the previous buffer
              outStream.Write(prevBuffer, blockSize, bytesRead);
           } else {
              // The bytes to output come from the previous buffer and
              // the current buffer
              outStream.Write(prevBuffer, blockSize, bytesToOmit);
              outStream.Write(curBuffer, 0, bytesRead - bytesToOmit);
           }
           break;
        }
        // swap buffers for next iteration
        byte[] swap = prevBuffer;
        prevBuffer = curBuffer;
        curBuffer = swap;
     }
  }

static void Assert(Stream inStream, Stream outStream) {
   // Routine that tests the copy worked as expected
         inStream.Seek(0, SeekOrigin.Begin);
         outStream.Seek(0, SeekOrigin.Begin);
         Debug.Assert(outStream.Length == Math.Max(inStream.Length - bytesToOmit, 0));
         for( int i = 0; i < outStream.Length; i++ ) {
            int byte1 = inStream.ReadByte();
            int byte2 = outStream.ReadByte();
            Debug.Assert(byte1 == byte2);
         }

      }

一个更简单的编码解决方案，但速度较慢，因为它可以在字节级别工作，是在输入流和输出流之间使用中间队列。该进程将首先从输入流中读取 16 个字节并将其排入队列。然后它将遍历剩余的输入字节，从输入流中读取一个字节，将其入队，然后将一个字节出队。出队的字节将被写入输出流，直到输入流中的所有字节都被处理。不需要的 16 个字节应该留在中间队列中。

希望这会有所帮助！

=)

Answer 3

使用循环缓冲区听起来不错，但 .NET 中没有循环缓冲区类，这意味着无论如何都需要额外的代码。我最终得到了以下算法，一种map and copy - 我认为这很简单。为了在此处进行自我描述，变量名称比平时长。

这流过缓冲区

[outStream] <== [tailBuf] <== [mainBuf] <== [inStream]

public byte[] CopyStreamExtractLastBytes(Stream inStream, Stream outStream,
                                         int extractByteCount)
{
    //var mainBuf = new byte[1024*4]; // 4K buffer ok for network too
    var mainBuf = new byte[4651]; // nearby prime for testing

    int mainBufValidCount;
    var tailBuf = new byte[extractByteCount];
    int tailBufValidCount = 0;

    while ((mainBufValidCount = inStream.Read(mainBuf, 0, mainBuf.Length)) > 0)
    {
        // Map: how much of what (passthru/tail) lives where (MainBuf/tailBuf)
        // more than tail is passthru
        int totalPassthruCount = Math.Max(0, tailBufValidCount + 
                                    mainBufValidCount - extractByteCount);
        int tailBufPassthruCount = Math.Min(tailBufValidCount, totalPassthruCount);
        int tailBufTailCount = tailBufValidCount - tailBufPassthruCount;
        int mainBufPassthruCount = totalPassthruCount - tailBufPassthruCount;
        int mainBufResidualCount = mainBufValidCount - mainBufPassthruCount;

        // Copy: Passthru must be flushed per FIFO order (tailBuf then mainBuf)
        outStream.Write(tailBuf, 0, tailBufPassthruCount);
        outStream.Write(mainBuf, 0, mainBufPassthruCount);

        // Copy: Now reassemble/compact tail into tailBuf
        var tempResidualBuf = new byte[extractByteCount];
        Array.Copy(tailBuf, tailBufPassthruCount, tempResidualBuf, 0, 
                      tailBufTailCount);
        Array.Copy(mainBuf, mainBufPassthruCount, tempResidualBuf, 
                      tailBufTailCount, mainBufResidualCount);
        tailBufValidCount = tailBufTailCount + mainBufResidualCount;
        tailBuf = tempResidualBuf;
    }
    return tailBuf;
}