使用 FileStream ASPNET 发送 500MB 大文件时出现 OutOfMemoryException

Question

我正在使用 Filestream 读取大文件 (> 500 MB)，我得到了 OutOfMemoryException。

我使用 Asp.net、.net 3.5、win2003、iis 6.0

我想在我的应用中使用这个：

从 Oracle 读取数据

使用 FileStream 和 BZip2 解压缩文件

读取未压缩的文件并将其发送到asp.net页面进行下载。

当我从磁盘读取文件时，失败！！！并获得 OutOfMemory...

。我的代码是：

using (var fs3 = new FileStream(filePath2, FileMode.Open, FileAccess.Read)) 
        { 
          byte[] b2 = ReadFully(fs3, 1024); 
        } 

 // http://www.yoda.arachsys.com/csharp/readbinary.html
 public static byte[] ReadFully(Stream stream, int initialLength) 
  { 
    // If we've been passed an unhelpful initial length, just 
    // use 32K. 
    if (initialLength < 1) 
    { 
      initialLength = 32768; 
    } 

    byte[] buffer = new byte[initialLength]; 
    int read = 0; 

    int chunk; 
    while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0) 
    { 
      read += chunk; 

      // If we've reached the end of our buffer, check to see if there's 
      // any more information 
      if (read == buffer.Length) 
      { 
        int nextByte = stream.ReadByte(); 

        // End of stream? If so, we're done 
        if (nextByte == -1) 
        { 
          return buffer; 
        } 

        // Nope. Resize the buffer, put in the byte we've just 
        // read, and continue 
        byte[] newBuffer = new byte[buffer.Length * 2]; 
        Array.Copy(buffer, newBuffer, buffer.Length); 
        newBuffer[read] = (byte)nextByte; 
        buffer = newBuffer; 
        read++; 
      } 
    } 
    // Buffer is now too big. Shrink it. 
    byte[] ret = new byte[read]; 
    Array.Copy(buffer, ret, read); 
    return ret; 
  } 

现在，我可以更好地说明我的问题。

使用 FileStream 和 BZip2 解压文件是可以的，一切正常。

问题如下：

在 byte[] 中读取磁盘中的大文件 (> 500 MB) 并将字节发送到 Response (asp.net) 以供下载。

使用时

http://www.yoda.arachsys.com/csharp/readbinary.html

public static byte[] ReadFully

我收到错误：OutOfMemoryException...

如果 BufferedStream 比 Stream (FileStream, MemoryStream, ...) 更好？？

使用 BufferedStream ，我可以读取 700 MB 的大文件吗？ （任何使用 BufferedStream 下载大文件的示例代码源）

我认为，这就是问题所在：不是“如何将 500mb 文件读入内存？” , 但是“如何将大文件发送到 ASPNET 响应流？”

我通过 Cheeso 找到了这段代码：

using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))  
{  
   Response.BufferOutput= false;   // to prevent buffering 
   byte[] buffer = new byte[1024]; 
   int bytesRead = 0; 
   while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)  
   { 
       Response.OutputStream.Write(buffer, 0, bytesRead); 
   } 
}

这是好代码吗？对高性能有何改进？？

一位同事说我，使用

Response.TransmitFile(filePath);

现在，另一个问题，更好的 TransmitFile 或 Cheeso 的代码？？

多年前，在 msdn 杂志上出现过关于它的精彩文章，但我无法访问 http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/，

更新：您可以使用链接中的webarchive访问：https://web.archive.org/web/20070627063111/http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/

有什么建议、cmets、示例代码源？？

Answer 1

我已经创建了下载页面，允许用户在几个月前下载最多 4gb（可能更多）。这是我的工作 sn-p：

  private void TransmitFile(string fullPath, string outFileName)
    {
        System.IO.Stream iStream = null;

        // Buffer to read 10K bytes in chunk:
        byte[] buffer = new Byte[10000];

        // Length of the file:
        int length;

        // Total bytes to read:
        long dataToRead;

        // Identify the file to download including its path.
        string filepath = fullPath;

        // Identify the file name.
        string filename = System.IO.Path.GetFileName(filepath);

        try
        {
            // Open the file.
            iStream = new System.IO.FileStream(filepath, System.IO.FileMode.Open,
                        System.IO.FileAccess.Read, System.IO.FileShare.Read);


            // Total bytes to read:
            dataToRead = iStream.Length;

            Response.Clear();
            Response.ContentType = "application/octet-stream";
            Response.AddHeader("Content-Disposition", "attachment; filename=" + outFileName);
            Response.AddHeader("Content-Length", iStream.Length.ToString());

            // Read the bytes.
            while (dataToRead > 0)
            {
                // Verify that the client is connected.
                if (Response.IsClientConnected)
                {
                    // Read the data in buffer.
                    length = iStream.Read(buffer, 0, 10000);

                    // Write the data to the current output stream.
                    Response.OutputStream.Write(buffer, 0, length);

                    // Flush the data to the output.
                    Response.Flush();

                    buffer = new Byte[10000];
                    dataToRead = dataToRead - length;
                }
                else
                {
                    //prevent infinite loop if user disconnects
                    dataToRead = -1;
                }
            }
        }
        catch (Exception ex)
        {
            throw new ApplicationException(ex.Message);
        }
        finally
        {
            if (iStream != null)
            {
                //Close the file.
                iStream.Close();
            }
            Response.Close();
        }
    }

Answer 2

您不需要将整个文件保存在内存中，只需读取它并在循环中写入响应流。

Answer 3

我在搜索从控制器返回 FileStreamResult 时遇到了这个问题，因为我在处理大型流时遇到问题，因为 .Net 试图一次构建整个响应。 Pavel Morshenyuk's answer 是一个巨大的帮助，但我想我会分享我最终得到的 BufferedFileStreamResult。

/// <summary>Based upon https://stackoverflow.com/a/3363015/595473 </summary>
public class BufferedFileStreamResult : System.Web.Mvc.FileStreamResult
{
    public BufferedFileStreamResult(System.IO.Stream stream, string contentType, string fileDownloadName)
        : base(stream, contentType)
    {
        FileDownloadName = fileDownloadName;
    }

    public int BufferSize { get; set; } = 16 * 1024 * 1024;//--16MiB

    protected override void WriteFile(System.Web.HttpResponseBase response)
    {
        try
        {
            response.Clear();
            response.Headers.Set("Content-Disposition", $"attachment; filename={FileDownloadName}");
            response.Headers.Set("Content-Length", FileStream.Length.ToString());

            byte[] buffer;
            int bytesRead;

            while (response.IsClientConnected)//--Prevent infinite loop if user disconnects
            {
                buffer = new byte[BufferSize];

                //--Read the data in buffer
                if ((bytesRead = FileStream.Read(buffer, 0, BufferSize)) == 0)
                {
                    break;//--Stop writing if there's nothing left to write
                }

                //--Write the data to the current output stream
                response.OutputStream.Write(buffer, 0, bytesRead);

                //--Flush the data to the output
                response.Flush();
            }
        }
        finally
        {
            FileStream?.Close();
            response.Close();
        }
    }
}

现在，在我的控制器中，我可以

return new BufferedFileStreamResult(stream, contentType, fileDownloadName);

Answer 4

解决方案不止一种

1- 使用 RecyclableMemoryStream 代替 MemoryStream 解决方案

您可以在此处阅读有关 RecyclableMemoryStream 的更多信息： http://www.philosophicalgeek.com/2015/02/06/announcing-microsoft-io-recycablememorystream/

https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream

2- 使用 MemoryTributary 代替 MemoryStream

您可以在此处阅读有关 MemoryTributary 的更多信息：

https://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream?msg=5257615#xx5257615xx

    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Runtime.InteropServices;

   namespace LiquidEngine.Tools
       {
/// <summary>
/// MemoryTributary is a re-implementation of MemoryStream that uses a dynamic list of byte arrays as a backing store, instead of a single byte array, the allocation
/// of which will fail for relatively small streams as it requires contiguous memory.
/// </summary>
public class MemoryTributary : Stream       /* http://msdn.microsoft.com/en-us/library/system.io.stream.aspx */
{
    #region Constructors

    public MemoryTributary()
    {
        Position = 0;
    }

    public MemoryTributary(byte[] source)
    {
        this.Write(source, 0, source.Length);
        Position = 0;
    }

    /* length is ignored because capacity has no meaning unless we implement an artifical limit */
    public MemoryTributary(int length)
    {
        SetLength(length);
        Position = length;
        byte[] d = block;   //access block to prompt the allocation of memory
        Position = 0;
    }

    #endregion

    #region Status Properties

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return true; }
    }

    public override bool CanWrite
    {
        get { return true; }
    }

    #endregion

    #region Public Properties

    public override long Length
    {
        get { return length; }
    }

    public override long Position { get; set; }

    #endregion

    #region Members

    protected long length = 0;

    protected long blockSize = 65536;

    protected List<byte[]> blocks = new List<byte[]>();

    #endregion

    #region Internal Properties

    /* Use these properties to gain access to the appropriate block of memory for the current Position */

    /// <summary>
    /// The block of memory currently addressed by Position
    /// </summary>
    protected byte[] block
    {
        get
        {
            while (blocks.Count <= blockId)
                blocks.Add(new byte[blockSize]);
            return blocks[(int)blockId];
        }
    }
    /// <summary>
    /// The id of the block currently addressed by Position
    /// </summary>
    protected long blockId
    {
        get { return Position / blockSize; }
    }
    /// <summary>
    /// The offset of the byte currently addressed by Position, into the block that contains it
    /// </summary>
    protected long blockOffset
    {
        get { return Position % blockSize; }
    }

    #endregion

    #region Public Stream Methods

    public override void Flush()
    {
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        long lcount = (long)count;

        if (lcount < 0)
        {
            throw new ArgumentOutOfRangeException("count", lcount, "Number of bytes to copy cannot be negative.");
        }

        long remaining = (length - Position);
        if (lcount > remaining)
            lcount = remaining;

        if (buffer == null)
        {
            throw new ArgumentNullException("buffer", "Buffer cannot be null.");
        }
        if (offset < 0)
        {
            throw new ArgumentOutOfRangeException("offset",offset,"Destination offset cannot be negative.");
        }

        int read = 0;
        long copysize = 0;
        do
        {
            copysize = Math.Min(lcount, (blockSize - blockOffset));
            Buffer.BlockCopy(block, (int)blockOffset, buffer, offset, (int)copysize);
            lcount -= copysize;
            offset += (int)copysize;

            read += (int)copysize;
            Position += copysize;

        } while (lcount > 0);

        return read;

    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        switch (origin)
        {
            case SeekOrigin.Begin:
                Position = offset;
                break;
            case SeekOrigin.Current:
                Position += offset;
                break;
            case SeekOrigin.End:
                Position = Length - offset;
                break;
        }
        return Position;
    }

    public override void SetLength(long value)
    {
        length = value;
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        long initialPosition = Position;
        int copysize;
        try
        {
            do
            {
                copysize = Math.Min(count, (int)(blockSize - blockOffset));

                EnsureCapacity(Position + copysize);

                Buffer.BlockCopy(buffer, (int)offset, block, (int)blockOffset, copysize);
                count -= copysize;
                offset += copysize;

                Position += copysize;

            } while (count > 0);
        }
        catch (Exception e)
        {
            Position = initialPosition;
            throw e;
        }
    }

    public override int ReadByte()
    {
        if (Position >= length)
            return -1;

        byte b = block[blockOffset];
        Position++;

        return b;
    }

    public override void WriteByte(byte value)
    {
        EnsureCapacity(Position + 1);
        block[blockOffset] = value;
        Position++;
    }

    protected void EnsureCapacity(long intended_length)
    {
        if (intended_length > length)
            length = (intended_length);
    }

    #endregion

    #region IDispose

    /* http://msdn.microsoft.com/en-us/library/fs2xkftw.aspx */
    protected override void Dispose(bool disposing)
    {
        /* We do not currently use unmanaged resources */
        base.Dispose(disposing);
    }

    #endregion

    #region Public Additional Helper Methods

    /// <summary>
    /// Returns the entire content of the stream as a byte array. This is not safe because the call to new byte[] may 
    /// fail if the stream is large enough. Where possible use methods which operate on streams directly instead.
    /// </summary>
    /// <returns>A byte[] containing the current data in the stream</returns>
    public byte[] ToArray()
    {
        long firstposition = Position;
        Position = 0;
        byte[] destination = new byte[Length];
        Read(destination, 0, (int)Length);
        Position = firstposition;
        return destination;
    }

    /// <summary>
    /// Reads length bytes from source into the this instance at the current position.
    /// </summary>
    /// <param name="source">The stream containing the data to copy</param>
    /// <param name="length">The number of bytes to copy</param>
    public void ReadFrom(Stream source, long length)
    {
        byte[] buffer = new byte[4096];
        int read;
        do
        {
            read = source.Read(buffer, 0, (int)Math.Min(4096, length));
            length -= read;
            this.Write(buffer, 0, read);

        } while (length > 0);
    }

    /// <summary>
    /// Writes the entire stream into destination, regardless of Position, which remains unchanged.
    /// </summary>
    /// <param name="destination">The stream to write the content of this stream to</param>
    public void WriteTo(Stream destination)
    {
        long initialpos = Position;
        Position = 0;
        this.CopyTo(destination);
        Position = initialpos;
    }

    #endregion
}

}