如何将文件动态添加到存储在 Azure Blob 存储中的 zip 存档？

Question

我在 Azure 中有一个进程，它生成大量 pdf 报告文件并将它们存储在 blob 存储中。我没有单独发送所有这些链接，而是生成一个 zip 文件并将此链接发送给用户。

这个过程全部在一个进程中完成，并且一直运行良好。最近，我在将文件添加到 zip 存档时遇到 OutOfMemory 异常错误，我正在努力寻找解决方案。

下面是我用来创建 zip 文件的代码（注意：使用 SharpLibZip 库）。目前，它在添加大约 45 个每个文件（PDF）约 3.5Mb 的文件后失败并出现 OutOfMemoryException。当我打线时发生故障：zipStream.PutNextEntry(newEntry)。

有谁知道我可以如何改进这个过程？在这个级别上失败似乎很小的 zip 文件。

Using outputMemStream As New MemoryStream()

    Using zipStream As New ICSharpCode.SharpZipLib.Zip.ZipOutputStream(outputMemStream)
          zipStream.SetLevel(7)

          Dim collD3 As UserSurveyReportCollection = GetFileList(RequestID)

          For Each entityD2 As UserSurveyReport In collD3

              Try
                  Dim strF As String = entityD2.FileLocation

                 'Download blob as memorystream and add this stream to the zip file
                 Dim msR As New MemoryStream 
                 msR = objA.DownloadBlobAsMemoryStream(azureAccount, ReportFolder, entityD2.FileName)
                 msR.Seek(0, SeekOrigin.Begin)

                'Determine file name used in zip file archive for item
                 Dim strZipFileName As String = DetermineZipSourceName(entityD2, strFolder, strFileName)

                 'Add MemoryStream to ZipFile Stream
                 Dim newEntry As ICSharpCode.SharpZipLib.Zip.ZipEntry = New ICSharpCode.SharpZipLib.Zip.ZipEntry(strZipFileName)
                 newEntry.DateTime = DateTime.Now

                 zipStream.PutNextEntry(newEntry)
                 msR.CopyTo(zipStream)
                 zipStream.CloseEntry()

                 msR = Nothing
                 zipStream.Flush()

                 intCounter += 1

        End If

    Catch exZip As Exception

    End Try

  Next


    zipStream.IsStreamOwner = False
    zipStream.Finish()
    zipStream.Close()

    outputMemStream.Position = 0

    Dim bytes As Byte() = outputMemStream.ToArray()
    result.Comment = objA.UploadBlob(bytes, azureAccount, ReportFolder, entityReport.FileName).AbsolutePath


    End Using
  End Using

Answer 1

对于从事 C# 交易并希望将大型 zip 文件写入 blob 存储的任何人：

var blob = container.GetBlockBlobReference(outputFilename);
using (var stream = await blob.OpenWriteAsync())
using (var zip = new ZipArchive(stream, ZipArchiveMode.Create))
{
    for (int i = 0; i < 2000; i++)
    {
        using (var randomStream = CreateRandomStream(2))
        {
            var entry = zip.CreateEntry($"{i}.zip", CompressionLevel.Optimal);
            using (var innerFile = entry.Open())
            {
                await randomStream.CopyToAsync(innerFile);
            }
        }
    }
}

这出奇的好。应用程序内存大约 20Mb，在流向 Azure 时 CPU 非常低。我已经毫无问题地创建了非常大的输出文件（> 4.5Gb）

Answer 2

我找到了解决方案。这种方法似乎最大限度地减少了内存中 zip 文件创建的内存使用量，并将生成的 zip 存档加载到 Azure 中的 blob 存储。这使用本机 System.IO.Compression 库而不是 3rd 方 zip 库。

我创建了一个名为 ZipModel 的类，它只有一个文件名和 blob。我创建了一个列表，并将其传递给下面的函数。我希望这可以帮助处于同样困境的其他人。

    Private Function SendBlobsToZipFile(ByVal destinationBlob As CloudBlockBlob, ByVal sourceBlobs As List(Of ZipModel)) As ResultDetail

    Dim result As Boolean = True
    Dim resultCounter as Integer = 0

    Using blobWriteStream As Stream = destinationBlob.OpenWrite()

        Using archive As ZipArchive = New ZipArchive(blobWriteStream, ZipArchiveMode.Create)

            For Each zipM As ZipModel In sourceBlobs
                Try
                    Dim strName As String = String.Format("{0}\{1}", zipM.FolderName, zipM.FileName)
                    Dim archiveEntry As ZipArchiveEntry = archive.CreateEntry(strName, CompressionLevel.Optimal)

                    Using archiveWriteStream As Stream = archiveEntry.Open()
                        zipM.ZipBlob.DownloadToStream(archiveWriteStream)
                        resultCounter  += 1
                    End Using
                Catch ex As Exception

                    result = False

                End Try

            Next

        End Using
    End Using

    Return result


End Function