【发布时间】：2008-10-28 16:42:40
【问题描述】：
当我通过java.util.zip.* 创建一个 zip 存档时，有没有办法将生成的存档拆分为多个卷？
假设我的整个存档有 filesize 和 24 MB，我想将其拆分为 3 个文件，每个文件的大小限制为 10 MB。
是否有具有此功能的 zip API？或者有什么其他好的方法来实现这一点？
谢谢 托尔斯滕
【问题讨论】：
【发布时间】：2008-10-28 16:42:40
【问题描述】：
检查：http://saloon.javaranch.com/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic&f=38&t=004618
我不知道有任何公共 API 可以帮助您做到这一点。 （尽管如果您不想以编程方式执行此操作，可以使用 WinSplitter 等实用程序来执行此操作）
我没有尝试过，但是，使用 ZippedInput/OutputStream 时的每个 ZipEntry 都有一个压缩大小。在创建压缩文件时，您可能会粗略估计压缩文件的大小。如果您需要 2MB 的压缩文件，那么您可以在条目的累积大小变为 1.9MB 后停止写入文件，为 Manifest 文件和其他 zip 文件特定元素占用 0.1MB。 因此，简而言之，您可以在 ZippedInputStream 上编写一个包装器，如下所示：
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class ChunkedZippedOutputStream {
private ZipOutputStream zipOutputStream;
private final String path;
private final String name;
private long currentSize;
private int currentChunkIndex;
private final long MAX_FILE_SIZE = 16000000; // Whatever size you want
private final String PART_POSTFIX = ".part.";
private final String FILE_EXTENSION = ".zip";
public ChunkedZippedOutputStream(String path, String name) throws FileNotFoundException {
this.path = path;
this.name = name;
constructNewStream();
}
public void addEntry(ZipEntry entry) throws IOException {
long entrySize = entry.getCompressedSize();
if ((currentSize + entrySize) > MAX_FILE_SIZE) {
closeStream();
constructNewStream();
} else {
currentSize += entrySize;
zipOutputStream.putNextEntry(entry);
}
}
private void closeStream() throws IOException {
zipOutputStream.close();
}
private void constructNewStream() throws FileNotFoundException {
zipOutputStream = new ZipOutputStream(new FileOutputStream(new File(path, constructCurrentPartName())));
currentChunkIndex++;
currentSize = 0;
}
private String constructCurrentPartName() {
// This will give names is the form of <file_name>.part.0.zip, <file_name>.part.1.zip, etc.
return name + PART_POSTFIX + currentChunkIndex + FILE_EXTENSION;
}
}
上述程序只是方法的提示，无论如何都不是最终解决方案。
【讨论】：
-
这只是几个单独的 zip 文件，对吧？它不是一个单一的多卷 zip 文件。
-
我们在这个问题的答案中得到了这段代码 - stackoverflow.com/questions/11104889/…
如果目标是让输出与 pkzip 和 winzip 兼容，我不知道有任何开源库可以做到这一点。我们对我们的一个应用程序有类似的要求，我最终编写了自己的实现（与 zip 标准兼容）。如果我记得，对我们来说最困难的事情是我们必须即时生成单个文件（大多数 zip 实用程序的工作方式是它们创建大 zip 文件，然后再返回并拆分它 - 这要容易得多实现。花了大约一天的时间来编写和调试。
zip 标准解释了文件格式的外观。如果你不怕卷起袖子，这绝对是可行的。您必须自己实现一个 zip 文件生成器，但您可以使用 Java 的 Deflator 类为压缩数据生成分段流。您必须自己生成文件和节标题，但它们只是字节 - 一旦您深入了解，就没什么难了。
这是zip specification - K 部分包含您正在寻找的具体信息，但您还需要阅读 A、B、C 和 F。如果您正在处理非常大的文件（我们曾经是），您也必须使用 Zip64 的东西 - 但对于 24 MB，您没问题。
如果您想深入了解并尝试一下 - 如果您遇到问题，请回帖，我会看看能否提供一些建议。
【讨论】：
-
我在处理多卷 zip 文件时遇到问题。特别是当单个文件组件在多个磁盘文件之间拆分时。在 file.zx01 我有文件头和压缩数据的第一部分，然后在 file.zx02 我有其余的压缩数据。但是由于某种原因我无法重新组装文件，我不确定为什么。你在这里有什么经验吗？
为了它的价值，我喜欢在任何地方使用 try-with-resources。如果你喜欢这种设计模式，那么你会喜欢这个。此外，如果条目大于所需的零件尺寸，这也解决了空零件的问题。在最坏的情况下，您将至少拥有与条目一样多的部分。
在：
我的存档.zip
输出：
my-archive.part1of3.zip
我的存档.part2of3.zip
my-archive.part3of3.zip
注意：我正在使用日志记录和 Apache Commons FilenameUtils，但请随意使用您工具包中的内容。
/**
* Utility class to split a zip archive into parts (not volumes)
* by attempting to fit as many entries into a single part before
* creating a new part. If a part would otherwise be empty because
* the next entry won't fit, it will be added anyway to avoid empty parts.
*
* @author Eric Draken, 2019
*/
public class Zip
{
private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;
private static final String ZIP_PART_FORMAT = "%s.part%dof%d.zip";
private static final String EXT = "zip";
private static final Logger logger = LoggerFactory.getLogger( MethodHandles.lookup().lookupClass() );
/**
* Split a large archive into smaller parts
*
* @param zipFile Source zip file to split (must end with .zip)
* @param outZipFile Destination zip file base path. The "part" number will be added automatically
* @param approxPartSizeBytes Approximate part size
* @throws IOException Exceptions on file access
*/
public static void splitZipArchive(
@NotNull final File zipFile,
@NotNull final File outZipFile,
final long approxPartSizeBytes ) throws IOException
{
String basename = FilenameUtils.getBaseName( outZipFile.getName() );
Path basePath = outZipFile.getParentFile() != null ? // Check if this file has a parent folder
outZipFile.getParentFile().toPath() :
Paths.get( "" );
String extension = FilenameUtils.getExtension( zipFile.getName() );
if ( !extension.equals( EXT ) )
{
throw new IllegalArgumentException( "The archive to split must end with ." + EXT );
}
// Get a list of entries in the archive
try ( ZipFile zf = new ZipFile( zipFile ) )
{
// Silliness check
long minRequiredSize = zipFile.length() / 100;
if ( minRequiredSize > approxPartSizeBytes )
{
throw new IllegalArgumentException(
"Please select a minimum part size over " + minRequiredSize + " bytes, " +
"otherwise there will be over 100 parts."
);
}
// Loop over all the entries in the large archive
// to calculate the number of parts required
Enumeration<? extends ZipEntry> enumeration = zf.entries();
long partSize = 0;
long totalParts = 1;
while ( enumeration.hasMoreElements() )
{
long nextSize = enumeration.nextElement().getCompressedSize();
if ( partSize + nextSize > approxPartSizeBytes )
{
partSize = 0;
totalParts++;
}
partSize += nextSize;
}
// Silliness check: if there are more parts than there
// are entries, then one entry will occupy one part by contract
totalParts = Math.min( totalParts, zf.size() );
logger.debug( "Split requires {} parts", totalParts );
if ( totalParts == 1 )
{
// No splitting required. Copy file
Path outFile = basePath.resolve(
String.format( ZIP_PART_FORMAT, basename, 1, 1 )
);
Files.copy( zipFile.toPath(), outFile );
logger.debug( "Copied {} to {} (pass-though)", zipFile.toString(), outFile.toString() );
return;
}
// Reset
enumeration = zf.entries();
// Split into parts
int currPart = 1;
ZipEntry overflowZipEntry = null;
while ( overflowZipEntry != null || enumeration.hasMoreElements() )
{
Path outFilePart = basePath.resolve(
String.format( ZIP_PART_FORMAT, basename, currPart++, totalParts )
);
overflowZipEntry = writeEntriesToPart( overflowZipEntry, zf, outFilePart, enumeration, approxPartSizeBytes );
logger.debug( "Wrote {}", outFilePart );
}
}
}
/**
* Write an entry to the to the outFilePart
*
* @param overflowZipEntry ZipEntry that didn't fit in the last part, or null
* @param inZipFile The large archive to split
* @param outFilePart The part of the archive currently being worked on
* @param enumeration Enumeration of ZipEntries
* @param approxPartSizeBytes Approximate part size
* @return Overflow ZipEntry, or null
* @throws IOException File access exceptions
*/
private static ZipEntry writeEntriesToPart(
@Nullable ZipEntry overflowZipEntry,
@NotNull final ZipFile inZipFile,
@NotNull final Path outFilePart,
@NotNull final Enumeration<? extends ZipEntry> enumeration,
final long approxPartSizeBytes
) throws IOException
{
try (
ZipOutputStream zos =
new ZipOutputStream( new FileOutputStream( outFilePart.toFile(), false ) )
)
{
long partSize = 0;
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
while ( overflowZipEntry != null || enumeration.hasMoreElements() )
{
ZipEntry entry = overflowZipEntry != null ? overflowZipEntry : enumeration.nextElement();
overflowZipEntry = null;
long entrySize = entry.getCompressedSize();
if ( partSize + entrySize > approxPartSizeBytes )
{
if ( partSize != 0 )
{
return entry; // Finished this part, but return the dangling ZipEntry
}
// Add the entry anyway if the part would otherwise be empty
}
partSize += entrySize;
zos.putNextEntry( entry );
// Get the input stream for this entry and copy the entry
try ( InputStream is = inZipFile.getInputStream( entry ) )
{
int bytesRead;
while ( (bytesRead = is.read( buffer )) != -1 )
{
zos.write( buffer, 0, bytesRead );
}
}
}
return null; // Finished splitting
}
}
【讨论】：
下面的代码是我根据所需大小将目录结构中的 zip 文件拆分为块的解决方案。我发现以前的答案很有用，所以想用类似但更简洁的方法做出贡献。这段代码可以满足我的特定需求，我相信还有改进的余地。
import java.io.*;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;
class ChunkedZip {
private final static long MAX_FILE_SIZE = 1000 * 1000 * 1024; // around 1GB
private final static String zipCopyDest = "C:\\zip2split\\copy";
public static void splitZip(String zipFileName, String zippedPath, String coreId) throws IOException {
System.out.println("process whole zip file..");
FileInputStream fis = new FileInputStream(zippedPath);
ZipInputStream zipInputStream = new ZipInputStream(fis);
ZipEntry entry = null;
int currentChunkIndex = 0;
//using just to get the uncompressed size of the zipentries
long entrySize = 0;
ZipFile zipFile = new ZipFile(zippedPath);
Enumeration enumeration = zipFile.entries();
String copDest = zipCopyDest + "\\" + coreId + "_" + currentChunkIndex + ".zip";
FileOutputStream fos = new FileOutputStream(new File(copDest));
BufferedOutputStream bos = new BufferedOutputStream(fos);
ZipOutputStream zos = new ZipOutputStream(bos);
long currentSize = 0;
try {
while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {
ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
System.out.println(zipEntry.getName());
System.out.println(zipEntry.getSize());
entrySize = zipEntry.getSize();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
//long entrySize = entry.getCompressedSize();
//entrySize = entry.getSize(); //gives -1
if ((currentSize + entrySize) > MAX_FILE_SIZE) {
zos.close();
//construct a new stream
//zos = new ZipOutputStream(new FileOutputStream(new File(zippedPath, constructCurrentPartName(coreId))));
currentChunkIndex++;
zos = getOutputStream(currentChunkIndex, coreId);
currentSize = 0;
} else {
currentSize += entrySize;
zos.putNextEntry(new ZipEntry(entry.getName()));
byte[] buffer = new byte[8192];
int length = 0;
while ((length = zipInputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, length);
}
byte[] unzippedFile = outputStream.toByteArray();
zos.write(unzippedFile);
unzippedFile = null;
outputStream.close();
zos.closeEntry();
}
//zos.close();
}
} finally {
zos.close();
}
}
public static ZipOutputStream getOutputStream(int i, String coreId) throws IOException {
System.out.println("inside of getOutputStream()..");
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipCopyDest + "\\" + coreId + "_" + i + ".zip"));
// out.setLevel(Deflater.DEFAULT_COMPRESSION);
return out;
}
public static void main(String args[]) throws IOException {
String zipFileName = "Large_files_for_testing.zip";
String zippedPath = "C:\\zip2split\\Large_files_for_testing.zip";
String coreId = "Large_files_for_testing";
splitZip(zipFileName, zippedPath, coreId);
}
}
【讨论】：
这是我的解决方案：
public abstract class ZipHelper {
public static NumberFormat formater = NumberFormat.getNumberInstance(new Locale("pt", "BR"));
public static List<Path> zip(Collection<File> inputFiles, long maxSize) throws IOException {
byte[] buffer = new byte[1024];
int count = 0;
long currentZipSize = maxSize;
List<Path> response = new ArrayList<>();
ZipOutputStream zip = null;
for (File currentFile : inputFiles) {
long nextFileSize = currentFile.length();
long predictedZipSize = currentZipSize + nextFileSize;
boolean needNewFile = predictedZipSize >= maxSize;
System.out.println("[=] ZIP current (" + formater.format(currentZipSize) + ") + next file (" + formater.format(nextFileSize) + ") = predicted (" + formater.format(predictedZipSize) + ") > max (" + formater.format(maxSize) + ") ? " + needNewFile);
if (needNewFile) {
safeClose(zip);
Path tmpFile = Files.createTempFile("teste-", (".part." + count++ + ".zip"));
System.out.println("[#] Starting new file: " + tmpFile);
zip = new ZipOutputStream(Files.newOutputStream(tmpFile));
zip.setLevel(Deflater.BEST_COMPRESSION);
response.add(tmpFile);
currentZipSize = 0;
}
ZipEntry zipEntry = new ZipEntry(currentFile.getName());
System.out.println("[<] Adding to ZIP: " + currentFile.getName());
zip.putNextEntry(zipEntry);
FileInputStream in = new FileInputStream(currentFile);
zip.write(in.readAllBytes());
zip.closeEntry();
safeClose(in);
long compressed = zipEntry.getCompressedSize();
System.out.println("[=] Compressed current file: " + formater.format(compressed));
currentZipSize += zipEntry.getCompressedSize();
}
safeClose(zip);
return response;
}
public static void safeClose(Closeable... closeables) {
if (closeables != null) {
for (Closeable closeable : closeables) {
if (closeable != null) {
try {
System.out.println("[X] Closing: (" + closeable.getClass() + ") - " + closeable);
closeable.close();
} catch (Throwable ex) {
System.err.println("[!] Error on close: " + closeable);
ex.printStackTrace();
}
}
}
}
}
}
还有控制台输出：
[?] Files to process: [\data\teste\TestFile(1).pdf, \data\teste\TestFile(2).pdf, \data\teste\TestFile(3).pdf, \data\teste\TestFile(4).pdf, \data\teste\TestFile(5).pdf, \data\teste\TestFile(6).pdf, \data\teste\TestFile(7).pdf]
[=] ZIP current (3.145.728) + next file (1.014.332) = predicted (4.160.060) > max (3.145.728) ? true
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip
[<] Adding to ZIP: TestFile(1).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3d99d22e
[=] Compressed current file: 940.422
[=] ZIP current (940.422) + next file (1.511.862) = predicted (2.452.284) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(2).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@49fc609f
[=] Compressed current file: 1.475.178
[=] ZIP current (2.415.600) + next file (2.439.287) = predicted (4.854.887) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@cd2dae5
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip
[<] Adding to ZIP: TestFile(3).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@4973813a
[=] Compressed current file: 2.374.718
[=] ZIP current (2.374.718) + next file (2.385.447) = predicted (4.760.165) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@6321e813
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip
[<] Adding to ZIP: TestFile(4).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@79be0360
[=] Compressed current file: 2.202.203
[=] ZIP current (2.202.203) + next file (292.918) = predicted (2.495.121) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(5).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@22a67b4
[=] Compressed current file: 230.491
[=] ZIP current (2.432.694) + next file (4.197.512) = predicted (6.630.206) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@57855c9a
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip
[<] Adding to ZIP: TestFile(6).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3b084709
[=] Compressed current file: 3.020.115
[=] ZIP current (3.020.115) + next file (1.556.237) = predicted (4.576.352) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@3224f60b
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip
[<] Adding to ZIP: TestFile(7).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@63e2203c
[=] Compressed current file: 1.460.566
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@1efed156
[>] Generated ZIP files(s): [C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip, C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip, C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip, C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip, C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip]
【讨论】：