【问题标题】:Out of memory exception when decrypt large file using Cipher使用 Cipher 解密大文件时出现内存不足异常
【发布时间】:2020-08-30 16:52:11
【问题描述】:

我试图使用 javax.crypto 下的类和用于输入/输出的文件流来实现加密/解密程序。 为了限制内存使用,我使用 -Xmx256m 参数运行。

它适用于较小文件的加密和解密。 但是当解密一个大文件(1G大小)时,会出现内存不足的异常:

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)
    at com.sun.crypto.provider.CipherCore.update(CipherCore.java:782)
    at com.sun.crypto.provider.CipherCore.update(CipherCore.java:667)
    at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
    at javax.crypto.Cipher.update(Cipher.java:1831)
    at javax.crypto.CipherOutputStream.write(CipherOutputStream.java:166)

这是解密代码:

private final int _readSize = 0x10000;//64k

...

GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(gcmTagSize, iv);
Key keySpec = new SecretKeySpec(key, keyParts[0]);
Cipher decCipher = Cipher.getInstance("AES/GCM/PKCS5Padding");

decCipher.init(Cipher.DECRYPT_MODE, keySpec, gcmParameterSpec);

try (InputStream fileInStream = Files.newInputStream(inputEncryptedFile);
    OutputStream fileOutStream = Files.newOutputStream(outputDecryptedFile)) {
    try (CipherOutputStream cipherOutputStream = new CipherOutputStream(fileOutStream, decCipher)) {
        long count = 0L;
        byte[] buffer = new byte[_readSize];

        int n;
        for (; (n = fileInStream.read(buffer)) != -1; count += (long) n) {
            cipherOutputStream.write(buffer, 0, n);
        }
    }
}

gcmTagSize 和 iv 等关键参数是从密钥文件中读取的,它适用于较小的文件,例如大小约为 50M 的文件。

据我了解,每次只有 64k 数据传递给解密,为什么它会耗尽堆内存? 我怎样才能避免这种情况?

编辑:

其实我试过用 4k 作为缓冲区大小,同样的异常失败了。

编辑 2:

通过更多测试,它可以处理的最大文件大小约为堆大小的 1/4。比如设置-Xmx256m,大​​于64M的文件将无法解密。

【问题讨论】:

  • 将读取大小减少到更合理的值,例如 16K 甚至 4K。
  • 4k 因同样的异常而失败。

标签: java exception encryption out-of-memory heap-memory


【解决方案1】:

这似乎是 GCM 模式实施的问题。我不确定您是否可以解决它。

如果您查看堆栈跟踪:

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at com.sun.crypto.provider.GaloisCounterMode.decrypt(GaloisCounterMode.java:505)

GaloisCounterMode 内写入ByteArrayOutputStream 时发生内存不足错误。你使用了FileOutputStream,所以要么你没有显示正确的代码,要么这个ByteArrayStream在内部使用。

如果您查看source for GaloisCounterMode,您会发现它定义了一个内部ByteArrayOutputStream(它实际上定义了两个,但我认为这是问题所在):

    // buffer for storing input in decryption, not used for encryption
    private ByteArrayOutputStream ibuffer = null;

然后,稍后,它将字节写入此流。注意代码注释。

    int decrypt(byte[] in, int inOfs, int len, byte[] out, int outOfs) {
        processAAD();

        if (len > 0) {
            // store internally until decryptFinal is called because
            // spec mentioned that only return recovered data after tag
            // is successfully verified
            ibuffer.write(in, inOfs, len);
        }
        return 0;
    }

直到decryptFinal(),该缓冲区才会重置。


编辑:看着this CSx answer,GCM 似乎需要缓冲整个流。如果您有大文件且内存不足,那将是一个非常糟糕的选择。

我认为您最好的解决方案是切换到 CBC 模式。

【讨论】:

  • 你有"buried the lead"。您的编辑应该在答案的顶部。
  • @PresidentJamesK.Polk - 我希望 OP 能够理解他们的异常,而不是立即跳到答案。
【解决方案2】:

坏消息是:恕我直言,该错误是由原生 Java 中 AES GCM 模式的错误实现引起的。即使您可以让它工作,您也会发现解密一个大文件(1 GB 左右)将花费大量时间(可能是几个小时?)。 但也有好消息:您可以/应该使用 BouncyCastle 作为解密任务的服务提供商 - 这样解密将起作用并且速度更快。

以下完整示例将创建一个 1 GB 大小的示例文件,使用 BouncyCastle 对其进行加密,然后对其进行解密。最后有一个文件比较显示明文和解密文件内容相等,文件将被删除。 您的设备上需要临时总共超过 3 GB 的可用空间才能运行此示例。

使用 64 KB 的缓冲区,我正在使用以下数据运行此示例:

加密毫秒数:14295 |解密:16249

1 KB 的缓冲区在加密方面有点慢,但在解密任务上要慢得多:

加密毫秒数:15250 |解密:21952

关于您的密码的最后一句话 - “AES/GCM/PKCS5Padding”在某些实现中不存在且“可用”,但实际使用的算法是“AES/GCM/NoPadding”(有关更多详细信息,请参阅Can PKCS5Padding be in AES/GCM mode?)。

import org.bouncycastle.jce.provider.BouncyCastleProvider;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.nio.file.Files;
import java.security.*;
import java.util.Arrays;

public class GcmTestBouncyCastle {
    public static void main(String[] args) throws IOException, NoSuchPaddingException, InvalidAlgorithmParameterException,
            NoSuchAlgorithmException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException, InvalidKeyException {
        System.out.println("Encryption & Decryption with BouncyCastle AES-GCM-Mode");
        System.out.println("https://stackoverflow.com/questions/61792534/out-of-memory-exception-when-decrypt-large-file-using-cipher");
        // you need bouncy castle, get version 1.65 here:
        // https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on/1.65
        Security.addProvider(new BouncyCastleProvider());
        // setup files
        // filenames
        String filenamePlain = "plain.dat";
        String filenameEncrypt = "encrypt.dat";
        String filenameDecrypt = "decrypt.dat";
        // generate a testfile of 1024 byte | 1 gb
        //createFileWithDefinedLength(filenamePlain, 1024);
        createFileWithDefinedLength(filenamePlain, 1024 * 1024 * 1024); // 1 gb
        // time measurement
        long startMilli = 0;
        long encryptionMilli = 0;
        long decryptionMilli = 0;
        // generate nonce/iv
        int GCM_NONCE_LENGTH = 12; // for a nonce of 96 bit length
        int GCM_TAG_LENGTH = 16;
        int GCM_KEY_LENGTH = 32; // 32 = 256 bit keylength, 16 = 128 bit keylength
        SecureRandom r = new SecureRandom();
        byte[] nonce = new byte[GCM_NONCE_LENGTH];
        r.nextBytes(nonce);
        // key should be generated as random byte[]
        byte[] key = new byte[GCM_KEY_LENGTH];
        r.nextBytes(key);
        // encrypt file
        startMilli = System.currentTimeMillis();
        encryptWithGcmBc(filenamePlain, filenameEncrypt, key, nonce, GCM_TAG_LENGTH);
        encryptionMilli = System.currentTimeMillis() - startMilli;
        startMilli = System.currentTimeMillis();
        decryptWithGcmBc(filenameEncrypt, filenameDecrypt, key, nonce, GCM_TAG_LENGTH);
        decryptionMilli = System.currentTimeMillis() - startMilli;
        // check that plain and decrypted files are equal
        System.out.println("SHA256-file compare " + filenamePlain + " | " + filenameDecrypt + " : "
                + Arrays.equals(sha256File(filenamePlain), sha256File(filenameDecrypt)));
        System.out.println("Milliseconds for Encryption: " + encryptionMilli + " | Decryption: " + decryptionMilli);
        // clean up with files
        Files.deleteIfExists(new File(filenamePlain).toPath());
        Files.deleteIfExists(new File(filenameEncrypt).toPath());
        Files.deleteIfExists(new File(filenameDecrypt).toPath());
    }

    public static void encryptWithGcmBc(String filenamePlain, String filenameEnc, byte[] key, byte[] nonce, int gcm_tag_length)
            throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
            InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
        Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
        SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
        GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
        cipher.init(Cipher.ENCRYPT_MODE, keySpec, gcmSpec);

        try (FileInputStream fis = new FileInputStream(filenamePlain);
             BufferedInputStream in = new BufferedInputStream(fis);
             FileOutputStream out = new FileOutputStream(filenameEnc);
             BufferedOutputStream bos = new BufferedOutputStream(out)) {
            //byte[] ibuf = new byte[1024];
            byte[] ibuf = new byte[0x10000]; // = 65536
            int len;
            while ((len = in.read(ibuf)) != -1) {
                byte[] obuf = cipher.update(ibuf, 0, len);
                if (obuf != null)
                    bos.write(obuf);
            }
            byte[] obuf = cipher.doFinal();
            if (obuf != null)
                bos.write(obuf);
        }
    }

    public static void decryptWithGcmBc(String filenameEnc, String filenameDec, byte[] key, byte[] nonce, int gcm_tag_length)
            throws IOException, NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException,
            InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException {
        try (FileInputStream in = new FileInputStream(filenameEnc);
             FileOutputStream out = new FileOutputStream(filenameDec)) {
            //byte[] ibuf = new byte[1024];
            byte[] ibuf = new byte[0x10000]; // = 65536
            int len;
            Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
            SecretKeySpec keySpec = new SecretKeySpec(key, "AES");
            GCMParameterSpec gcmSpec = new GCMParameterSpec(gcm_tag_length * 8, nonce);
            cipher.init(Cipher.DECRYPT_MODE, keySpec, gcmSpec);
            while ((len = in.read(ibuf)) != -1) {
                byte[] obuf = cipher.update(ibuf, 0, len);
                if (obuf != null)
                    out.write(obuf);
            }
            byte[] obuf = cipher.doFinal();
            if (obuf != null)
                out.write(obuf);
        }
    }

    // just for creating a large file within seconds
    private static void createFileWithDefinedLength(String filenameString, long sizeLong) throws IOException {
        RandomAccessFile raf = new RandomAccessFile(filenameString, "rw");
        try {
            raf.setLength(sizeLong);
        } finally {
            raf.close();
        }
    }

    // just for file comparing
    public static byte[] sha256File(String filenameString) throws IOException, NoSuchAlgorithmException {
        byte[] buffer = new byte[8192];
        int count;
        MessageDigest md = MessageDigest.getInstance("SHA-256");
        BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filenameString));
        while ((count = bis.read(buffer)) > 0) {
            md.update(buffer, 0, count);
        }
        bis.close();
        return md.digest();
    }
}

【讨论】:

  • 嗨,Kai,很高兴听到此解决方案对您有用。您能否将此标记为已接受的答案?谢谢。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2011-11-04
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2012-05-23
相关资源
最近更新 更多