【问题标题】:Deflater entropy issueDeflater熵问题
【发布时间】:2017-04-10 14:48:59
【问题描述】:

我正在使用Deflater 通过网络发送一些消息。

我正在使用:

public static byte[] compress(byte[] data,int level) throws IOException {

     Instant starts = Instant.now();

     Deflater deflater = new Deflater();  
     deflater.setInput(data);  
     deflater.setLevel(/*level*/9);

     ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length);   
     deflater.finish();  

     byte[] buffer = new byte[1024];   
     while (!deflater.finished()) {  
          int count = deflater.deflate(buffer); // returns the generated code... index  
          outputStream.write(buffer, 0, count);   
     }  
     outputStream.close();  
     byte[] output = outputStream.toByteArray();  
     System.out.println("Original: " + data.length / 1024 + " Kb");  
     System.out.println("Compressed: " + output.length / 1024 + " Kb");  

     Instant ends = Instant.now();
     System.out.println("deflating time:"+Duration.between(starts, ends));

     return output;  
}   

压缩一个通用字节数组,

和:

msg.setDataString(Base64.getEncoder().encodeToString(data));

将对象嵌入到 xml 中。

一段时间内一切正常,但今天使用我看到的调试器:

msg.dataString

(这是一组几乎是空的图像,作为 rgb 像素(3 字节)数组)

eNrt3e1i4jYQBdCR3/+hu+XTYM1I2IEku+fUbbdRDARMf1wmVxEAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAPCXan/++v/Iltvp6J8Z9bkAAAAAAPArtc1x/ur1z8/r58S8ncTmuP+7v9qKc8+n
3+/n+Rj9GAAAAAAA8HHt/odNrn2KuzP16jVKn7FsjqGojqiOMFcPAAAAAEClSJHLBPo2Ut7J
2x/n1F8N05fXg/bJYzaN7y4O0ngAAAAAAP5hRWBczoJHlU2fvrhEHa1fAvBuJr5c/2vZHO36
z+y4rG5PvX5D/Dk697rc8/beY4rxbHy5CgAAAADAL9dmJth7A9tVWn5thMmmxidmzR9D8Ids
e1mu4fombV9Oi2edzPx84mV5m7cvSyy9KP56W3H6hshn44vPEGJutYzizc0DAAAAAPxUtx1L
k0LycoR9ULBet7QsVRq/TETxS8c1he9rvZh98rjcczZ0P/qIYNSOE3nwPlEcf3stpfEAAAAA
AG9Vpq/5hPUgUp/ctXRYGdPL29uqH2Y7pN725u39JpnJvH0pEvWHfpt95fD1M51/7BFa4wEA
AAAAvkwZuLb8u+JWLJ7kvFHlvIcS9Wq+fXnoY++MkdfJ9mBGvU7Us2L4ien3slU+b6QfpPGD
Lvxhj/5l91phPAAAAADARZWSpuXfqyH1Tt5elsKst/kcjbfvG+ZOB8GrHU8Hafh6R9TOfPty
oFKm7n4vem4Gsfr4CYvRavSn4K9V+tHs0QoAAAAAsJpDz4fYW/ZdrdrXNKodPGdi3qm8vc7i
u6u3vVDTspm8UKYVM+qrID9to0kaY5aJufrBBPvM5rGd1WhHmnvyYvhrGU111QEAAAAA/Ep5
pF6F7ecZ9SRvP082L3FKbV/sfDmumuiu91Ldn7fXvTKrr2Z5fLa6erj9+faHR/98twd+FyCO
Z+0Tv7/Q2Q1X5g4AAAAA/FK3ALSTtterLS0UmdjUNPauRvtuWXJdz54XI+pFkU0b5+1TI+y9
Bpx3mauhyfqCRqevLkFNNQAAAADAh9X55GaLy1Vj+72Mu3ezt9VtGN/enebuy2vjQM5f5+1x
y8NbMTy/7Jv2Hybk1cz+O36T4N0vcH4VRUQrd+mt9/AFAAAAAP51bbDaBhO/STnMbSl6Q+rx
1PKxLf5IK7qPZ+b1yTun32OJj6TIO5rj39iu83lRPpkxlbe3yLbaXUXv2xqj+3p3D9+2Ljx6
upzv7wYAAAAA4O9Qz+UWm5a20Q0OTu6Gm9ns+331SF1MnbdHfnzX5PXuTwGO9Oesb7nzVKzW
OgH0tzxTh+73C57F+ncf0tViw4LLWykpVwIAAAAAfprdLRltbrV7j6th387JRej9mLe/GoxH
Of883/Bd32/1kF7dPfTrYuYqM09z4OTcde/K6DOR7rlvCeSjrBSKGP24R5/ifb83UV110Qbv
hZC9AwAAAMBnHemdbhPHy3fcZufb66R3V+59sGV9ZoZ5x/0eiZ9jVEh+K1rp5N7lapWox/A1
SpL1Ud4e4zb85O4vgXuyl+7h3Lvt+wwoDr2C1d4E6+e5/+6TxgMAAADAV6szt5m8/dXbnL/j
PL1Mht8nAvmsi6bNnbivFWRiSnl3IczOo/7Z00rxzVOfp/H1ZyLFK5h+1nL5O6n+v9z+5fpJ
utKTkvVRu/v8tbFjdbKnqLsUxdVzf0KzR5W9u5scHgAAAAB+jDaceG/5qa/Nz28z2Zmkd8dU
9nSzR5WK93PgfEp5kORHGdXGTKI+0fvT2yD0+aXqvopteJlk10b+mG4z6hO39nxhPO5S2t/3
dLT66mciUWbmT5utvrzH7+xs/MufiQx3SZj734BWeQD49f4DmhUpMw==

由于缺乏熵真的很可怕,显然我做错了什么来得到这个结果,而且我在浪费带宽,或者“A”的序列是有效信息.但这太可怕了。

【问题讨论】:

    标签: java base64 compression deflate


    【解决方案1】:

    那里发生的情况是,deflate 格式的压缩能力对于数据的第一部分已经饱和。 deflate 中可能的最大压缩比为 1032:1。

    在压缩后的数据中,前 1,083,300 个字节全部为零。几乎所有这些都压缩到大约 1,000 个零字节。

    一切都好。你没有做错什么。不用担心。

    【讨论】:

    • 谢谢,所以基本上它超过 9000 XD ...我每秒最多发送 5 个数据包,并且 45k/s 的优化似乎是必须的.​​..我对此进行研究。您可以告诉任何方法来发现和删除无用的序列?也许我应该选择另一种压缩策略?
    • 如果您希望发送大量这些空图像,那么您可以简单地再次压缩它。这 2,416 字节在第二次压缩时压缩到 1,403 字节。
    • 会不会被 1032 倍压缩到几个字节?你会建议 9 级放气和 1 级放气吗?
    • 这一系列 1000 个零 实际上压缩到几个字节。但是还有另外 1400 字节的压缩数据,第二次完全没有压缩。
    • 第二次压缩可以是 1 级,它应该采用简单的运行长度编码。只需对关卡进行试验,看看哪种最适合您。
    猜你喜欢
    • 2013-12-17
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2022-11-20
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多