【问题标题】:What's the most standard Java way to store raw binary data along with XML?将原始二进制数据与 XML 一起存储的最标准 Java 方式是什么?
【发布时间】:2011-03-15 18:00:31
【问题描述】:

我需要将大量二进制数据存储到一个文件中,但我还想以 XML 格式读取/写入该文件的标题。

是的,我可以将二进制数据存储到某个 XML 值中,然后使用 base64 编码对其进行序列化。但这不会节省空间。

我能否以或多或少标准化的方式“混合”XML 数据和原始二进制数据?

我在考虑两个选项:

  • 有没有办法使用 JAXB 做到这一点?

  • 或者有没有办法获取一些现有的 XML 数据并将二进制数据附加到其中,从而识别边界?

  • 我正在寻找的概念不是被 / 用于 SOAP 吗?

  • 还是在电子邮件标准中使用? (二进制附件的分离)

我想要实现的方案:

[meta-info-about-boundary][XML-data][boundary][raw-binary-data]

谢谢!

【问题讨论】:

  • 由于您不能在 XML 文档中包含二进制零,因此您必须具有某种编码。为什么 Base64 不够好?

标签: java xml serialization xml-serialization jaxb


【解决方案1】:

您可以为此利用 AttachementMarshaller 和 AttachmentUnmarshaller。这是 JAXB/JAX-WS 用来将二进制内容作为附件传递的桥梁。您可以利用相同的机制来做您想做的事。


概念证明

下面是它的实现方式。这应该适用于任何 JAXB impl(它适用于我的 EclipseLink JAXB (MOXy) 和参考实现)。

消息格式

[xml_length][xml][attach1_length][attach1]...[attachN_length][attachN]

这是一个具有多个 byte[] 属性的对象。

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class Root {

    private byte[] foo;
    private byte[] bar;

    public byte[] getFoo() {
        return foo;
    }

    public void setFoo(byte[] foo) {
        this.foo = foo;
    }

    public byte[] getBar() {
        return bar;
    }

    public void setBar(byte[] bar) {
        this.bar = bar;
    }

}

演示

这个类用来演示MessageWriter和MessageReader是如何使用的:

import java.io.FileInputStream;
import java.io.FileOutputStream;
import javax.xml.bind.JAXBContext;

public class Demo {

    public static void main(String[] args) throws Exception {
        JAXBContext jc = JAXBContext.newInstance(Root.class);

        Root root = new Root();
        root.setFoo("HELLO WORLD".getBytes());
        root.setBar("BAR".getBytes());

        MessageWriter writer = new MessageWriter(jc);
        FileOutputStream outStream = new FileOutputStream("file.xml");
        writer.write(root, outStream);
        outStream.close();

        MessageReader reader = new MessageReader(jc);
        FileInputStream inStream = new FileInputStream("file.xml");
        Root root2 = (Root) reader.read(inStream);
        inStream.close();

        System.out.println(new String(root2.getFoo()));
        System.out.println(new String(root2.getBar()));
    }

}

MessageWriter

负责将消息写入所需的格式:

import java.io.ByteArrayOutputStream;
import java.io.ObjectOutputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.List;

import javax.activation.DataHandler;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.Marshaller;
import javax.xml.bind.attachment.AttachmentMarshaller;

public class MessageWriter {

    private JAXBContext jaxbContext;

    public MessageWriter(JAXBContext jaxbContext) {
        this.jaxbContext = jaxbContext;
    }

    /**
     * Write the message in the following format:
     * [xml_length][xml][attach1_length][attach1]...[attachN_length][attachN] 
     */
    public void write(Object object, OutputStream stream) {
        try {
            Marshaller marshaller = jaxbContext.createMarshaller();
            marshaller.setProperty(Marshaller.JAXB_FRAGMENT, true);
            BinaryAttachmentMarshaller attachmentMarshaller = new BinaryAttachmentMarshaller();
            marshaller.setAttachmentMarshaller(attachmentMarshaller);
            ByteArrayOutputStream xmlStream = new ByteArrayOutputStream();
            marshaller.marshal(object, xmlStream);
            byte[] xml = xmlStream.toByteArray();
            xmlStream.close();

            ObjectOutputStream messageStream = new ObjectOutputStream(stream);

            messageStream.write(xml.length); //[xml_length]
            messageStream.write(xml); // [xml]

            for(Attachment attachment : attachmentMarshaller.getAttachments()) {
                messageStream.write(attachment.getLength()); // [attachX_length]
                messageStream.write(attachment.getData(), attachment.getOffset(), attachment.getLength());  // [attachX]
            }

            messageStream.flush();
        } catch(Exception e) {
            throw new RuntimeException(e);
        }
    }

    private static class BinaryAttachmentMarshaller extends AttachmentMarshaller {

        private static final int THRESHOLD = 10;

        private List<Attachment> attachments = new ArrayList<Attachment>();

        public List<Attachment> getAttachments() {
            return attachments;
        }

        @Override
        public String addMtomAttachment(DataHandler data, String elementNamespace, String elementLocalName) {
            return null;
        }

        @Override
        public String addMtomAttachment(byte[] data, int offset, int length, String mimeType, String elementNamespace, String elementLocalName) {
            if(data.length < THRESHOLD) {
                return null;
            }
            int id = attachments.size() + 1;
            attachments.add(new Attachment(data, offset, length));
            return "cid:" + String.valueOf(id);
        }

        @Override
        public String addSwaRefAttachment(DataHandler data) {
            return null;
        }

        @Override
        public boolean isXOPPackage() {
            return true;
        }

    }

    public static class Attachment {

        private byte[] data;
        private int offset;
        private int length;

        public Attachment(byte[] data, int offset, int length) {
            this.data = data;
            this.offset = offset;
            this.length = length;
        }

        public byte[] getData() {
            return data;
        }

        public int getOffset() {
            return offset;
        }

        public int getLength() {
            return length;
        }

    }

}

消息阅读器

负责阅读消息:

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.ObjectInputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;

import javax.activation.DataHandler;
import javax.activation.DataSource;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.Unmarshaller;
import javax.xml.bind.attachment.AttachmentUnmarshaller;

public class MessageReader {

    private JAXBContext jaxbContext;

    public MessageReader(JAXBContext jaxbContext) {
        this.jaxbContext = jaxbContext;
    }

    /**
     * Read the message from the following format:
     * [xml_length][xml][attach1_length][attach1]...[attachN_length][attachN] 
     */
    public Object read(InputStream stream) {
        try {
            ObjectInputStream inputStream = new ObjectInputStream(stream);
            int xmlLength = inputStream.read();  // [xml_length]

            byte[] xmlIn = new byte[xmlLength]; 
            inputStream.read(xmlIn);  // [xml]

            BinaryAttachmentUnmarshaller attachmentUnmarshaller = new BinaryAttachmentUnmarshaller();
            int id = 1;
            while(inputStream.available() > 0) {
                int length = inputStream.read();  // [attachX_length]
                byte[] data = new byte[length];  // [attachX]
                inputStream.read(data);
                attachmentUnmarshaller.getAttachments().put("cid:" + String.valueOf(id++), data);
            }

            Unmarshaller unmarshaller = jaxbContext.createUnmarshaller();
            unmarshaller.setAttachmentUnmarshaller(attachmentUnmarshaller);
            ByteArrayInputStream byteInputStream = new ByteArrayInputStream(xmlIn);
            Object object = unmarshaller.unmarshal(byteInputStream);
            byteInputStream.close();
            inputStream.close();
            return object;
        } catch(Exception e) {
            throw new RuntimeException(e);
        }
    }

    private static class BinaryAttachmentUnmarshaller extends AttachmentUnmarshaller {

        private Map<String, byte[]> attachments = new HashMap<String, byte[]>();

        public Map<String, byte[]> getAttachments() {
            return attachments;
        }

        @Override
        public DataHandler getAttachmentAsDataHandler(String cid) {
            byte[] bytes = attachments.get(cid);
            return new DataHandler(new ByteArrayDataSource(bytes));
        }

        @Override
        public byte[] getAttachmentAsByteArray(String cid) {
            return attachments.get(cid);
        }

        @Override
        public boolean isXOPPackage() {
            return true;
        }

    }

    private static class ByteArrayDataSource implements DataSource {

        private byte[] bytes;

        public ByteArrayDataSource(byte[] bytes) {
            this.bytes = bytes;
        }

        public String getContentType() {
            return  "application/octet-stream";
        }

        public InputStream getInputStream() throws IOException {
            return new ByteArrayInputStream(bytes);
        }

        public String getName() {
            return null;
        }

        public OutputStream getOutputStream() throws IOException {
            return null;
        }

    }

}

更多信息

【讨论】:

  • 谢谢!我会看看这个。
  • 嗯,这些类是非常低级的。看来我还是得自己定义边界的处理。唯一的优点是与 JAXB 的兼容性。
  • 它们使您能够做的是将二进制数据存储在文档之外,并在文档中放置一个带有 ID 的占位符。这就是 JAX-WS (SOAP) 使用 JAXB 的方式。
  • 在您的示例中,您需要收集传递给附件编组器的字节 [],然后在编组操作后将它们写入输出流。
  • @java.is.for.desktop - 我已经用这种方法的概念证明更新了我的答案。
【解决方案2】:

我遵循了 Blaise Doughan 提出的概念,但没有附件编组器:

我让XmlAdapterbyte[] 转换为URI 引用并返回,而引用指向存储原始数据的单独文件。然后将 XML 文件和所有二进制文件放入一个 zip 文件中。

类似于OpenOffice的做法和ODF格式,其实就是一个带有少量XML和二进制文件的zip。

(在示例代码中,没有写入实际的二进制文件,也没有创建 zip。)

Bindings.java

import java.net.*;
import java.util.*;
import javax.xml.bind.annotation.*;
import javax.xml.bind.annotation.adapters.*;

final class Bindings {

  static final String SCHEME = "storage";
  static final Class<?>[] ALL_CLASSES = new Class<?>[]{
    Root.class, RawRef.class
  };

  static final class RawRepository
      extends XmlAdapter<URI, byte[]> {

    final SortedMap<String, byte[]> map = new TreeMap<>();
    final String host;
    private int lastID = 0;

    RawRepository(String host) {
      this.host = host;
    }

    @Override
    public byte[] unmarshal(URI o) {
      if (!SCHEME.equals(o.getScheme())) {
        throw new Error("scheme is: " + o.getScheme()
            + ", while expected was: " + SCHEME);
      } else if (!host.equals(o.getHost())) {
        throw new Error("host is: " + o.getHost()
            + ", while expected was: " + host);
      }

      String key = o.getPath();
      if (!map.containsKey(key)) {
        throw new Error("key not found: " + key);
      }

      byte[] ret = map.get(key);
      return Arrays.copyOf(ret, ret.length);
    }

    @Override
    public URI marshal(byte[] o) {
      ++lastID;
      String key = String.valueOf(lastID);
      map.put(key, Arrays.copyOf(o, o.length));

      try {
        return new URI(SCHEME, host, "/" + key, null);
      } catch (URISyntaxException ex) {
        throw new Error(ex);
      }
    }

  }

  @XmlRootElement
  @XmlType
  static final class Root {

    @XmlElement
    final List<RawRef> element = new LinkedList<>();
  }

  @XmlType
  static final class RawRef {

    @XmlJavaTypeAdapter(RawRepository.class)
    @XmlElement
    byte[] raw = null;
  }

}

Main.java

import java.io.*;
import javax.xml.bind.*;

public class _Run {

  public static void main(String[] args)
      throws Exception {
    JAXBContext context = JAXBContext.newInstance(Bindings.ALL_CLASSES);
    Marshaller marshaller = context.createMarshaller();
    marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
    Unmarshaller unmarshaller = context.createUnmarshaller();

    Bindings.RawRepository adapter = new Bindings.RawRepository("myZipVFS");
    marshaller.setAdapter(adapter);

    Bindings.RawRef ta1 = new Bindings.RawRef();
    ta1.raw = "THIS IS A STRING".getBytes();
    Bindings.RawRef ta2 = new Bindings.RawRef();
    ta2.raw = "THIS IS AN OTHER STRING".getBytes();

    Bindings.Root root = new Bindings.Root();
    root.element.add(ta1);
    root.element.add(ta2);

    StringWriter out = new StringWriter();
    marshaller.marshal(root, out);

    System.out.println(out.toString());
  }

}

输出

<root>
    <element>
        <raw>storage://myZipVFS/1</raw>
    </element>
    <element>
        <raw>storage://myZipVFS/2</raw>
    </element>
</root>

【讨论】:

    【解决方案3】:

    这不是 JAXB 本身支持的,因为您不希望将二进制数据序列化为 XML,但通常可以在使用 JAXB 时在更高级别完成。 我使用 Web 服务(SOAP 和 REST)的方法是使用 MIME 多部分/混合消息(检查multipart specification)。最初是为电子邮件设计的,非常适合发送带有二进制数据的 xml,并且大多数 Web 服务框架(例如 axis 或 jersey)都以几乎透明的方式支持它。

    这是一个使用带有jersey-multipart extension 的 Jersey 发送 XML 中的对象以及带有 REST Web 服务的二进制文件的示例。

    XML 对象

    @XmlRootElement
    public class Book {
       private String title;
       private String author;
       private int year;
    
       //getter and setters...
    }
    

    客户

    byte[] bin = some binary data...
    
    Book b = new Book();
    b.setAuthor("John");
    b.setTitle("wild stuff");
    b.setYear(2012);
    
    MultiPart multiPart = new MultiPart();
        multiPart.bodyPart(new BodyPart(b, MediaType.APPLICATION_XML_TYPE));
        multiPart.bodyPart(new BodyPart(bin, MediaType.APPLICATION_OCTET_STREAM_TYPE));
    
        response = service.path("rest").path("multipart").
                type(MultiPartMediaTypes.MULTIPART_MIXED).
                post(ClientResponse.class, multiPart);
    

    服务器

    @POST
    @Consumes(MultiPartMediaTypes.MULTIPART_MIXED)
    public Response post(MultiPart multiPart) {
        for(BodyPart part : multiPart.getBodyParts()) {
            System.out.println(part.getMediaType());
        }
    
        return Response.status(Response.Status.ACCEPTED).
                entity("Attachements processed successfully.").
                type(MediaType.TEXT_PLAIN).build();
    
    }
    

    我尝试发送一个 110917 字节的文件。使用wireshark,您可以看到数据是直接通过HTTP 发送的,如下所示:

    Hypertext Transfer Protocol
       POST /org.etics.test.rest.server/rest/multipart HTTP/1.1\r\n
       Content-Type: multipart/mixed; boundary=Boundary_1_353042220_1343207087422\r\n
       MIME-Version: 1.0\r\n
       User-Agent: Java/1.7.0_04\r\n
       Host: localhost:8080\r\n
       Accept: text/html, image/gif, image/jpeg\r\n
       Connection: keep-alive\r\n
       Content-Length: 111243\r\n
       \r\n
       [Full request URI: http://localhost:8080/org.etics.test.rest.server/rest/multipart]
    
       MIME Multipart Media Encapsulation, Type: multipart/mixed, Boundary: "Boundary_1_353042220_1343207087422"
         [Type: multipart/mixed]
         First boundary: --Boundary_1_353042220_1343207087422\r\n
            Encapsulated multipart part:  (application/xml)
            Content-Type: application/xml\r\n\r\n
            eXtensible Markup Language
              <?xml
              <book>
                <author>
                  John
                </author>
                <title>
                  wild stuff
                </title>
                <year>
                  2012
                </year>
              </book>
         Boundary: \r\n--Boundary_1_353042220_1343207087422\r\n
            Encapsulated multipart part:  (application/octet-stream)
            Content-Type: application/octet-stream\r\n\r\n
            Media Type
              Media Type: application/octet-stream (110917 bytes)
         Last boundary: \r\n--Boundary_1_353042220_1343207087422--\r\n
    

    如您所见,发送的二进制数据具有八位字节流,不会浪费空间,这与在 xml 中内联发送二进制数据时发生的情况相反。这只是开销非常低的 MIME 信封。 对于 SOAP,原理是相同的(只是它具有 SOAP 信封)。

    【讨论】:

      【解决方案4】:

      我不这么认为——XML 库通常不是为使用 XML+extra-data 而设计的。

      但是您可能能够摆脱像特殊流包装器这样简单的东西 - 它会公开一个包含“XML”的流和一个二进制流(来自特殊的“格式” )。然后 JAXB(或任何其他 XML 库)可以与“XML”流一起播放,并且二进制流保持独立。

      还要记住将“二进制”文件与“文本”文件考虑在内。

      编码愉快。

      【讨论】:

      • JAXB 提供 AttachmentMarshaller 和 AttachmentUnmarshaller 就是为了这个目的:stackoverflow.com/questions/5315968/…
      • @Blaise Doughan 我一直认为这些是用于处理编码的 MIME/base64 数据。为您 +1 :)
      猜你喜欢
      • 2014-01-23
      • 2016-11-03
      • 1970-01-01
      • 1970-01-01
      • 2012-03-30
      • 1970-01-01
      • 2011-11-18
      • 2017-04-10
      • 1970-01-01
      相关资源
      最近更新 更多