处理 xml 文件时的 UTF8 编码无效答案

【问题标题】：Invalid UTF8 encoding on processing xml file处理 xml 文件时的 UTF8 编码无效
【发布时间】：2019-07-18 23:30:05
【问题描述】：

我有一个处理 XML 文件以读取一些值的 Java 代码。我收到一个错误：Invalid UTF8 encoding，我试图将文件内容复制到 NotePad++ 上的另一个文件中，该过程运行良好，但如果我只将文件另存为其他名称，则会给出相同的错误. 抱歉，我不能把我的 XML 文件放在这里，因为它太大了，我只会放 header 和 trailer。感谢您提供任何帮助来解决此错误。我处理 xml 文件的 java 代码：

XPathFactory f=XPathFactory.newInstance();
    XPath x=f.newXPath();

    InputSource source=new InputSource(new FileInputStream("C:\\Users\\cc\\eclipse-workspace\\data\\file.xml") );
    InputSource source2=new InputSource(new FileInputStream("C:\\Users\\cc\\eclipse-workspace\\data\\file.xml") );

    XPathExpression trlr=x.compile("pers/trailer/text()");
    XPathExpression hdr=x.compile("pers/header/CD/text()");

    String s=trlr.evaluate(source);
    String s2=hdr.evaluate(source2);
    System.out.println("header :"+s+" trailer"+s2);

pers 是 xml 文件中的根标签：

XML 文件如下所示：

<?xml version = '1.0' encoding = 'UTF-8'?>
<pers>
 <header>555</header>
 .
 .
 .
 .
 <trailer>666</trailer>

</pers>

堆栈跟踪：

java.io.UTFDataFormatException: Invalid UTF8 encoding.
    at oracle.xml.parser.v2.XMLUTF8Reader.checkUTF8Byte(XMLUTF8Reader.java:229)
    at oracle.xml.parser.v2.XMLUTF8Reader.readUTF8Char(XMLUTF8Reader.java:274)
    at oracle.xml.parser.v2.XMLUTF8Reader.fillBuffer(XMLUTF8Reader.java:189)
    at oracle.xml.parser.v2.XMLByteReader.saveBuffer(XMLByteReader.java:452)
    at oracle.xml.parser.v2.XMLReader.fillBuffer(XMLReader.java:2776)
    at oracle.xml.parser.v2.XMLReader.scanNameChars(XMLReader.java:1352)
    at oracle.xml.parser.v2.XMLReader.readQName(XMLReader.java:2149)
    at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1579)
    at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:448)
    at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:394)
    at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:236)
    at oracle.xml.jaxp.JXDocumentBuilder.parse(JXDocumentBuilder.java:175)
    at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:302)
    at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:332)
    at tasklets.HeaderFooter.execute(HeaderFooter.java:39)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272)
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128)
    at main.IncomeResponseFile.main(IncomeResponseFile.java:39)
--------------- linked to ------------------
javax.xml.xpath.XPathExpressionException: java.io.UTFDataFormatException: Invalid UTF8 encoding.
    at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:305)
    at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:332)
    at tasklets.HeaderFooter.execute(HeaderFooter.java:39)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272)
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128)
    at main.IncomeResponseFile.main(IncomeResponseFile.java:39)
Caused by: java.io.UTFDataFormatException: Invalid UTF8 encoding.
    at oracle.xml.parser.v2.XMLUTF8Reader.checkUTF8Byte(XMLUTF8Reader.java:229)
    at oracle.xml.parser.v2.XMLUTF8Reader.readUTF8Char(XMLUTF8Reader.java:274)
    at oracle.xml.parser.v2.XMLUTF8Reader.fillBuffer(XMLUTF8Reader.java:189)
    at oracle.xml.parser.v2.XMLByteReader.saveBuffer(XMLByteReader.java:452)
    at oracle.xml.parser.v2.XMLReader.fillBuffer(XMLReader.java:2776)
    at oracle.xml.parser.v2.XMLReader.scanNameChars(XMLReader.java:1352)
    at oracle.xml.parser.v2.XMLReader.readQName(XMLReader.java:2149)
    at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1579)
    at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:448)
    at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:394)
    at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:236)
    at oracle.xml.jaxp.JXDocumentBuilder.parse(JXDocumentBuilder.java:175)
    at com.sun.org.apache.xpath.internal.jaxp.XPathExpressionImpl.evaluate(XPathExpressionImpl.java:302)
    ... 23 more

【问题讨论】：

文件真的是UTF-8吗？有什么特殊字符吗？
我建议也发布堆栈跟踪。也许您有一些来自非 unicode 源的扩展字符，并且在将文本粘贴到 np++ 并保存新文件时以某种方式进行了转换？
@Mena ，我更新了堆栈跟踪
@ piet.t ，很难说，该文件有大约 100 000 条记录（每条记录），每条记录有 100 个字段，即数字和字符串。
没有什么是容易的，但我想你必须找到那个有问题的字符并通过十六进制编辑器检查它的编码。并始终牢记：仅仅因为 xml 文件在其标头中声称是 UTF-8，这不一定是正确的。

标签： java xml xslt spring-batch

【解决方案1】：

使用 java 编写脚本来检测有问题的行。

AtomicInteger lineno = new AtomicInteger();
Path path = Paths.get("... .xml");
Files.lines(path, StandardCharsets.ISO_8859_1)
    .forEach(line -> {
        int no = lineno.incrementAndGet();
        byte[] b = line.getBytes(StandardCharsets.ISO_8859_1);
        try {
            new String(b, StandardCharsets.UTF_8);
        } catch (Exception e) {
            System.out.printf("[%d] %s%n%s%n", no, line, e.getMessage());
            //throw new IllegalStateException(e);
        }
    });

人们可能会认为这是一个数据错误。

一般来说，它也可能是错误的缓冲读取：当一个多字节序列在缓冲区边界上被破坏时；然后可能会出现两个错误的半序列。在标准库代码中不太可能。

为了确保new String(...)的代码不会被JVM丢弃，可能：

int sowhat = Files.lines(path, StandardCharsets.ISO_8859_1)
    .mapToInt(line -> {
        int no = lineno.incrementAndGet();
        byte[] b = line.getBytes(StandardCharsets.ISO_8859_1);
        try {
            return new String(b, StandardCharsets.UTF_8).length();
        } catch (Exception e) {
            System.out.printf("[%d] %s%n%s%n", no, line, e.getMessage());
            throw new IllegalStateException(e); // Must throw or return int
        }
    }).sum();
System.out.println("Ignore this: " + sowhat);

人们可能会认为这是一个数据错误。

一般来说，它也可能是错误的缓冲读取：当一个多字节序列在缓冲区边界上被破坏时；然后可能会出现两个错误的半序列。在标准库代码中不太可能。

为了确保new String(...)的代码不会被JVM丢弃，可能：

int sowhat = Files.lines(path, StandardCharsets.ISO_8859_1)
    .mapToInt(line -> {
        int no = lineno.incrementAndGet();
        byte[] b = line.getBytes(StandardCharsets.ISO_8859_1);
        try {
            return new String(b, StandardCharsets.UTF_8).length();
        } catch (Exception e) {
            System.out.printf("[%d] %s%n%s%n", no, line, e.getMessage());
            throw new IllegalStateException(e); // Must throw or return int
        }
    }).sum();

非法 XML 字符（在 1.0 版中）？ [#x1-#x8] | [#xB-#xC] | [#xE-#x1F] | [#x7F-#x84] | [#x86-#x9F]

int sowhat = Files.lines(path, StandardCharsets.ISO_8859_1)
    .mapToInt(line -> {
        int no = lineno.incrementAndGet();
        byte[] b = line.getBytes(StandardCharsets.ISO_8859_1);
        if (!legal(b)) {
            System.out.printf("[%d] %s%n%s%n", no, line, e.getMessage());
            throw new IllegalStateException(e); // Must throw or return int
        }
    }).sum();

static boolean legal(byte[] bytes) {
    String s = new String(bytes, StandardCharsets.UTF_8);
    for (char ch : s.toCharArray()) {
        int x = ch;
        if ((0 <= x && x <= 8)               // ASCII control chars
                || (0xB <= x && x <= 0xC)
                || (0xE <= x && x <= 0x1F)
                || (0x7f <= x && x <= 0x84)  // DEL + Unicode control chars
                || (0x86 <= x && x <= 0x9F)) {
            return false;
        }
    }
    return true;
}

如果这不起作用，我已经让你足够久了。拆分文件并验证各个部分。

【讨论】：

感谢您的回答，我尝试了这个java代码，但没有给出任何错误。
也许，只是也许，JVM 会跳过 new String(...)。所以我添加了一个变体。堆栈跟踪似乎指向一个标签或更可能指向一个属性名称。在 XML 版本 1.0 中不应该有像 é 这样的非 ASCII 字母，所以这很奇怪。
我得到了这个：忽略这个：134087631，我不明白。这是什么意思？
只是它贯穿每一行并成功转换为 UTF-8。所以肯定还有另一个错误。存在非法的 XML 字符，可能错误具有误导性。
为明显的非法 XML 字符添加了代码，但出于其他几个原因，可以继续这种方式。也许它只是用户数据中的虚假< 或&#128;。

【解决方案2】：

我使用此代码将文件转换为 UTF-8 格式：

 File source = new File("C:\\Users\\cc\\eclipse-workspace\\data\\file.xml");
    String srcEncoding="ISO-8859-1";
    File target = new File("C:\\Users\\cc\\eclipse-workspace\\data\\file2.xml");
    String tgtEncoding="UTF-8";
      try (
        BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(source), srcEncoding));
        BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(target), tgtEncoding)); ) {
            char[] buffer = new char[16384];
            int read;
            while ((read = br.read(buffer)) != -1)
                bw.write(buffer, 0, read);

  }

在那之后我使用了 file2 它工作了。感谢：java: how to convert a file to utf8

【讨论】：