在 Google Cloud Dataflow 中使用 XmlSource 读取 XML 文件时出现 ClassCastException答案

【问题标题】：ClassCastException when reading XML file using XmlSource in Google Cloud Dataflow在 Google Cloud Dataflow 中使用 XmlSource 读取 XML 文件时出现 ClassCastException
【发布时间】：2017-01-10 05:33:50
【问题描述】：

我正在使用 XmlSource.from 读取存储在 Cloud Storage 存储分区中的 XML 文件。

 XmlSource<Data> source = XmlSource.<Data>from("gs://<my-url>/TestData.xml")
        .withRootElement("data")
        .withRecordElement("record")
        .withRecordClass(Data.class);

p.apply(Read.from(source))
        .apply(RemoveDuplicates.<Data>create())
        .apply(ParDo.of(new XMLPipeline.CreateItemQtyMapping()))
        .apply(Combine.<String, Integer>perKey(new SumIntegers()))
        .apply("FormatResults", MapElements.via(
                new SimpleFunction<KV<String, Integer>, String>() {
                  @Override
                  public String apply(KV<String, Integer> input) {
                    return input.getKey() + "," + input.getValue();
                  }
                }))
        .apply(TextIO.Write.to("gs://<my-url>.appspot.com/pos-pipeline-output/ItemCounts"));

p.run();

但我得到了这个例外：

017-01-09T14:01:31.107Z: Error:   (c88c756cabe0dbec): java.io.IOException: Failed to start reading from source: StaticValueProvider{value=gs://<my-url>/TestData.xml} range [48524, 97048)
at com.google.cloud.dataflow.sdk.runners.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:534)
at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:387)
at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:217)
at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:182)
at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:69)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:284)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:220)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:170)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.doWork(DataflowWorkerHarness.java:192)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:172)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:159)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException: com.sun.xml.internal.stream.XMLInputFactoryImpl cannot be cast to org.codehaus.stax2.XMLInputFactory2
    at com.google.cloud.dataflow.sdk.io.XmlSource$XMLReader.setUpXMLParser(XmlSource.java:490)
    at com.google.cloud.dataflow.sdk.io.XmlSource$XMLReader.startReading(XmlSource.java:356)
    at com.google.cloud.dataflow.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:528)
    at com.google.cloud.dataflow.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:281)
    at com.google.cloud.dataflow.sdk.runners.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:531)
    ... 14 more

这些是我的 pom.xml 中的依赖项：

<dependencies>
<dependency>
  <groupId>com.google.cloud.dataflow</groupId>
  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
  <version>1.9.0</version>
</dependency>

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-storage</artifactId>
  <version>0.7.0</version>
</dependency>

<dependency>
  <groupId>org.codehaus.woodstox</groupId>
  <artifactId>stax2-api</artifactId>
  <version>4.0.0</version>
</dependency>

我不确定这里有什么问题。有人可以指点一下吗？

谢谢，

阿布舍克

【问题讨论】：

这看起来可能是一个错误。我将深入挖掘，但您可以使用 SDK 1.8.0 解决。

标签： xml cloud google-cloud-dataflow dataflow gcp

【解决方案1】：

这有点微妙，但看起来您还需要包含适当的运行时依赖项。根据https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/io/XmlSource，你想：

显式声明对 org.codehaus.woodstox:stax2-api 的依赖
在运行时在类路径中包含兼容的实现，比如org.codehaus.woodstox:woodstox-core-asl

看起来你正确地完成了#1，但没有完成#2。

【讨论】：

【解决方案2】：

我来解决 java.lang.ClassCastException：com.sun.xml.internal.stream.XMLInputFactoryImpl 无法转换为 org.codehaus.stax2.XMLInputFactory2

答案是仅使用 org.codehaus.woodstox:woodstox.core.asl

的依赖项

已经间接依赖于 stax 和 stax2 (javax.xml.stream - stax-api, org.codehaus.woodstox - stax2-api)。

【讨论】：