Spark 抱怨 java.io.IOException: Null or empty fields is found

Question

我的执行者使用 stacktrace 抛出异常

java.io.IOException: Null or empty fields is found
    at org.apache.parquet.crypto.CryptoMetadataRetriever.getFileEncryptionProperties(CryptoMetadataRetriever.java:114)
    at org.apache.parquet.crypto.CryptoClassLoader.getFileEncryptionPropertiesOrNull(CryptoClassLoader.java:74)
    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:405)
    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:362)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:163)
    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:253)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:440)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1371)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:446)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

当我尝试通过以下方式保存到镶木地板文件时：

DataFrameWriter<Row> dfw =
        sparkSession.createDataFrame(javaSparkContext.parallelize(uuids), MyCustomDataClass.class).write();

其中 uuids 是 ArrayList 类型。

如何在没有此类错误的情况下以镶木地板格式保存到配置单元表？

Answer 1

我修好了

DataFrameWriter<Row> dfw =
        sparkSession.createDataFrame(uuids, CustomDataClass.class).write();

（删除了并行化）。或者，一般来说，确保您传递的 Bean 类与您传递给 createDataFrame 的列表中的类型相匹配。