带有 JSON 数据的 SPARK CONTEXT

Question

我的不足之处在哪里？

jsonStrings= '{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PARC PARQUE","State":"PR"}'

jsonRDD = spark.sparkContext.parallelize([jsonStrings])

df = spark.read.option('multiline', "true").json(jsonRDD)

print(df.show())

ERROR----------------

raise converted from None pyspark.sql.utils.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: {"Zipcode":704,%22ZipCodeType%22:%22STANDARD%22,%22City%22:%22PARC%20PARQUE%22,%22State%22:%22PR%22%7D

Answer 1

json() 这里 spark.read.option('multiline', "true").json(jsonRDD) 需要数据路径，而不是 RDD。文档是here

如果你想从 json 字符串创建 df，你可以这样做：

from pyspark.sql import Row

df = spark.createDataFrame(
   spark.sparkContext.parallelize([Row(json=jsonStrings)])
)