使用 pyspark 将 Dstream 转换为数据帧

Question

如何将 DStream 转换为数据帧？ 这是我的实际代码

localhost = "127.0.0.1"
addresses = [(localhost, 9999)]
schema = ['event', 'id', 'time','occurence']
flumeStream = FlumeUtils.createPollingStream(ssc, addresses)
counts = flumeStream.map(lambda line: str(line).split(",")) \
        .filter(lambda line: len(line)>1) \
        .map(lambda line: (line[29],line[30],line[67],1)) \
        .foreachRDD(lambda rdd: sqlContext.createDataFrame(rdd))

counts.show()

ssc.start()
ssc.awaitTerminationOrTimeout(62)
ssc.stop()

它给了我以下错误：

AttributeError: 'NoneType' object has no attribute 'show'

Answer 1

将你的 DStream 转换为 RDD 再转换为 DataFrame，即 dstrea.rdd.to_df