按火花数据框中的特定单词过滤答案

【问题标题】：Filter By Specific words in spark dataframe按火花数据框中的特定单词过滤
【发布时间】：2016-11-20 17:33:07
【问题描述】：

我有一个包含以下数据的 spark 数据框

    +---------------------------------------------------------------------------------------------------------------------------------------------------+
    |text                                                                                                                                               |
    +---------------------------------------------------------------------------------------------------------------------------------------------------+
    |Know what you don't do at 1:30 when you can't sleep? Music shopping. Now I want to dance. #shutUpAndDANCE                                          |
    |Serasi ade haha @AdeRais "@SMTOWNGLOBAL: #SHINee ONEW(@skehehdanfdldi) and #AMBER(@llama_ajol) at KBS 'Music Bank'."        |
    |Happy Birhday Ps.Jeffrey Rachmat #JR50 #flipagram  ? Music: This I Believe (The Creed) - Hillsong…                          |

数据框是一列“文本”，其中包含#。例如 '#shutUpAndDANCE'

我正在尝试阅读每个单词并过滤掉，以便我得到一个仅带有哈希的单词列表

代码：

#Gets only those rows containing
hashtagList = sqlContext.sql("SELECT text FROM tweetstable WHERE text LIKE '%#%'")
print hashtagList.show(100, truncate=False)

#Process Rows to get the words
hashtagList = hashtagList.map(lambda p: p.text).map(lambda x: x.split(" ")).collect() 
print hashtagList

输出是：

[[u'Know', u'what', u'you', u"don't", u'do', u'at', u'1:30', u'when', u'you', u"can't", u'sleep?', u'Music', u'shopping.', u'Now', u'I', u'want', u'to', u'dance.', u'#shutUpAndDANCE'], [...]]

有没有一种方法可以在我的地图阶段过滤掉所有内容并只保留#words。

hashtagList = hashtagList.map(lambda p: p.text).map(lambda x: x.split(" "))<ADD SOMETHING HERE TO FETCH ONLY #>.collect()

【问题讨论】：

标签： apache-spark apache-spark-sql spark-dataframe

【解决方案1】：

试试这个。

from pyspark.sql import Row
from __future__ import print_function

str = "Know what you don't do at 1:30 when you can't sleep? Music shopping. Now I want to dance. #shutUpAndDANCE Serasi ade haha @AdeRais @SMTOWNGLOBAL: #SHINee ONEW(@skehehdanfdldi) and #AMBER(@llama_ajol) at KBS 'Music Bank'.Happy Birhday Ps.Jeffrey Rachmat #JR50 #flipagram? Music: This I Believe (The Creed) - Hillsong"
df = spark.createDataFrame([Row(str)]);
words = df.rdd.flatMap(list).flatMap(lambda line: line.split()).filter(lambda word: word.startswith("#"));
words.foreach(print)

【讨论】：

但这是在 scala 中，对吧？什么是 python 等价物，使用 lambda？
hashtagList.rdd.map(lambda p: p.text.encode("ascii", "ignore")).flatMap(lambda line: line.split(" ")).filter(lambda word: word.startsWith("#")).collect() ，我在这一行得到错误
修改并添加了python版本

【解决方案2】：

用途：

>>> from pyspark.sql.functions import split, explode, col
>>>
>>> df.select(explode(split("text", "\\s+")).alias("word")) \
...     .where(col("word").startswith("#"))

【讨论】：