在 Python 的 NLP 中应用 n-gram 模式时向 Dataframe 添加常用词答案

【问题标题】：Adding Frequent words to Dataframe while applying n-gram pattern in NLP in Python在 Python 的 NLP 中应用 n-gram 模式时向 Dataframe 添加常用词
【发布时间】：2018-08-24 01:17:27
【问题描述】：

场景是，我将推文分为 6 种不同的极性，即。正面、中度正面、高度正面、负面、中度负面和高度负面。

因为，这个过程经历了 NLP 的步骤（使用 NLTK）我需要逐个推文/推文。

问题：
这些极性由 POS 标记后的模式定义。其中一种模式包括：动词+动词+形容词包括在D（干旱相关术语）和F（常用词）中

我需要那些常用词，将这句话更改为这 6 种极性中的任何一种，以保存到我的数据框中。

片段：
这是我尝试过的

for (w1, tag1), (w2, tag2), (w3, tag3) in nltk.trigrams(PoS_TAGS):
    if tag1.startswith("RB") and tag2.startswith("RB") and tag3.startswith("JJ"):
        tri_pairs.append((w1, w2, w3))
        if tri_pairs[0] or tri_pairs[1] or tri_pairs[2] in D:
            print("[True]: Tri Pairs are found in Drought Rel. Term")

            for j in range(len(F)):
                if tri_pairs[0] or tri_pairs[1] or tri_pairs[2] in F[j]:
                    print("[True]: Tri Pairs are found in Frequent Wordset")
                    if RES is "Positive":
                        RES = "Highly Positive"
                    elif RES is "Negative":
                        RES = "Highly Negative"
                    print "="*25,F[j]
                    FW_list.append(F[j])
                else:
                    print"[False]: Doesn't Match with Frequent Wordset\n"

        else:
            print"[False]: Tri Pairs Matched Nowhere in D\n"
    else:
        print "[TriPair(F)]: Pattern for Adverb, Adverb, Adjective did not match.\n Looking for Bi-Pair Patterns\n"
print(tri_pairs)
print(">"*13,FW)

如您所见，我尝试以大多数方式使用 List 甚至内部循环进行打印。两者都没有返回任何有用的东西。同样，其他两种模式决定了被忽略的极性。

我还编写了代码将其添加到数据框中：

fuzzy_df = fuzzy_df.append({'Tweets': tweets[i], 'Classified': RES, 'FreqWord': FW}, ignore_index=True)

但到目前为止，该列的 csv 为空白。

编辑 1：

我已经可以使用常用词。它们如下：

>>> F
['drought', 'water', 'love', 'rain', 'year', 'famine', 'farmers', 'crops', 'south', 'http', 'europe', 'scarcity', 'near', 'thought', 'ever', 'devastates', 'feed', 'message', 'eduaubdedubu', 'instant', 'italy', 'severe', 'by', 'beaches', 'wildfires', 'heat', 'us']

编辑 2

CSV 看起来像这样：

Tweets,Classified,FreqWord
 real time strategy password wastelands depletion groundwater skyrocketing debts make years anantapur drought worse,Negative,
 calm director day science meetings nasal talk cutting edge remote sensing research drought veg fluorescence calm love,Positive,
 love thought drought,Positive,
 neville rooney end ever tons trophy drought,Positive,
 lakes drought,Positive,
 lakes fan joint trailblazers dot forget play drought,Positive,
 reign mother kerr funny none tried make come back drought,Positive,
 wonder could help thai market b post reuters drought devastates south europe crops,Negative,

输入文件：

tweets,polarity
real time strategy password wastelands depletion groundwater skyrocketing debts make years anantapur drought worse,Positive
calm director day science meetings nasal talk cutting edge remote sensing research drought veg fluorescence calm love,Positive
hate thought drought,Negative

虽然，我上面显示的输出是标记化的，并且停用词也被删除了。

预期输出文件：

Tweets,Classified,FreqWord
     real time strategy password wastelands depletion groundwater skyrocketing debts make years anantapur drought worse,Negative,water
     calm director day science meetings nasal talk cutting edge remote sensing research drought veg fluorescence calm love,Positive,drought
     love thought drought,Positive,drought
     neville rooney end ever tons trophy drought,Positive,rain
     lakes drought,Positive,drought
     lakes fan joint trailblazers dot forget play drought,Positive,farmer
     reign mother kerr funny none tried make come back drought,Positive,crops
     wonder could help thai market b post reuters drought devastates south europe crops,Negative,crops

编辑 3

FW = ''
for i in range(len(tweets)):
    sent = nltk.word_tokenize(tweets[i])
    PoS_TAGS = nltk.pos_tag(sent)

    from nltk.sentiment.vader import SentimentIntensityAnalyzer
    sia = SentimentIntensityAnalyzer()

    one_sentence = tweets.iloc[i]
    scores = sia.polarity_scores(text=one_sentence)
    print "POS:", scores.get('pos')
    print "NEG:", scores.get('neg')
    print "NEU:", scores.get('neu')

    POS = scores.get('pos')
    NEG = scores.get('neg')
    NEU = scores.get('neu')
    RES = str()

    if POS > NEG:
        RES = 'Positive'
    elif NEG > POS:
        RES = 'Negative'
    elif NEU >= 0.5 or POS > NEU:
        RES = 'Positive'
    elif NEU < 0.5:
        RES = 'Negative'

    # -------------------------------------------------------- PATTERN ADVERB, ADVERB, ADJECTIVE (Down)
    tri_pairs = list()
    for (w1, tag1), (w2, tag2), (w3, tag3) in nltk.trigrams(PoS_TAGS):
        if tag1.startswith("RB") and tag2.startswith("RB") and tag3.startswith("JJ"):
            tri_pairs.append((w1, w2, w3))
            if tri_pairs[0] or tri_pairs[1] or tri_pairs[2] in D:
                print("[True]: Tri Pairs are found in Drought Rel. Term")
                # TRIGGER AREA
                for j in range(len(F)):
                    if tri_pairs[0] or tri_pairs[1] or tri_pairs[2] in F[j]:
                        print("[True]: Tri Pairs are found in Frequent Wordset")
                        if RES is "Positive":
                            RES = "Highly Positive"
                            FW = F[j]
                            #fuzzy_df['FreqWord'].map(lambda x: next((y for y in x.split() if y in F), 'Not Found'))
                        elif RES is "Negative":
                            RES = "Highly Negative"
                            FW = F[j]
                    else:
                        print"[False]: Doesn't Match with Frequent Wordset\n"

            else:
                print"[False]: Tri Pairs Matched Nowhere in D\n"

        else:
            print "[TriPair(F)]: Pattern for Adverb, Adverb, Adjective did not match.\n Looking for Bi-Pair Patterns\n"
    print(tri_pairs)

    # -------------------------------------------------------- PATTERN ADVERB, ADJECTIVE (Down)
    bi_pairs = list()
    for (w1, tag1), (w2, tag2) in nltk.bigrams(PoS_TAGS):
        if tag1.startswith("RB") and tag2.startswith("JJ"):
            bi_pairs.append((w1, w2))

            if bi_pairs[0] or bi_pairs[1] in D:
                print("[True]: Bi Pairs are found in Drought Rel. Term")

                for k in range(len(F)):
                    if bi_pairs[0] or bi_pairs[1] is F[k]:
                        print("[True]: Bi Pairs are found in Frequent Wordset")
                        if RES is "Positive":
                            RES = "Moderately Positive"
                            FW = F[k]
                        elif RES is "Negative":
                            RES = "Moderately Negative"
                            FW = F[k]
                    else:
                        print("[False]: Bi Pairs found missing in Freq. Wordset")

            else:
                print("[False]: Bi Pairs Matched Nowhere in D")

        else:
            print("[BiPair(F)]: Pattern Not Matched, Looking for Mono Pattern")
    print(bi_pairs)

    # -------------------------------------------------------- PATTERN ADJECTIVE (Down)
    for w, tag in PoS_TAGS:
        print w, " - ", tag
        if tag.startswith("JJ"):
            if w in D:
                print("Matched with D")
                for l in range(len(F)):
                    if w is F[l]:
                        print("Matched with F")
                        if RES is "Positive":
                            RES = "Positive"
                            FW = F[l]
                        elif RES is "Negative":
                            RES = "Negative"
                            FW = F[l]
                    else:
                        print("Unmatched in F")
                        FW = F[l] in sent
            else:
                print("Unmatched in D")
        else:
            print w, "is not an ADJECTIVE"


# -------------------------------------------------------- MAKING ENTRY OF RECORDS OF TWEETS and POLARITY RESULT
    fuzzy_df = fuzzy_df.append({'Tweets': tweets[i], 'Classified': RES, 'FreqWord': FW}, ignore_index=True)
# ADDING RECORDS IN DATAFRAME
fuzzy_df.to_csv("fuzzy.csv", index=False)

【问题讨论】：

我不确定我是否理解你的意思。但是要得到频繁的单词，你不能只做 nltk.tokenize，得到每个单词的计数，按降序排序，得到最频繁的单词吗？
我已经将常用词存储在一个列表中。我将使用列表 F 值更新问题@ManishSaraswat
您有示例 csv 文件吗？
@alvas 当然，请检查编辑 2
另外，您是否有一个示例输入文件和您希望获得的所需输出？

标签： python python-2.7 pandas csv nltk

【解决方案1】：

你想这样做吗？

import io
import pandas as pd
from collections import Counter

open_strio = io.StringIO("""Tweets,Classified,FreqWord
real time strategy password wastelands depletion groundwater skyrocketing debts make years anantapur drought worse,Negative,
calm director day science meetings nasal talk cutting edge remote sensing research drought veg fluorescence calm love,Positive,
love thought drought,Positive,
neville rooney end ever tons trophy drought,Positive,
lakes drought,Positive,
lakes fan joint trailblazers dot forget play drought,Positive,
reign mother kerr funny none tried make come back drought,Positive,
wonder could help thai market b post reuters drought devastates south europe crops,Negative,""")

with open_strio as fin:
    df = pd.read_csv(open_strio)


dictionary = Counter(['drought', 'water', 'love', 'rain', 'year', 'famine', 'farmers', 'crops', 'south', 'http', 'europe', 'scarcity', 'near', 'thought', 'ever', 'devastates', 'feed', 'message', 'eduaubdedubu', 'instant', 'italy', 'severe', 'by', 'beaches', 'wildfires', 'heat', 'us'])

df['FreqCounter'] = df['Tweets'].apply(lambda x: Counter(x.split()) & dictionary)
df['FreqWord'] = df['FreqCounter'].apply(lambda x: list(x.keys()))

首先从定义的单词列表中创建一个简单的Counter() 对象（即dictionary）。

然后将Counter() 交集应用到每行推文以创建df['FreqCounter'] 列。

最后，从df['FreqCounter'] 中提取唯一键集以填充df['FreqWord']

如果您不需要每行推文的dictionary 中的单词计数器，您可以简单地使用一个集合，即

import io
import pandas as pd

open_strio = io.StringIO("""Tweets,Classified,FreqWord
real time strategy password wastelands depletion groundwater skyrocketing debts make years anantapur drought worse,Negative,
calm director day science meetings nasal talk cutting edge remote sensing research drought veg fluorescence calm love,Positive,
love thought drought,Positive,
neville rooney end ever tons trophy drought,Positive,
lakes drought,Positive,
lakes fan joint trailblazers dot forget play drought,Positive,
reign mother kerr funny none tried make come back drought,Positive,
wonder could help thai market b post reuters drought devastates south europe crops,Negative,""")

with open_strio as fin:
    df = pd.read_csv(open_strio)


dictionary = set(['drought', 'water', 'love', 'rain', 'year', 'famine', 'farmers', 'crops', 'south', 'http', 'europe', 'scarcity', 'near', 'thought', 'ever', 'devastates', 'feed', 'message', 'eduaubdedubu', 'instant', 'italy', 'severe', 'by', 'beaches', 'wildfires', 'heat', 'us'])

df['FreqWord'] = df['Tweets'].apply(lambda x: set(x.split()) & dictionary)

如果您想要df['FreqCounter'] 中出现频率最高的单词，那么：

import io
import pandas as pd
from collections import Counter

open_strio = io.StringIO("""Tweets,Classified,FreqWord
real time strategy password wastelands depletion groundwater skyrocketing debts make years anantapur drought worse,Negative,
calm director day science meetings nasal talk cutting edge remote sensing research drought veg fluorescence calm love,Positive,
love thought drought,Positive,
neville rooney end ever tons trophy drought,Positive,
lakes drought,Positive,
lakes fan joint trailblazers dot forget play drought,Positive,
reign mother kerr funny none tried make come back drought,Positive,
wonder could help thai market b post reuters drought devastates south europe crops,Negative,""")

with open_strio as fin:
    df = pd.read_csv(open_strio)


dictionary = Counter(['drought', 'water', 'love', 'rain', 'year', 'famine', 'farmers', 'crops', 'south', 'http', 'europe', 'scarcity', 'near', 'thought', 'ever', 'devastates', 'feed', 'message', 'eduaubdedubu', 'instant', 'italy', 'severe', 'by', 'beaches', 'wildfires', 'heat', 'us'])

df['FreqCounter'] = df['Tweets'].apply(lambda x: Counter(x.split()) & dictionary)
df['FreqWord'] = df['FreqCounter'].apply(lambda x: x.most_common()[0][0])

【讨论】：

嗨，阿尔瓦斯，我试过了。这留下了我的FreqWord 列，其中包含列表。为了进一步处理，我必须先提取为列表。导致更多的代码行。此外，常用词列表中可能只有一个元素。
嗨，Alvas，我添加了更多的透视图。如果愿意，您可以检查编辑 3。这可能会增加更多洞察力。
添加更多代码无济于事，您必须更具体地了解您的需求。
非常正确。 Tweet 因为它被拆分为这些模式，所以我需要这些模式来确定哪个常用词影响了推文。基于此，我需要将其存储到数据框中。我希望这会有所帮助。

【解决方案2】：

使用这个最小的例子，你也可以尝试这个简单的方法：
next 将遍历推文行，检查推文中是否有任何常用词，如果没有，它将返回 Not Found。

# sample data frame
df = pd.DataFrame({'name': ['I am going somewhere','tomorrow is holiday']})

# list of frequent words
lst = ['holiday','am']

# check if any word in tweets exist in list of frequent words
df['freq'] = df['name'].map(lambda x: next((y for y in x.split() if y in lst), 'Not Found'))

print(df)
    name                    freq
0   I am going somewhere    am
1   tomorrow is holiday     holiday

【讨论】：

我试试看。
嘿Manish，试过这个。非常接近我的解决方案，但请注意内部 for 循环。我需要在 if-else 语句中确定这个常用词。我们可以调整它吗？
你的意思是if-else语句中要生成的常用词列表？频繁词表不是基于整个词汇表吗？
好吧，如果你注意到推文被转化为单词，我需要找出是哪种模式设置了paorality，以及使用了哪些频繁单词。所以在 3 种模式结束时，我将 value/freq 字写入 df。 fuzzy_df = fuzzy_df.append({'Tweets': tweets[i], 'Classified': RES, 'FreqWord': FW}, ignore_index=True)
您可能想检查编辑 3。我希望它提供一些见解。