如何对数据框列 Python 进行词形还原答案

【问题标题】：How to lemmatise a dataframe column Python如何对数据框列 Python 进行词形还原
【发布时间】：2020-05-24 14:00:07
【问题描述】：

如何对数据框列进行词形还原。 CSV 文件“train.csv”如下所示

id  tweet
1   retweet if you agree
2   happy birthday your majesty
3   essential oils are not made of chemicals

我执行了以下操作

import pandas as pd
from nltk.tokenize import TweetTokenizer
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer

train_data = pd.read_csv('train.csv', error_bad_lines=False)
print(train_data)

# Removing stop words
stop = stopwords.words('english')
test = pd.DataFrame(train_data['tweet'])
test.columns = ['tweet']

test['tweet_without_stopwords'] = test['tweet'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
print(test['tweet_without_stopwords'])

# TOKENIZATION
tt = TweetTokenizer()
test['tokenised_tweet'] = test['tweet_without_stopwords'].apply(tt.tokenize)
print(test)

输出：

0 retweet if you agree ... [retweet, agree]
1 happy birthday your majesty ... [happy, birthday, majesty]
2 essential oils are not made of chemicals ... [essential, oils, made, chemicals]

我尝试了以下方法来引理，但我收到了这个错误 TypeError: unhashable type: 'list'


lmtzr = WordNetLemmatizer()
lemmatized = [[lmtzr.lemmatize(word) for word in test['tokenised_tweet']]]
print(lemmatized)

【问题讨论】：

看来 test 是一个列表，所以你不能调用 test['tokenised_tweet']，我认为你需要提供更多代码的细节......什么是 test？
是吗？ test = pd.DataFrame(train_data['tweet'])

标签： python dataframe lemmatization

【解决方案1】：

我会对数据框本身进行计算：

改变：

lmtzr = WordNetLemmatizer()
lemmatized = [[lmtzr.lemmatize(word) for word in test['tokenised_tweet']]]
print(lemmatized)

lmtzr = WordNetLemmatizer()
test['lemmatize'] = test['tokenised_tweet'].apply(
                    lambda lst:[lmtzr.lemmatize(word) for word in lst])

完整代码：

from io import StringIO
import pandas as pd
data=StringIO(
"""id;tweet
1;retweet if you agree
2;happy birthday your majesty
3;essential oils are not made of chemicals"""
)
test = pd.read_csv(data,sep=";")

import pandas as pd
from nltk.tokenize import TweetTokenizer
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer

# Removing stop words
stop = stopwords.words('english')

test['tweet_without_stopwords'] = test['tweet'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
print(test['tweet_without_stopwords'])

# TOKENIZATION
tt = TweetTokenizer()
test['tokenised_tweet'] = test['tweet_without_stopwords'].apply(tt.tokenize)
print(test)

lmtzr = WordNetLemmatizer()
test['lemmatize'] = test['tokenised_tweet'].apply(
                    lambda lst:[lmtzr.lemmatize(word) for word in lst])
print(test['lemmatize'])

输出

0                    [retweet, agree]
1          [happy, birthday, majesty]
2    [essential, oil, made, chemical]
Name: lemmatize, dtype: object

【讨论】：