【发布时间】：2017-10-12 20:39:23
【问题描述】：
我的目标是在 Python 中为这样的文本构建一个词干分析器
它应该输出类似这样的没有词干的东西：
भधएवंतृततृततृततृतहै।अपनेककमेंमेंससउपलबउपलबधिककककसिललीलीलीलीकुछकुछकुछकुछकुछकुछकुछ पआपदेदेतोकुछअदभुतनेवतोतोतो ，，，नननीनीनीनीनीनीनीनीईईईसेसेसेसेसेंधींधींधींधींधींधींधी परन्तुमहिलाकीस्थितिमेंकितनापरिवर्तनआया? और आम महिला ने परिवर्तन को किस तरह से देखा? 7065534
但我得到的输出为
ï » ¿ à ¤ à ¤ ¾ à ¤ ° ¤ ¤ à ¤ • à ¤ ¾ à ¤ ‡ à ¤ ¤ à ¤ ¿ à ¤......像这样
这是字典（代码）中的任何内容在完成算法后都应该被条带化我收到一个错误编码或解码作为初学者很难解决
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import re
separators = [u"।", u",", u"."]
dat=open(r"C:\Users\User\Desktop\text1.txt",'r').read()
print dat
text=dat.decode("utf-8")
out=[]
suffx = {
1: ["ो", "े", "ू", "ु", "ी", "ि", "ा"],
2: ["कर", "ाओ", "िए", "ाई", "ाए", "ने", "नी", "ना", "ते", "ीं", "ती", "ता", "ाँ", "ां", "ों", "ें"],
3: ["ाकर", "ाइए", "ाईं", "ाया", "ेगी", "ेगा", "ोगी", "ोगे", "ाने", "ाना", "ाते", "ाती", "ाता", "तीं", "ाओं", "ाएं", "ुओं", "ुएं", "ुआं"],
4: ["ाएगी", "ाएगा", "ाओगी", "ाओगे", "एंगी", "ेंगी", "एंगे", "ेंगे", "ूंगी", "ूंगा", "ातीं", "नाओं", "नाएं", "ताओं", "ताएं", "ियाँ", "ियों", "ियां"],
5: ["ाएंगी", "ाएंगे", "ाऊंगी", "ाऊंगा", "ाइयाँ", "ाइयों", "ाइयां","िया","ीया","वाला","ेवाला","ाऊ","ाका","ालू","ेरा","ेया","हारा","ाक","ाड़ी","क","न","ावा"],
6: ["ंत","ाई","ावट","ाहट","या","कर","ना","कार","ेरा","वाला","ार","ची","पन","ीमा","िमा","पा","ाल","जा","िक","िया","टी","टा","ड़ी","ड़ा","हरा","सा","था","िय","ीय","ीला","िला","लु","वंत","वान","ौती"],
}
j=1;
for l in 1,2,3,4,5,6:
for j in suffx[l]:
#print j.decode("utf-8")
print j
import string
#print type( r3_bad)
#print type(r3_bad[1])
space=[]
suffix=[]
i=0;
def stemmed_word(word):
for k in 1,2,3,4,5,6:
j=1;
for j in suffx[k]:
if(word.endswith(j)):
return(word[:-j])
return word
for word in dat :
removed=""
removed=stemmed_word(word)
#i+=1
out.append(removed)
print out
result = u' '.join(out)
result = result.encode('utf-8')
result=' '.join(out)
writ=open("C:\\Users\\User\\Desktop\\text5.txt",'w')
writ.write(result)
writ.close()
def remove3(str_re,re_bad):
#to remove or replace the string having bad chars defined
outrm=str_re.translate(string.maketrans("","", ), re_bad)
print "function called:"+word
return outrm;
【问题讨论】：
-
python 2.7 与 unicode 不太容易相处，升级到 python 3 会更好
-
@JKirchartz 这将如何影响我的代码..？
-
@JKirchartz 仍然没有变化！
-
我想我知道你的程序出了什么问题，我会尽力修复它。以后请不要发布乱七八糟的代码。您应该发布关注您的问题的minimal reproducible example。
-
@PM2Ring 请尝试解决它！错误基本上是它无法将 unicode 转换回来，导致输出混乱！！
标签： python python-2.7 unicode utf-8