Can't get correct representation for some Italian words

Question

我需要使用 python3 和 nltk 对来自意大利语 wiki 的文本进行规范化，但我遇到了一个问题。大多数单词都可以，但有些单词映射不正确，更准确地说是一些符号。

例如：

'fruibilit\xe3'，'n\xe2\xba'，'citt\xe3'

我确定问题出在 à、è 等符号上。

代码：

# coding: utf8
import os

from nltk import corpus, word_tokenize, ConditionalFreqDist


it_sw_plus = corpus.stopwords.words('italian') + ['doc', 'https']
#it_folder_names = ['AA', 'AB', 'AC', 'AD', 'AE', 'AF']
it_path = os.listdir('C:\\Users\\1\\projects\\i')
it_corpora = []

def normalize(raw_text):
    tokens = word_tokenize(raw_text)
    norm_tokens = []
    for token in tokens:
        if token not in it_sw_plus and token.isalpha():
            token = token.lower().encode('utf8')
            norm_tokens.append(token)
    return norm_tokens

for folder_name in it_path:
    path_to_files = 'C:\\Users\\1\\projects\\i\\%s' % (folder_name)
    files_list = os.listdir(path_to_files)
    for file_name in files_list:
        file_path = path_to_files + '\\' + file_name
        text_file = open(file_path)
        raw_text = text_file.read().decode('utf8')
        norm_tokens = normalize(raw_text)
        it_corpora.append(norm_tokens)
    print(it_corpora)

我该如何解决这个问题？ 我在 Win7(rus) 上运行。

当我尝试这段代码时：

import io

with open('C:\\Users\\1\\projects\\i\\AA\\wiki_00', 'r', encoding='utf8') as fin:
    for line in fin:
        print (line) 

在 PowerShell 中：

    <doc id="2" url="https://it.wikipedia.org/wiki?curid=2" title="Armonium">

Armonium



Traceback (most recent call last):
  File "i.py", line 5, in <module>
    print (line)
  File "C:\Python35-32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position 3: character maps to <undefined>

在 Python 命令行中：

<doc id="2" url="https://it.wikipedia.org/wiki?curid=2" title="Armonium">

Armonium



Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\1\projects\i.py", line 5, in <module>
    print (line)
  File "C:\Python35-32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position
3: character maps to <undefined>

当我尝试请求时：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python35-32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position
90: character maps to <undefined>

Answer 1

如果您知道编码，请尝试在读取文件时指定编码，在python2中

import io
with io.open(filename, 'r', encoding='latin-1') as fin:
    for line in fin:
        print line # line should be encoded as latin-1

但就您而言，您发布的文件不是latin1 文件，而是utf8 文件，位于python3：

>>> import urllib.request
>>> url = 'https://raw.githubusercontent.com/GiteItAwayNow/TrueTry/master/it'
>>> response = urllib.request.urlopen(url)
>>> data = response.read()
>>> text = data.decode('utf8')
>>> print (text) # this prints the file perfectly.

读取python2中的'utf8'文件：

import io
with io.open(filename, 'r', encoding='utf8') as fin:
    for line in fin:
        print (line) # line should be encoded as utf8

要读取“utf8”文件，在python3：

with open(filename, 'r', encoding='utf8') as fin:
    for line in fin:
        print (line) # line should be encoded as utf8

作为一个好习惯，在处理文本数据时，尽可能尝试使用 unicode 和 python3。来看看

• https://docs.python.org/3.5/howto/unicode.html#the-string-type
• What's the deal with Python 3.4, Unicode, different languages and Windows?
• UnicodeEncodeError: 'charmap' codec can't encode - character maps to <undefined>, print function

另外，如果你还没有安装这个模块在windows控制台上打印utf8，你应该试试：

pip install win-unicode-console

或者下载这个：https://pypi.python.org/packages/source/w/win_unicode_console/win_unicode_console-0.4.zip 然后python setup.py install