使用 python 读取 CSV 文件时的编码问题

Question

我在尝试使用 python 读取 CSV 文件时遇到了障碍。

更新： 如果您只想跳过字符或错误，您可以像这样打开文件：

with open(os.path.join(directory, file), 'r', encoding="utf-8", errors="ignore") as data_file:

到目前为止我已经尝试过了。

for directory, subdirectories, files in os.walk(root_dir):
    for file in files:
        with open(os.path.join(directory, file), 'r') as data_file:
            reader = csv.reader(data_file)
            for row in reader:
                print (row)

我得到的错误是：

UnicodeEncodeError: 'charmap' codec can't encode characters in position 224-225: character maps to <undefined>

我试过了

with open(os.path.join(directory, file), 'r', encoding="UTF-8") as data_file:

错误：

UnicodeEncodeError: 'charmap' codec can't encode character '\u2026' in position 223: character maps to <undefined>

现在，如果我只是打印 data_file，它会说它们是 cp1252 编码的，但如果我尝试

with open(os.path.join(directory, file), 'r', encoding="cp1252") as data_file:

我得到的错误是：

UnicodeEncodeError: 'charmap' codec can't encode characters in position 224-225: character maps to <undefined>

我也试过推荐的包。

我得到的错误是：

UnicodeEncodeError: 'charmap' codec can't encode characters in position 224-225: character maps to <undefined>

我要解析的行是：

2015-11-28 22:23:58,670805374291832832,479174464,"MarkCrawford15","RT @WhatTheFFacts: The tallest man in the world was Robert Pershing Wadlow of Alton, Illinois. He was slighty over 8 feet 11 inches tall.","None

感谢任何想法或帮助。

Answer 1

我会使用csvkit，它使用自动检测适当的编码和解码。例如

import csvkit
reader = csvkit.reader(data_file)

正如聊天中讨论的那样-解决方案是-

for directory, subdirectories, files in os.walk(root_dir): 
    for file in files: 
        with open(os.path.join(directory, file), 'r', encoding="utf-8") as data_file: 
            reader = csv.reader(data_file) 
            for row in reader: 
                data = [i.encode('ascii', 'ignore').decode('ascii') for i in row] 
                print (data)