在python中加载大型csv文件“'utf-8'编解码器无法解密”时出错[重复]

Question

我有 1 GB 的 csv 文件，我无法读取该日志文件并在我的 csv 文件中的 python 和 pandas 代码中给出相同的错误，它不是一个多列的值，因为只有一个列值我所有的 CSV 值都是数字

with open("/Users/kiya/sep_sent.csv", encoding='utf-8') as f:
for i in f: 
   print(i.strip())

另一种方法：

with open("/Users/kiya/sep_sent.csv",encoding='cp1252') as f:
    for i in f:
      print(i.strip())

Traceback (most recent call last):
  File "/Users/kiya/test8.py", line 5, in <module>
    for i in f:
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 159: character maps to <undefined>

熊猫代码：

import pandas as pd
df = pd.read_csv("/Users/kiya/sep_sent.csv", encoding="utf-8")
print(df)

我的 csv 值如：

0
0
0
....
5294751024

错误：

0
0
0
0
0
Traceback (most recent call last):
  File "/Users/kiya//test8.py", line 4, in <module>
    for i in f:
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 52: invalid start byte

Answer 1

使用 utf-8 编码打开文件，它应该可以工作：

with open("/Users/kiya/sep_sent.csv", encoding='utf-8') as f:
    for i in f: 
       print(i.strip())

Answer 2

您也可以将编码参数传递给 read_csv

df = pd.read_csv("/Users/kiya/sep_sent.csv", encoding="utf-8")