在 Python 3 中未显式打开文件时处理从字节到字符串的转换

Question

我正在使用 Requests 模块进行授权，然后从 Web API 中提取 csv 内容，并让它在 Python 2.7 中运行良好。我现在想在 Python 3.5 中编写相同的脚本，但遇到了一些问题：

"iterator should return strings, not bytes (did you open the file in text mode?)"

requests.get 似乎返回字节而不是字符串，这似乎与迁移到 Python 3.x 时出现的编码问题有关。错误在最后一行的第 3 行引发：next(reader)。在 Python 2.7 中这不是问题，因为 csv 函数是在 'wb' 模式下处理的。

这篇文章非常相似，但由于我没有直接打开 csv 文件，我似乎无法强制以这种方式对响应文本进行编码： csv.Error: iterator should return strings, not bytes

countries = ['UK','US','CA']
datelist = [1,2,3,4]
baseurl = 'https://somewebsite.com/exporttoCSV.php'

#--- For all date/cc combinations
for cc in countries:
    for d in datelist:

        #---Build API String with variables
        url = (baseurl + '?data=chart&output=csv' +
               '&dataset=' + d + 
               '&cc=' + cc)

        #---Run API Call and create reader object
        r = requests.get(url, auth=(username, password))
        text = r.iter_lines()
        reader = csv.reader(text,delimiter=',')

        #---Write csv output to csv file with territory and date columns
        with open(cc + '_'+ d +'.csv','wt', newline='') as file:
            a = csv.writer(file)
            a.writerow(['position','id','title','kind','peers','territory','date']) #---Write header line
            next(reader) #---Skip original headers
            for i in reader:
                a.writerow(i +[countrydict[cc]] + [datevalue])

Answer 1

由于无法测试您的确切情况，我认为应该通过将text = r.iter_lines() 更改为：

text = (line.decode('utf-8') for line in r.iter_lines())

这应该将 r.iter_lines() 读取的每一行从字节字符串解码为 csv.reader 可用的字符串

我的测试用例如下：

>>> iter_lines = [b'1,2,3,4',b'2,3,4,5',b'3,4,5,6']
>>> text = (line.decode('utf-8') for line in iter_lines)
>>> reader = csv.reader(text, delimiter=',')
>>> next(reader)
['1', '2', '3', '4']
>>> for i in reader:
...     print(i)
...
['2', '3', '4', '5']
['3', '4', '5', '6']

Answer 2

某些文件必须以字节形式读取，例如来自Django SimpleUploadedFile，这是一个仅使用字节的测试类。以下是我的测试套件中的一些示例代码，说明我是如何工作的：

test_code.py

import os
from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import TestCase

class ImportDataViewTests(TestCase):

    def setUp(self):
        self.path = "test_in/example.csv"
        self.filename = os.path.split(self.file)[1]

    def test_file_upload(self):
        with open(self.path, 'rb') as infile:
            _file = SimpleUploadedFile(self.filename, infile.read())

        # now an `InMemoryUploadedFile` exists, so test it as you shall!

prod_code.py

import csv

def import_records(self, infile):
    csvfile = (line.decode('utf8') for line in infile)
    reader = csv.DictReader(csvfile)

    for row in reader:
        # loop through file and do stuff!