“UnicodeEncodeError: 'charmap' codec can't encode character” 使用 Webscraper 写入 csv 时

Question

我编写了一个网络爬虫，可以从basketball-reference 中抓取 NBA 得分数据。我的错误的具体网页：

UnicodeEncodeError: 'charmap' codec can't encode character '\u0107' in position 11: character maps to <undefined>

发生在here。最后，引发此错误并引发此特定 UnicodeEncodeError 的特定玩家数据是 this one（尽管我确信该错误更为笼统，并且会在任何包含晦涩重音符号的字符中产生）。

最小可重现代码：

def get_boxscore_basic_table(tag): #used to only get specific tables
    tag_id = tag.get("id")
    tag_class = tag.get("class")
    return (tag_id and tag_class) and ("basic" in tag_id and "section_wrapper" in tag_class and not "toggleable" in tag_class)

import requests
from bs4 import BeautifulSoup
import lxml
import csv
import re

website = 'https://www.basketball-reference.com/boxscores/202003110MIA.html'

r = requests.get(website).text
soup = BeautifulSoup(r, 'lxml')

tables = soup.find_all(get_boxscore_basic_table)

in_file = open('boxscore.csv', 'w', newline='')
csv_writer = csv.writer(in_file)
column_names = ['Player','Name','MP','FG','FGA','FG%','3P','3PA','3P%','FT','FTA','FT%','ORB','DRB','TRB','AST','STL','BLK','TOV','PF','PTS','+/-']
csv_writer.writerow(column_names)

for table in tables:    
    rows = table.select('tbody tr')

    for row in rows:
        building_player = [] #temporary container to hold player and stats
        player_name = row.th.text 
        if 'Reserves' not in player_name: 
            building_player.append(player_name)

        stats = row.select('td.right')

        for stat in stats:
            building_player.append(stat.text)

        csv_writer.writerow(building_player) #writing to csv

in_file.close()

解决这个问题的最佳方法是什么？

我在网上看到了一些关于更改编码的东西，特别是在写入 csv 之前在字符串上使用.encode('utf-8') 方法，但似乎这个 .encode() 方法，虽然它阻止了错误被抛出，有几个自己的问题。例如; player_name.encode('utf-8') 在写入 csv 之前将名称 'Willy Hernangómez' 转换为 csv 内的 'b'Willy Hernang\xc3\xb3mez'' ......这并不完全是朝着正确方向迈出的一步。

对此的任何帮助以及对正在发生的事情的解释将不胜感激！

Answer 1

使用

in_file = open('boxscore.csv', 'w', newline='',  encoding='utf-8')

而不是

in_file = open('boxscore.csv', 'w', newline='')

并保持一切不变。确保你在utf-8 encoding中打开Excel