UnicodeEncodeError: 'charmap' codec can't encode character '\u011b' in position 57: character maps to <undefined> (但无法使用 UTF-8)

Question

这个问题之前已经被问过几次了，但每次人们都说“只需添加 UTF-8”，一切都很好。据我了解，我现在正在处理的案例似乎无法通过 UTF-8 hack 解决？基本上我的程序从网站上抓取数据，但这些数据包含特殊的欧洲字符，如“č、š、ř”等......添加 encoding="UTF-8" 后错误消失但结果 CSV 文件包含完全损坏的字符特殊字符应该位于的位置。这会破坏整个文件并使其无法使用。

我自己无法在互联网上找到任何解决方案，我不知道如何处理它。我需要将这些特殊字符写入文件。另一个需要注意的是，我还需要脚本是跨平台的。我不希望它只是为了“摆脱错误”而以某种方式特定于 Windows。

这是我的代码：

with open('links.csv') as read:
    reader = csv.reader(read)
    link_list = list(reader)
    with open('ScrapedContent.csv', 'w+', newline='') as write:
        writer = csv.writer(write)
        for link in link_list:
            driver.get(', '.join(link))
            title = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "h1.page-title span.text.ng-binding")))
            offers = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "a.switcher.ng-binding.ng-scope span.ng-binding.ng-scope")))
            address = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "tr.c-aginfo__table__row td.ng-binding")))
            try:
                wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "button.value.link.ng-binding.ng-scope"))).click()
                phone_number = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "span.phone.ng-binding")))
            except TimeoutException:
                pass
            try:
                wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "button.value.link.ng-binding"))).click()
                email = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "a.value.link.ng-binding")))
            except TimeoutException:
                pass
            try:
                phone_number = phone_number.text
            except AttributeError:
                phone_number = ""
                pass
            try:
                email = email.text
            except AttributeError:
                email = ""
                pass
            print(title.text, " ", offers.text, " ", address.text, " ", phone_number, " ", email)
            writer.writerow([title.text, offers.text, address.text, phone_number, email])
        driver.quit()

我在代码中找不到任何可能导致这种情况发生的错误。非常感谢您对如何解决此问题提出任何建议！

Answer 1

不添加utf-8 时文件看起来是否正确？它们是什么编码的？

我曾经在抓取一个网页时遇到类似问题，该网页以与响应标头中所述不同的编码返回数据，这有点搞砸了requests。

我最终得到了为我解决它的以下函数：

def _load_xml_content(url):
    """Loade XML content from URL, ensuring the encoding is correct."""
    response = requests.get(url)
    try:
        xml = response.text.encode(response.encoding).decode('utf-8')
    except Exception:
        xml = response.text
    return xml

直到今天，我还不能 100% 确定发生了什么……但也值得一试 - 也许它也能神奇地为你解决问题。