urllib.request 在 unicode 字符串的 urlopen 上失败 [重复]答案

【问题标题】：urllib.request fails on urlopen of unicode string [duplicate]urllib.request 在 unicode 字符串的 urlopen 上失败 [重复]
【发布时间】：2017-05-21 17:51:58
【问题描述】：

from urllib.request import Request, urlopen, urlretrieve
from bs4 import BeautifulSoup
def save_picture(self, word):
    search_string = "https://www.google.nl/search?q={}&tbm=isch&tbs=isz:m".format(word)

    request = Request(search_string, headers={'User-Agent': 'Mozilla/5.0'})
    raw_website = urlopen(request).read()

    soup = BeautifulSoup(raw_website, "html.parser")
    image = soup.find("img").get("src")

    urlretrieve(image, "{}.jpg".format(word))

我编写了上面的函数来保存 Google 图片中的第一张缩略图。然而问题是当我输入一个非 ansii 单词时它会失败，比如：mañana

错误消息来自 urllib 模块。我正在使用 python 3.6

    Traceback (most recent call last):   File "c:\users\xxx\Desktop\script.py", line 19, in <module>
    main()   File "c:\users\xxx\Desktop\script.py", line 16, in main
    save_picture("mañana")   File "c:\users\xxx\Desktop\script.py", line 8, in save_picture
    raw_website = urlopen(request).read()   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 544, in _open
    '_open', req)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1250, in _send_request
    self.putrequest(method, url, **skips)   File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1117, in putrequest
    self._output(request.encode('ascii')) UnicodeEncodeError: 'ascii' codec can't encode character '\xf1' in position 16: ordinal not in range(128)

edit: 阅读后我发现有几个库可用于此任务，urllib、urllib2 和 requests（以及通过 pip : urllib3）。我收到此错误是因为我使用的是已弃用的库吗？

edit2：添加了完整的回溯

【问题讨论】：

发布完整的回溯，以便我们了解上下文。

标签： python unicode

【解决方案1】：

import requests
import mimetypes
from bs4 import BeautifulSoup

def save_picture(self, word):
    search_string = "https://www.google.nl/search?q={}&tbm=isch&tbs=isz:m".format(word)
    response = requests.get(search_string, headers={'User-Agent': 'Mozilla/5.0'})

    #find the tumbnail for first hit
    soup = BeautifulSoup(response.text, "html.parser")
    image_location = soup.find("img").get("src")

    # download image
    image = requests.get(image_location)
    content_type = image.headers.get('content-type')
    ext = mimetypes.guess_extension(content_type)

    with open(f"{word}{ext}", 'wb') as fd:
        for chunk in image.iter_content(chunk_size=128):
            fd.write(chunk)

我使用 requests 重写了该函数，它按预期处理 unicode 字符串。但是保存文件有点冗长

【讨论】：