为 request.get 创建参数时的错误

Question

我正在迭代这个网站并尝试抓取新闻文章的链接。 https://www.usnews.com/search?q=China%20COVID-19#gsc.tab=0&gsc.q=China%20COVID-19&gsc.page=1

首先，我需要获取页面的链接，所以我使用了这段代码：

def scrape(url):
    user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko'}
    request = 0
    params = {
        'q': 'China%20COVID-19',
        'gsc.tab': '0',
        'gsc.q': 'China%20COVID-19',
    }
    pagelinks = []
    
    myarticle = []
    for page_no in range(1,3):
        try:# to avoid "No connection adapters were found for" error
            params['gsc.page'] = page_no
            response = requests.get(url=url,
                                    headers=user_agent,
                                    params=params) 
            print(response.request.url)
            
        except Exception as e:
            print(e)

scrape('https://www.usnews.com/search/')

但是，结果不正确：

https://www.usnews.com/search?q=China%252520COVID-19&gsc.tab=0&gsc.q=China%252520COVID-19&gsc.page=1
https://www.usnews.com/search?q=China%252520COVID-19&gsc.tab=0&gsc.q=China%252520COVID-19&gsc.page=2

预期的结果应该是这样的：

https://www.usnews.com/search?q=China%20COVID-19#gsc.tab=0&gsc.q=China%20COVID-19&gsc.page=1
https://www.usnews.com/search?q=China%20COVID-19#gsc.tab=0&gsc.q=China%20COVID-19&gsc.page=2

有人可以帮我解决这个错误吗？非常感谢！

Answer 1

如果您在浏览器中打开您获得的 URL，您会看到搜索字符串是 China%2520COVID-19，而不是您所期望的 China COVID-19。

您在查询字符串中看到的%20 是一个urlencoded 空格字符。见percent-encoding on MDN。如果你urldecodeChina%2520COVID-19，你会得到China%20COVID-19，由此我知道%25是编码的百分比字符。

可能requests 已经urlencodes 你的查询字符串值，所以你不需要。你可以改变它的工作是使用解码的值，例如空格 () 而不是 %20。

    params = {
        'q': 'China COVID-19',
        'gsc.tab': '0',
        'gsc.q': 'China COVID-19',
    }

Answer 2

在浏览器中搜索“散列”网址 (https://www.usnews.com/search#...)，您需要自己构建网址。
通过 requests.get(..., params=params) 将参数传递给请求会创建一个常规查询字符串 (https://www.usnews.com/search?...)，该字符串会导致错误页面（始终是第一个）。

import requests
from urllib.parse import urlencode, unquote

def scrape(url):
    user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko'}
    request = 0
    url = 'https://www.usnews.com/search'
    params = {
        'q': 'China COVID-19',
        'gsc.tab': '0',
        'gsc.q': 'China COVID-19'
    }
    pagelinks = []
    myarticle = []

    for page_no in range(1,4):
        params['gsc.page'] = page_no
        _url = '%s#%s' % (url, urlencode(params))

        try:# to avoid "No connection adapters were found for" error
            response = requests.get(url=_url,
                                    headers=user_agent)
            print(_url, '>>', _url == unquote(response.request.url))

        except Exception as e:
            print(e)

scrape('https://www.usnews.com/search/')

输出：

https://www.usnews.com/search#q=China+COVID-19&gsc.tab=0&gsc.q=China+COVID-19&gsc.page=1 >> True
https://www.usnews.com/search#q=China+COVID-19&gsc.tab=0&gsc.q=China+COVID-19&gsc.page=2 >> True
https://www.usnews.com/search#q=China+COVID-19&gsc.tab=0&gsc.q=China+COVID-19&gsc.page=3 >> True