HtmlResponse requests.response 之间的区别

Question

我想提取this website webpage的香水品牌的所有链接。

我想做一个能找到所有href的scrapy scraper，就像我对BeautifulSoup所做的那样：

    soup = BeautifulSoup(requests.get('https://www.nosetime.com'+ url, 
                                      headers=headers).content, 'html.parser')
    print(soup)
    result = soup.find_all('a', {'class': 'imgborder'})
    for r in result:
        brand_url = r.attrs['href']

requests.get returns a Response object这里

但是这种自制技术会因 403 错误而崩溃，所以我想制作一个 scrapy 刮板，因为本教程声称它可以处理这些错误。

import scrapy
from bs4 import BeautifulSoup

class NosetimeScraper(scrapy.Spider):
    name = "nosetime"
    urls = ['/pinpai/2-a.html']
    start_urls = ['https://www.nosetime.com' + url for url in urls]

    def parse(self, response):
        # proceed to other pages of the listings
        soup = BeautifulSoup(response.content, 'html.parser')
        results = soup.find_all('a', {'class': 'imgborder'})
        for r in results:
            brand_url = r.attrs['href']
            yield scrapy.Request(url=brand_url, callback=self.parse)

        # then do something with the scrapy.Request() response that has been yielded ...

但它会返回：

soup = BeautifulSoup(response.content, 'html.parser')
AttributeError: 'HtmlResponse' object has no attribute 'content'

所以我猜 HtmlResponse requests.response 之间有区别？

Answer 1

这是因为scrapy.http.HtmlResponse 与requests.models.Response 不同。您可以改用response.body。

但是将.css() 与 Scrapy 一起使用会更好：

def parse(self, response):
    urls = response.css('a.imgborder::attr(href)').getall()
    for url in urls:
        yield scrapy.Request(url=url, callback=self.parse)