如何绕过被“Scrapy”阻止

Question

背景：

我正计划购买汽车，并想监控价格。 我想使用Scrapy 为我做这件事。但是该网站阻止了我的代码执行此操作。

MWE/代码：

#!/usr/bin/python3

# from bs4 import BeautifulSoup
import scrapy    # adding scrapy to our file

urls = ['https://www.carsales.com.au/cars/volkswagen/golf/7-series/wagon-bodystyle/diesel-fueltype/']

class HeadphoneSpider(scrapy.Spider):   # our class inherits from scrapy.Spider
    name = "headphones"
    def start_requests(self):
        urls = ['https://www.carsales.com.au/cars/volkswagen/golf/7-series/wagon-bodystyle/diesel-fueltype/']# list to enter our urls
        # urls = ['https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Daps&field-keywords=headphones&rh=i%3Aaps%2Ck%3Aheadphones&ajr=2']

        for url in urls:            
            yield scrapy.Request(url=url, callback=self.parse)  # we will explain the callback soon


    def parse(self, response):
        img_urls = response.css('img::attr(src)').extract()
        with open('urls.txt', 'w') as f:
            for u in img_urls:
                f.write(u + "\n")


    def main():
        scraper()

输出：

   ...some stuff above it
   2020-01-10 00:37:59 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.carsales.com.au/cars/volkswagen/golf/7-series/wagon-bodystyle/diesel-fueltype/>: HTTP status code is not handled or not allowed
   ..some more stuff underneath

问题：

我只是不知道如何绕过这个not allowed 来解析价格、公里数等。这会让我的生活变得更轻松。我怎样才能越过这个街区？ FWIW 我也用 BeautifulSoup 进行了尝试，但没有成功。

Answer 1

有多种方法可以避免在抓取时被网站阻止：

• 设置ROBOTSTXT_OBEY = False
• 在您的请求之间增加 DOWNLOAD_DELAY，例如 3 到 4 秒，具体取决于站点行为
• 将CONCURRENT_REQUESTS 设置为1
• 通过自定义 proxy_middleware 使用代理或代理池并服务于原因
• 在请求中携带站点 cookie，以便站点无法识别机器人行为

您可以依次尝试这些解决方案