无法跟踪 Scrapy 中的链接答案

【问题标题】：Not being able to follow links in Scrapy无法跟踪 Scrapy 中的链接
【发布时间】：2021-10-29 13:56:52
【问题描述】：

当我运行我的 scrapy 网络爬虫时，它没有按照页面来抓取我代码中的数据。

import scrapy
from ..items import YellowpagesItem

class YSpider(scrapy.Spider):
    name = 'yp2'
    allowed_domains = ['yellowpages.com']
    start_urls = [
        'https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Conshohocken%2C+PA'
    ]

    def parse(self, response):
        for link in response.css('a.businesss-name'):
            yield response.follow(link.attrib.get('href'), callback=self.parse_business)

    def parse_business(self, response):
        item = YellowpagesItem()
        item['name'] = response.css('h1::text').get()
        item['phone'] = response.css('p.phone::text').get()
        item['street'] = response.css('h2 > span::text').get()
        item['city_state'] = response.css('div.contact > h2.address::text').get()
        item['tags'] = ','.join([item.get() for item in response.css('p.cats > a::text')])
        item['email'] = response.css('a.email-business').attrib.get('href')
        yield item

【问题讨论】：

标签： python python-3.x web-scraping scrapy

【解决方案1】：

选择器上的简单错字，应该是a.business-name。

    def parse(self, response):
        for link in response.css('a.businesss-name'):
        yield response.follow(link.attrib.get('href'), callback=self.parse_business)

如果你还不知道，你可以在 Scrapy shell 上测试你的选择器和跟随功能，这样可以避免这种情况，在你的终端输入：

scrapy shell 'https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Conshohocken%2C+PA'

然后像这样测试选择器：

>> response.css('a.business-name')
<< 
>> response.css('a.business-name')
<< [<Selector xpath="descendant-or-self:...>,...]
>> response.follow(response.css('a.business-name::attr(href)').get())
<< <GET https://www.yellowpages.com/conshohocken-pa/mip/tony-joes-pizzeria-10728468?lid=1002028703627>

【讨论】：