网络爬虫不跟随链接答案

【问题标题】：Web Crawler not following the links网络爬虫不跟随链接
【发布时间】：2019-04-06 06:33:18
【问题描述】：

我想使用 Scrapy 抓取新闻网站。该代码从当前链接检索相关新闻，但不跟随下一页链接。新闻网站具有以下链接属性

我正在关注的代码：

import scrapy 

class fakenews(scrapy.Spider):
    name = "bb8"
    allowed_domains = ["snopes.com"]
    start_urls = [
        "https://www.snopes.com/fact-check/category/science/"

    ]

    custom_settings = {'FEED_URI': "fakenews_%(time)s.csv",
                       'FEED_FORMAT': 'csv'}

    def parse(self, response):

        name1 = input(" Please enter input :     ")
        name1 = name1.lower()

        links =response.xpath("//div[@class='media-list']/article/a/@href").extract()
        headers = response.xpath('//div[@class="media-body"]/h5/text()').extract()
        headers1 = [c.strip().lower() for c in headers]

        raw_data=zip(headers1,links)
        for header, link in raw_data:

            p = header
            l=link
            if name1 in p:
                scrap_info3 = {'page': response.url, 'title': header, 'link':l}

                yield scrap_info3

                next_page = response.css("//a[@class='btn-next btn']/@href").get()
                if next_page is not None:
                    next_page = response.urljoin(next_page)
                    yield scrapy.Request(next_page, callback=self.parse)

虽然从当前页面返回信息但也显示错误。

对于我输入的输入：NASA

【问题讨论】：

标签： python scrapy web-crawler

【解决方案1】：

主要错误是您有css 函数和xpath next_page 选择器：

next_page = response.css("//a[@class='btn-next btn']/@href").get()

下一个问题是您在for 循环中产生了下一页的请求。这会导致调用大量重复请求。

所以我想这些变化：

def parse(self, response):
    name1 = input(" Please enter input :     ")
    name1 = name1.lower()

    links = response.xpath("//div[@class='media-list']/article/a/@href").extract()
    headers = response.xpath('//div[@class="media-body"]/h5/text()').extract()
    headers1 = [c.strip().lower() for c in headers]

    # my changes since this moment:
    raw_data = zip(headers1, links)
    # use less variables in loop (yes, just cosmetic, but your code will more readable)
    for header, link in raw_data:
        if name1 in header:
            yield {'page': response.url, 'title': header, 'link': link}

    # use proper selector here
    next_page = response.css("a.btn-next::attr(href)").get()
    # move all this block out of for loop
    if next_page:
        yield response.follow(next_page)

【讨论】：