【发布时间】：2020-02-08 01:37:30
【问题描述】：
我正在尝试创建一个scrapy 脚本，旨在获取有关媒体网站上各个帖子的信息。 现在，不幸的是，它需要 3 个深度的链接。每年链接，该年中的每个月，然后是月份中的每一天链接。
我已经设法为每年、当年的每个月和每一天获取每个单独的链接。但是，我似乎无法处理各个日页面。
我不完全确定我是否混淆了使用规则和使用带有回调的函数来获取链接。关于如何递归处理这种类型的分页没有太多指导。我尝试过使用函数和 response.follow 本身却无法让它运行。
parse_item 函数字典是必需的，因为各个日页面上的几篇文章有几种不同的标题分类方法，令人讨厌。所以我创建了一个函数来获取标题，而不管获取标题所需的实际 XPATH。
最后一个函数 get_tag 是必需的，因为在每篇文章中都需要获取标签。
如果能深入了解如何完成最后一步并让各个链接通过 parse_item 函数（shell o），我将不胜感激。我应该说没有比我在 shell 中看到的明显错误。
如有任何进一步的信息，请告诉我。
谢谢！
代码：
import scrapy
from medium.items import MediumItem
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class DataSpider(CrawlSpider):
name = 'data'
allowed_domains = ['medium.com']
start_urls = ['https://medium.com/tag/python/archive/']
rules = (
Rule(
LinkExtractor(
allow=(),
restrict_xpaths=('//div[@class="timebucket u-inlineBlock u-width50"]/a')), follow=True, callback='years'),
Rule(
LinkExtractor(
allow=(),
restrict_xpaths=('//div[@class="timebucket u-inlineBlock u-width80"]/a')), follow=True, callback='months' ),
Rule(
LinkExtractor(
allow=(),restrict_xpaths=('//div[@class="timebucket u-inlineBlock u-width35"]/a')),callback='days'
))
def years(self,response):
for year in response.xpath('//div[@class="timebucket u-inlineBlock u-width50"]/a/@href').extract():
yield response.follow(year, callback=self.months)
def months(self,response):
for month in response.xpath('//div[@class="timebucket u-inlineBlock u-width50"]/a/@href').extract():
yield response.follow(month,callback=self.days)
def days(self,response):
for day in response.xpath('//div[@class="timebucket u-inlineBlock u-width35"]/a/@href').extract():
yield response.follow(day,callback=self.parse_item)
def parse_item(self,response):
dic = {}
dic['title1'] = 'h3[@class="graf graf--h3 graf-after--figure graf--title"]/text()'
dic['title2'] = 'h3[@class="graf graf--h3 graf-after--figure graf--trailing graf--title"]/text()'
dic['title3'] = 'h3[@class="graf graf--h3 graf-after--h4 graf--trailing graf--title"]/text()'
dic['title4'] = 'h3[@class="graf graf--h3 graf--leading graf--title"]/text()'
dic['title5'] = 'h3[@class="graf graf--h3 graf-after--figure graf--title"]/span[@class="markup--anchor markup--h3-anchor"]/text()'
dic['title6'] = 'h3[@class="graf graf--h3 graf-after--h3 graf--trailing"]/text()'
articles = response.xpath('//div[@class="postArticle postArticle--short js-postArticle js-trackPostPresentation js-trackPostScrolls"]')
def title(dict, articles):
for article in articles:
for num in dic.keys():
if article.xpath('.//' + dic.get(f'{num}')).extract_first() == None:
continue
else:
title = article.xpath('.//' + dic.get(f'{num}')).extract_first()
return title
for article in articles:
item = MediumItem()
item['date'] = article.css('time::text').extract_first() + ' 2019'
item['read'] = article.css("span::attr(title)").extract_first()
item['publication'] = article.xpath('.//a[@class="ds-link ds-link--styleSubtle link--darken link--accent u-accentColor--textNormal"]/text()').extract_first()
item['name'] = article.xpath('.//a[@class="ds-link ds-link--styleSubtle link link--darken link--accent u-accentColor--textNormal u-accentColor--textDarken"]/text()').extract_first()
item['claps'] = article.xpath('.//button[@class="button button--chromeless u-baseColor--buttonNormal js-multirecommendCountButton u-disablePointerEvents"]/text()').extract_first()
item['responses'] = article.xpath('.//a[@class="button button--chromeless u-baseColor--buttonNormal"]/text()').extract_first()
item['text'] = title(dict,articles)
yield item
link = article.xpath('.//a[@class="link link--darken"]/@href').extract_first()
yield response.follow(link, callback=self.get_link, meta={'item':item})
def get_tag(self,response):
item = response.meta['item']
item['tag'] = response.css("ul > li > a::text").getall()
yield item
输出
2020-02-06 16:56:38 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows
NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
2020-02-06 16:56:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET
https://medium.com/tag/python/archive/2019/03/04> (referer:
https://medium.com/tag/python/archive/2019/03) ['cached']
2020-02-06 16:56:38 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows
NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
2020-02-06 16:56:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET
https://medium.com/tag/python/archive/2019/03/03> (referer:
https://medium.com/tag/python/archive/2019/03) ['cached']
2020-02-06 16:56:38 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0
(Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36
2020-02-06 16:56:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET
https://medium.com/tag/python/archive/2019/03/02> (referer:
https://medium.com/tag/python/archive/2019/03) ['cached']
2020-02-06 16:56:38 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-06 16:56:38 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1236771,
'downloader/request_count': 1984,
'downloader/request_method_count/GET': 1984,
'downloader/response_bytes': 137377143,
'downloader/response_count': 1984,
'downloader/response_status_count/200': 1984,
'dupefilter/filtered': 52095,
'elapsed_time_seconds': 76.366135,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 2, 6, 16, 56, 38, 573687),
'httpcache/hit': 1984,
'log_count/DEBUG': 3970,
'log_count/INFO': 11,
'log_count/WARNING': 45,
'request_depth_max': 4,
'response_received_count': 1984,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 1983,
'scheduler/dequeued/memory': 1983,
'scheduler/enqueued': 1983,
'scheduler/enqueued/memory': 1983,
'start_time': datetime.datetime(2020, 2, 6, 16, 55, 22, 207552)}
2020-02-06 16:56:38 [scrapy.core.engine] INFO: Spider closed (finished)
【问题讨论】：
-
去掉三个函数
years,months,days根本不需要它们，在天xpathcallback='parse_item的规则中
标签： python web-scraping scrapy