【发布时间】：2019-01-26 02:47:21
【问题描述】：
我正在尝试从以下网站抓取航空公司的一些数据：http://www.airlinequality.com/airline-reviews/airasia-x[1]。
我设法获得了我需要的数据，但我在网页上的分页上遇到了困难。我正在尝试获取评论的所有标题（不仅仅是第一页中的标题）。
页面的链接格式为：http://www.airlinequality.com/airline-reviews/airasia-x/page/3/ 其中3 是页面的编号。
我尝试遍历这些 URL 以及以下代码，但无法通过分页抓取。
# follow pagination links
for href in response.css('#main > section.layout-section.layout-2.closer-top > div.col-content > div > article > ul li a'):
yield response.follow(href, self.parse)
如何解决？
import scrapy
import re # for text parsing
import logging
from scrapy.crawler import CrawlerProcess
class AirlineSpider(scrapy.Spider):
name = 'airlineSpider'
# page to scrape
start_urls = ['http://www.airlinequality.com/review-pages/a-z-airline-reviews/']
def parse(self, response):
# take each element in the list of the airlines
for airline in response.css("div.content ul.items li"):
# go inside the URL for each airline
airline_url = airline.css('a::attr(href)').extract_first()
# Call parse_airline
next_page = airline_url
if next_page is not None:
yield response.follow(next_page, callback=self.parse_article)
# follow pagination links
for href in response.css('#main > section.layout-section.layout-2.closer-top > div.col-content > div > article > ul li a'):
yield response.follow(href, self.parse)
# to go to the pages inside the links (for each airline) - the page where the reviews are
def parse_article(self, response):
yield {
'appears_ulr': response.url,
# use sub to replace \n\t \r from the result
'title': re.sub('\s+', ' ', (response.css('div.info [itemprop="name"]::text').extract_first()).strip(' \t \r \n').replace('\n', ' ') ).strip(),
'reviewTitle': response.css('div.body .text_header::text').extract(),
#'total': response.css('#main > section.layout-section.layout-2.closer-top > div.col-content > div > article > div.pagination-total::text').extract_first().split(" ")[4],
}
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'FEED_FORMAT': 'json',
'FEED_URI': 'air_test.json'
})
# minimizing the information presented on the scrapy log
logging.getLogger('scrapy').setLevel(logging.WARNING)
process.crawl(AirlineSpider)
process.start()
要遍历航空公司，我使用以下代码解决了它： 它使用上面的代码：
req = Request("http://www.airlinequality.com/review-pages/a-z-airline-reviews/" , headers={'User-Agent': 'Mozilla/5.0'})
html_page = urlopen(req)
soupAirlines = BeautifulSoup(html_page, "lxml")
URL_LIST = []
for link in soupAirlines.findAll('a', attrs={'href': re.compile("^/airline-reviews/")}):
URL_LIST.append("http://www.airlinequality.com"+link.get('href'))
【问题讨论】：
标签： python web-scraping pagination scrapy