【发布时间】:2023-03-13 04:35:01
【问题描述】:
我正在尝试抓取 TripAdvisor 的评论,但我找不到 Xpath 让它动态浏览所有页面。我尝试了yield 和回调,但问题是我找不到转到下一页的行的 xpath。我说的是This site
这是我的代码(已更新):
from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.http import Request
from scrapingtest.items import ScrapingTestingItem
class scrapingtestspider(Spider):
name = "scrapytesting"
allowed_domains = ["tripadvisor.in"]
base_uri = "tripadvisor.in"
start_urls = [
"http://www.tripadvisor.in/Hotel_Review-g297679-d300955-Reviews-Ooty_Fern_Hill_A_Sterling_Holidays_Resort-Ooty_Tamil_Nadu.html"]
output_json_dict = {}
def parse(self, response):
sel = Selector(response)
sites = sel.xpath('//a[contains(text(), "Next")]/@href').extract()
items = []
i=0
for sites in sites:
item = ScrapingTestingItem()
#item['reviews'] = sel.xpath('//p[@class="partial_entry"]/text()').extract()
item['subjects'] = sel.xpath('//span[@class="noQuotes"]/text()').extract()
item['stars'] = sel.xpath('//*[@class="rate sprite-rating_s rating_s"]/img/@alt').extract()
item['names'] = sel.xpath('//*[@class="username mo"]/span/text()').extract()
items.append(item)
i+=1
sites = sel.xpath('//a[contains(text(), "Next")]/@href').extract()
if(sites and len(sites) > 0):
yield Request(url="tripadvisor.in" + sites[i], callback=self.parse)
else:
yield items
【问题讨论】:
-
您的代码中没有错误吗?我得到了一个
IndexError,因为网站的索引非常大...... -
这是为什么还是因为列表中只有 1 个链接?我确实得到了一个错误。但我认为有 70 页并不重要
-
我在scrapy shell中测试了它,它循环通过了吗?
标签: python python-2.7 xpath web-scraping scrapy