【发布时间】：2020-03-12 21:52:35
【问题描述】：
我一直在从事房地产网站的网络抓取项目。这个想法是收集网页中所有属性的价格和一般信息。
由于我在 Jupyter Notebook 中使用 Scrapy 框架，因此这是我目前的代码：
class QuotesSpider(scrapy.Spider):
name = "quotes"
link = 'https://www.vivareal.com.br/venda/sp/sao-paulo/?pagina={number}#onde=BR-Sao_Paulo-NULL-Sao_Paulo&tipos=apartamento_residencial'
start_urls = ['https://www.vivareal.com.br/venda/sp/sao-paulo/apartamento_residencial/#onde=BR-Sao_Paulo-NULL-Sao_Paulo&tipos=apartamento_residencial',
]
for i in range(1,3):
start_urls.append(link.format(number=i))
#print(start_urls)
start_urls = ['https://www.vivareal.com.br/venda/sp/sao-paulo/apartamento_residencial/#onde=BR-Sao_Paulo-NULL-Sao_Paulo&tipos=apartamento_residencial',
]
custom_settings = {
'LOG_LEVEL': logging.WARNING,
'ITEM_PIPELINES': {'__main__.JsonWriterPipeline': 1}, # Used for pipeline 1
'FEED_FORMAT':'json', # Used for pipeline 2
'FEED_URI': 'quoteresult.json', # Used for pipeline 2
'CLOSESPIDER_PAGECOUNT': 3
}
def parse(self, response):
display(response.body)
table_rows = response.css('div.property-card__main-content') #//*[@id="js-site-main"]/div[2]/div
for quote in table_rows:
time.sleep(0.2)
yield {
'address': quote.css('span.property-card__address::text').extract_first(),#.re(r'.*'),
'title': quote.css('a.property-card__title::text').extract_first(),
'area': quote.css('span.js-property-card-detail-area::text').extract_first(),
'price': quote.css('div.js-property-card__price-small::text').extract_first(),
'cond_price': quote.css('strong.js-condo-price::text').extract_first(),
'bedrooms': quote.css('span.property-card__detail-value::text').extract()[1],
'bathrooms': quote.css('span.property-card__detail-value::text').extract()[3],
'amenities': quote.css('.amenities__item::attr(title)').extract(),
#'pictures': quote.css('div.carousel__item-wrapper::text').extract()[2]
}
此代码在第一页中运行良好，包含 36 个属性。但是，当它进入下一页时，网站需要一些时间来更新内容，我得到了一个混乱的网站版本（第一页和第二页的属性混合在一起）。
我阅读了一些关于下一个的示例，并用以下内容补充了我的代码
class QuotesSpider(scrapy.Spider):
name = "quotes"
link = 'https://www.vivareal.com.br/venda/sp/sao-paulo/?pagina={number}#onde=BR-Sao_Paulo-NULL-Sao_Paulo&tipos=apartamento_residencial'
start_urls = ['https://www.vivareal.com.br/venda/sp/sao-paulo/apartamento_residencial/#onde=BR-Sao_Paulo-NULL-Sao_Paulo&tipos=apartamento_residencial',
]
for i in range(1,3):
start_urls.append(link.format(number=i))
#print(start_urls)
start_urls = ['https://www.vivareal.com.br/venda/sp/sao-paulo/apartamento_residencial/#onde=BR-Sao_Paulo-NULL-Sao_Paulo&tipos=apartamento_residencial',
]
custom_settings = {
'LOG_LEVEL': logging.WARNING,
'ITEM_PIPELINES': {'__main__.JsonWriterPipeline': 1}, # Used for pipeline 1
'FEED_FORMAT':'json', # Used for pipeline 2
'FEED_URI': 'quoteresult.json', # Used for pipeline 2
'CLOSESPIDER_PAGECOUNT': 3
}
def parse(self, response):
display(response.body)
table_rows = response.css('div.property-card__main-content') #//*[@id="js-site-main"]/div[2]/div
for quote in table_rows:
time.sleep(0.2)
yield {
'address': quote.css('span.property-card__address::text').extract_first(),#.re(r'.*'),
'title': quote.css('a.property-card__title::text').extract_first(),
'area': quote.css('span.js-property-card-detail-area::text').extract_first(),
'price': quote.css('div.js-property-card__price-small::text').extract_first(),
'cond_price': quote.css('strong.js-condo-price::text').extract_first(),
'bedrooms': quote.css('span.property-card__detail-value::text').extract()[1],
'bathrooms': quote.css('span.property-card__detail-value::text').extract()[3],
'amenities': quote.css('.amenities__item::attr(title)').extract(),
#'pictures': quote.css('div.carousel__item-wrapper::text').extract()[2]
}
# next_page = /page-{}/ where {} number of page.
next_page = response.xpath('//*[@id="js-site-main"]/div[2]/div/section/div[2]/div[2]/div/ul/li[9]/a').extract_first()
# next_page = https://sanet.st/page-{}/ where {} number of page.
next_page = response.urljoin(next_page)
# If next_page have value
if next_page:
# Recall parse with url https://sanet.st/page-{}/ where {} number of page.
yield scrapy.Request(url=next_page, callback=self.parse)
但是，我只获得第一页结果，并且我无法理解如何限制获得的页数，因为我无法访问循环计数。换句话说，我如何定义它什么时候停止？
谢谢你
【问题讨论】：
-
您可以“让出”所有最后带有“?page=x”的网址。否则它一次只会做一个。
-
你能解释清楚一点吗，@pguardiario？谢谢
标签： javascript python scrapy