【发布时间】：2015-02-15 23:23:52
【问题描述】：
我正在尝试运行我的 Base spider 进行动态分页，但我没有成功爬取。我使用了 selenium ajax 动态分页。我使用的网址是：http://www.demo.com。这是我的代码：
# -*- coding: utf-8 -*-
import scrapy
import re
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector
from scrapy.spider import BaseSpider
from demo.items import demoItem
from selenium import webdriver
def removeUnicodes(strData):
if(strData):
#strData = strData.decode('unicode_escape').encode('ascii','ignore')
strData = strData.encode('utf-8').strip()
strData = re.sub(r'[\n\r\t]',r' ',strData.strip())
#print 'Output:',strData
return strData
class demoSpider(scrapy.Spider):
name = "demourls"
allowed_domains = ["demo.com"]
start_urls = ['http://www.demo.com']
def __init__(self):
self.driver = webdriver.Firefox()
def parse(self, response):
print "*****************************************************"
self.driver.get(response.url)
print response.url
print "______________________________"
hxs = Selector(response)
item = sumItem()
finalurls = []
while True:
next = self.driver.find_element_by_xpath('//div[@class="showMoreCars hide"]/a')
try:
next.click()
# get the data and write it to scrapy items
item['pageurl'] = response.url
item['title'] = removeUnicodes(hxs.xpath('.//h1[@class="page-heading"]/text()').extract()[0])
urls = hxs.xpath('.//a[@id="linkToDetails"]/@href').extract()
print '**********************************************2***url*****************************************',urls
for url in urls:
print '---------url-------',url
finalurls.append(url)
item['urls'] = finalurls
except:
break
self.driver.close()
return item
我的 items.py 是
from scrapy.item import Item, Field
class demoItem(Item):
page = Field()
urls = Field()
pageurl = Field()
title = Field()
当我尝试抓取它并将其转换为 json 时，我的 json 文件为：
[{"pageurl": "http://www.demo.com", "urls": [], "title": "demo"}]
由于动态加载，我无法抓取所有网址。
【问题讨论】：
标签： python ajax selenium pagination scrapy