【发布时间】：2017-11-26 10:06:42
【问题描述】：
我正在尝试从网页soundseasy.com.au 中抓取一些日期，但有时我会收到错误消息：
AttributeError: 'SoundseasySpider' object has no attribute 'crawler'
这是我的代码，它使用 selenium web 驱动程序（self.browser 实例）从动态页面获取数据：
import scrapy
from ProductsScraper.items import ProductDataItem, ProductDataLoader
from utilities.common import MODE_SINGLE
from utilities.DynamicPageLoader import DynamicPageLoader
def start_requests(self):
# scrape multi page data
for page_count, url in zip(self.pages_counts, self.start_urls):
yield scrapy.Request(url=url, callback=self.multi_parse,
meta={'page_count': page_count},
dont_filter=True)
def multi_parse(self, response):
"""
Method fetched the pages, gets the product url links and scrape it
by calling parse_product
"""
selector = self.get_dynamic_page(url=response.url,
page_count=response.meta.get('page_count', '1'))
product_urls = selector.xpath('//div[@class="isp_product_info"]/a/@href').extract()
self.logger.info('{} items should be scraped from the page: {},'
' scroll_count:{}'.format(len(product_urls),
response.url, response.meta.get('page_count', '1')))
for product_url in product_urls:
# construct absolute url
url = "https://www.{}{}".format(self.allowed_domains[0], product_url)
yield scrapy.Request(url=url, callback=self.parse_product, dont_filter=True)
def get_dynamic_page(self, url, page_count):
"""
Fetch dynamic page using DynamicDownloader and return selector object
"""
# construct search page url with the page count included
pages_url = url + '&page_num={}'.format(page_count)
self.logger.info("get_dynamic_page: {}".format(pages_url))
self.browser.load_page(pages_url)
return scrapy.Selector(text=self.browser.get_html_page())
我做错了什么？任何帮助表示赞赏。
编辑： 我得到了以下异常：
File "/home/user/python3.6.1/lib/python3.6/site-packages/Twisted-17.9.0-py3.6-linux-x86_64.egg/twisted/internet/defer.py", line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/user/python3.6.1/lib/python3.6/site-packages/Twisted-17.9.0-py3.6-linux-x86_64.egg/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/user/python3.6.1/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean f
ashion: Connection lost.>]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/swampblu/python3.6.1/lib/python3.6/site-packages/Twisted-17.9.0-py3.6-linux-x86_64.egg/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/home/swampblu/python3.6.1/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 66, in process_exception
spider=spider)
File "/home/swampblu/python3.6.1/lib/python3.6/site-packages/scrapy/downloadermiddlewares/retry.py", line 61, in process_exception
return self._retry(request, exception, spider)
File "/home/swampblu/python3.6.1/lib/python3.6/site-packages/scrapy/downloadermiddlewares/retry.py", line 71, in _retry
stats = spider.crawler.stats
AttributeError: 'SoundsEasySpider' object has no attribute 'crawler'
【问题讨论】：
-
能否包含导入部分？我怀疑错误的原因可能位于那里。
-
完成了，但我想如果是导入部分的问题，那么它永远不会起作用，但有时我得到了一些好的结果
-
错误消息应该显示哪一行代码有问题 - 这就是为什么你应该总是把完整的错误（Traceback）放在有问题的地方（作为文本，而不是截图）
-
好点@furas。我认为我对进口申报有误。可能是您的 selenium 浏览器加载页面的速度不够快，以至于通过了空页面或部分页面。您可以尝试添加延迟或更具体的等待指令，以确保在解析响应之前加载页面。
-
做一个班级老兄，
class SoundseasySpider(scrapy.Spider)并遵循正确的scrapy约定