Scrapy + Selenium 问题

Question

我正在尝试使用 Selenium 和 Scrapy 抓取一家著名的英国零售商的网站（请参见下面的代码）。我收到了[scrapy.core.scraper] ERROR: Spider error processing 并且不知道还能做什么（已经工作了三个小时左右）。感谢大家的支持！

import scrapy
from selenium import webdriver
from nl_scrape.items import NlScrapeItem
import time

class ProductSpider(scrapy.Spider):
    name = "product_spider"
    allowed_domains = ['newlook.com']
    start_urls = ['http://www.newlook.com/uk/womens/clothing/c/uk-womens-clothing?comp=NavigationBar%7Cmn%7Cwomens%7Cclothing#/?q=:relevance&page=1&sort=relevance&content=false']

def __init__(self):
    self.driver = webdriver.Safari()
    self.driver.set_window_size(800,600)
    time.sleep(4)

def parse(self, response):
    self.driver.get(response.url)
    time.sleep(4)

    # Collect products
    products = driver.find_elements_by_class_name('plp-item ng-scope')

    # Iterate over products; extract data and append individual features to NlScrapeItem
    for item in products:

        # Pull features
        desc = item.find_element_by_class_name('product-item__name link--nounderline ng-binding').text
        href = item.find_element_by_class_name('plp-carousel__img-link ng-scope').get_attribute('href')

        # Price Symbol removal and integer conversion
        priceString = item.find_element_by_class_name('price ng-binding').text
        priceInt = priceString.split('£')[1]
        price = float(priceInt)

        # Generate a product identifier
        identifier = href.split('/p/')[1].split('?comp')[0]
        identifier = int(identifier)

        # datetime
        dt = date.today()
        dt = dt.isoformat()

        # NlScrapeItem
        item = NlScrapeItem()

        # Append product to NlScrapeItem
        item['id'] = identifier
        item['href'] = href
        item['description'] = desc
        item['price'] = price
        item['firstSighted'] = dt
        item['lastSighted'] = dt
        yield item

    self.driver.close()

2017-08-26 15:48:38 [scrapy.core.scraper] 错误：蜘蛛错误处理 http://www.newlook.com/uk/womens/clothing/c/uk-womens-clothing?comp =NavigationBar%7Cmn%7Cwomens%7Cclothing#/?q=:relevance&page=1&sort=relevance&content=false>（引用者：无）

Traceback（最近一次调用最后一次）： _runCallbacks 中的文件“/Users/username/Documents/nl_scraping/nl_env/lib/python3.6/site-packages/twisted/internet/defer.py”，第 653 行 current.result = callback(current.result, *args, **kw) 解析中的文件“/Users/username/Documents/nl_scraping/nl_scrape/nl_scrape/spiders/product_spider.py”，第 18 行 products = driver.find_elements_by_class_name('plp-item ng-scope') NameError：名称“驱动程序”未定义

Answer 1

所以你的代码有两个问题

def parse(self, response):
    self.driver.get(response.url)
    time.sleep(4)

    # Collect products
    products = driver.find_elements_by_class_name('plp-item ng-scope')

您将self.driver 更改为driver 非常方便。那样不行。您应该在函数顶部添加

def parse(self, response):
    driver = self.driver
    driver.get(response.url)
    time.sleep(4)

    # Collect products
    products = driver.find_elements_by_class_name('plp-item ng-scope')

接下来，您在函数末尾使用了self.driver.close()。因此，一旦您处理了一个 url，您将关闭浏览器。那是错的。所以删除那行。