Scrapyd 不会将包含 selenium 模块的蜘蛛项目部署到我的本地主机

Question

我正在尝试通过 scrapyd 在我的本地主机上部署我的 scrapy-spider，spider 脚本包含 selenium 模块来执行一些网络自动化任务。当我尝试部署它时出现问题。

从命令行运行 scrapyd 以运行本地主机后。我在浏览器上键入本地主机地址，并在线和收听。然后在另一个 cmd 窗口中输入 scrapyd-deploy 命令，它会像这样卡住几个小时（没有错误消息）。

$ scrapyd-deploy local
Packing version 1560251984
Deploying to project "crawler" in http://localhost:6800/addversion.json

stuck image description

顺便说一句，我在我的 Windows 机器上使用 gitbash，我也尝试过使用普通 cmd，但它仍然是无休止的等待和延迟。

在我运行 scrapyd 命令打开本地主机的第一个 cmd 窗口中，我得到了类似这样的内容。

DevTools listening on ws://127.0.0.1:9137/devtools/browser/07f179fa-02ce-4b31-a5
96-9b700654f105

devtools listening stuck image

据我了解，这似乎是无头模式下的 selenium 浏览器试图启动，但它一直在无休止地等待。

当我打开我的项目目录时，我看到了新的文件夹，如鸡蛋、项目鸡蛋信息和构建。看起来它使项目变得蛋疼，但在尝试在本地主机上部署和运行它时会遇到延迟。

这是我的蜘蛛脚本

import scrapy
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait as Webwait
from selenium.webdriver.support import expected_conditions as exco
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import random

C_options = Options()
C_options.add_argument("--disable-extensions")
C_options.add_argument("--disable-gpu")
C_options.add_argument("--headless")

class CloupAppSpider(scrapy.Spider):
    driver = webdriver.Chrome(options=C_options,
                              executable_path=r"C:\.....\chromedriver.exe")
    driver.get("https://scrapingclub.com/exercise/basic_login/")
    cookie = driver.get_cookies()
    driver.add_cookie(cookie[0])
    name = 'crawl'
    allowed_domains = ['scrapingclub.com']
    start_urls = ['https://scrapingclub.com/exercise/basic_login/']

    def __int__(self, name=None, passwd=None, *args, **kwargs):
        super(CloupAppSpider, self).__init__(*args, **kwargs)
        self.passwd = passwd
        self.name = name

    def parse(self, response):
        pword = self.passwd
        uname = self.name
        Webwait(self.driver, 10).until(exco.presence_of_element_located((By.ID, "id_name")))
        Webwait(self.driver, 10).until(exco.presence_of_element_located((By.ID, "id_password")))
        CloupAppSpider.driver.find_element_by_id("id_name").send_keys(pword)
        CloupAppSpider.driver.find_element_by_id("id_password").send_keys(uname)
        CloupAppSpider.driver.find_element_by_css_selector(".btn.btn-primary").click()
        Webwait(self.driver, 10).until(exco.presence_of_element_located((By.CLASS_NAME, "col-lg-8")))
        html = CloupAppSpider.driver.execute_script("return document.documentElement.outerHTML")
        bs_obj = BeautifulSoup(html, "html.parser")
        text = bs_obj.find("div", {"class": "col-lg-8"}).find("p")
        obj = text.get_text()
        obj = obj + str(random.randint(0, 100))
        self.driver.close()
        yield{
            'text': obj
        }

这是我的 scrapy.cfg 内容

[settings]
default = crawler.settings
[deploy:local]
url = http://localhost:6800/
project = crawler

谁能帮我解释一下我哪里出错了，我一无所知，因为我在部署时没有收到任何错误代码。它只是无休止地等待，我的猜测是它试图用硒处理。

Answer 1

我解决了。我的错误是，我在主蜘蛛类下启动了 selenium 进程，而不是在蜘蛛类中包含的 def parse 函数下。我只是重新编辑了代码如下。

import scrapy
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait as Webwait
from selenium.webdriver.support import expected_conditions as exco
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import random

C_options = Options()
C_options.add_argument("--disable-extensions")
C_options.add_argument("--disable-gpu")
C_options.add_argument("--headless")

class CloupAppSpider(scrapy.Spider):

    name = 'crawl'
    allowed_domains = ['scrapingclub.com']
    start_urls = ['https://scrapingclub.com/exercise/basic_login/']

    def __int__(self, name=None, passwd=None, *args, **kwargs):
        super(CloupAppSpider, self).__init__(*args, **kwargs)
        self.passwd = passwd
        self.name = name

    def parse(self, response):
        driver = webdriver.Chrome(options=C_options,                                  
         executable_path=r"C:\......\chromedriver.exe")
        driver.get("https://scrapingclub.com/exercise/basic_login/")
        cookie = driver.get_cookies()
        driver.add_cookie(cookie[0])
        pword = self.passwd
        uname = self.name
        Webwait(self.driver, 10).until(exco.presence_of_element_located((By.ID, "id_name")))
        Webwait(self.driver, 10).until(exco.presence_of_element_located((By.ID, "id_password")))
        CloupAppSpider.driver.find_element_by_id("id_name").send_keys(pword)
        CloupAppSpider.driver.find_element_by_id("id_password").send_keys(uname)
        CloupAppSpider.driver.find_element_by_css_selector(".btn.btn-primary").click()
        Webwait(self.driver, 10).until(exco.presence_of_element_located((By.CLASS_NAME, "col-lg-8")))
        html = CloupAppSpider.driver.execute_script("return document.documentElement.outerHTML")
        bs_obj = BeautifulSoup(html, "html.parser")
        text = bs_obj.find("div", {"class": "col-lg-8"}).find("p")
        obj = text.get_text()
        obj = obj + str(random.randint(0, 100))
        self.driver.close()
        yield{
            'text': obj
        }