Scrapy--无法将项目导入我的蜘蛛（没有模块名称 behance.items）

Question

我是scrapy的新手，在运行蜘蛛爬行时behance

import scrapy
from scrapy.selector import Selector
from behance.items import BehanceItem
from selenium import webdriver
from scrapy.http import TextResponse

from scrapy.crawler import CrawlerProcess

class DmozSpider(scrapy.Spider):
    name = "behance"
    #allowed_domains = ["behance.com"]
    start_urls = [

        "https://www.behance.net/gallery/29535305/Mind-Your-Monsters",


    ]


    def __init__ (self):
        self.driver = webdriver.Firefox()

    def parse(self, response):

            self.driver.get(response.url)
            response = TextResponse(url=response.url, body=self.driver.page_source, encoding='utf-8')
            item = BehanceItem()
            hxs = Selector(response)

            item['link'] = response.xpath("//div[@class='js-project-module-image-hd project-module module image project-module-image']/@data-hd-src").extract()

            yield   item

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(DmozSpider)
process.start()

当我运行爬虫时，我在命令行上收到以下错误

Traceback（最近一次调用最后一次）： 文件“/home/davy/behance/behance/spiders/behance_spider.py”，第 3 行，在 从 behance.items 导入 BehanceItem

ImportError: 没有名为 behance.items 的模块

我的目录结构：

behance/
├── behance
│   ├── __init__.py
│   ├── items.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders
│       ├── __init__.py
│       └── behance_spider.py
-── scrapy.cfg

Answer 1

尝试使用以下命令运行您的蜘蛛：

scrapy crawl behance

或者改变你的蜘蛛文件：

import scrapy
from scrapy.selector import Selector
from behance.items import BehanceItem
from selenium import webdriver
from scrapy.http import TextResponse

from scrapy.crawler import CrawlerProcess

class BehanceSpider(scrapy.Spider):
    name = "behance"
    allowed_domains = ["behance.com"]
    start_urls = [

    "https://www.behance.net/gallery/29535305/Mind-Your-Monsters",


]


    def __init__ (self):
        self.driver = webdriver.Firefox()

    def parse(self, response):

        self.driver.get(response.url)
        response = TextResponse(url=response.url, body=self.driver.page_source, encoding='utf-8')
        item = BehanceItem()
        hxs = Selector(response)

        item['link'] = response.xpath("//div[@class='js-project-module-image-hd project-module module image project-module-image']/@data-hd-src").extract()

        yield   item

并在您的 settings.py 文件所在的目录中创建另一个 python 文件。

run.py

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

process.crawl("behance")
process.start()

现在像运行普通 python 脚本一样运行这个文件。 python run.py

Answer 2

你可以将它添加到你的 python 路径中：

export PYTHONPATH=$PYTHONPATH:/home/davy/behance/