Scrapy 和 xpath 来抓取我的网站并导出 URL - 我做错了什么？

【问题标题】：Scrapy and xpath to crawl my site and export URLs - what am I doing wrong?Scrapy 和 xpath 来抓取我的网站并导出 URL - 我做错了什么？
【发布时间】：2016-04-13 17:18:06
【问题描述】：

我正在尝试设置一个基本的 Scrapy 来抓取我的网站并提取我网站的所有页面 URL。我认为这将相当容易。

这是我的 items.py，复制自教程：

from scrapy.item import Item, Field
class Website(Item):
    name = Field()
    description = Field()
    url = Field()

这是我的 Spider，在教程中命名为 example.py。

from scrapy.spiders import Spider
from scrapy.selector import Selector

from cspenn.items import Website


class DmozSpider(Spider):
    name = "cspenn"
    allowed_domains = ["christopherspenn.com"]
    start_urls = ["http://www.christopherspenn.com/"]

    def parse(self, response):

        sel = Selector(response)
        sites = sel.xpath('//a')
        items = []

        for site in sites:
            item = Website()
            item['name'] = site.xpath('a/text()').extract()
            item['url'] = site.xpath('a/@href').extract()
            item['description'] = site.xpath('text()').re('-\s[^\n]*\\r')
            items.append(item)

        return items

我从机器人得到的回报是：

scrapy crawl cspenn
2016-04-13 13:15:25 [scrapy] INFO: Scrapy 1.0.5 started (bot: cspenn)
2016-04-13 13:15:25 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-04-13 13:15:25 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'cspenn.spiders', 'SPIDER_MODULES': ['cspenn.spiders'], 'BOT_NAME': 'cspenn'}
2016-04-13 13:15:25 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-04-13 13:15:26 [boto] DEBUG: Retrieving credentials from metadata server.
2016-04-13 13:15:27 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/boto/utils.py", line 210, in retry_url
    r = opener.open(req, timeout=timeout)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error timed out>
2016-04-13 13:15:27 [boto] ERROR: Unable to read instance data, giving up
2016-04-13 13:15:27 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-04-13 13:15:27 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-04-13 13:15:27 [scrapy] INFO: Enabled item pipelines: 
2016-04-13 13:15:27 [scrapy] INFO: Spider opened
2016-04-13 13:15:27 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-04-13 13:15:27 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-04-13 13:15:27 [scrapy] DEBUG: Crawled (200) <GET http://www.christopherspenn.com/> (referer: None)
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] DEBUG: Scraped from <200 http://www.christopherspenn.com/>
{'description': [], 'name': [], 'url': []}
2016-04-13 13:15:27 [scrapy] INFO: Closing spider (finished)
2016-04-13 13:15:27 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 222,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 14302,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 4, 13, 17, 15, 27, 262789),
 'item_scraped_count': 93,
 'log_count/DEBUG': 96,
 'log_count/ERROR': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2016, 4, 13, 17, 15, 27, 77084)}
2016-04-13 13:15:27 [scrapy] INFO: Spider closed (finished)

我做错了什么？我几乎完全按照教程进行操作。所需的输出是标题、页面 URL 和描述的 CSV 文件。

【问题讨论】：

    标签： python xpath web-scraping scrapy


    【解决方案1】：

    您没有正确创建特定于上下文的 xpath。您已经在上下文中拥有a，在site 变量内，无需将a 预先添加到循环内的XPath 表达式中：

    sel = Selector(response)
sites = sel.xpath('//a')
for site in sites:
    item = Website()
    item['name'] = site.xpath('text()').extract()
    item['url'] = site.xpath('@href').extract()
    item['description'] = site.xpath('text()').re('-\s[^\n]*\\r')
    yield item

    而且，由于输出中也有空的描述，我怀疑正则表达式也需要调整。不过，这取决于您尝试从链接文本中提取的具体内容。

    【讨论】：

    • 我基本上是在寻找整个站点 URL 的干净导出。抓取整个站点并导出所有页面 URL。谢谢！
    • @ChristopherPenn 好的，让我们一步一步来，你现在得到什么输出？你现在看到非空的name 和url了吗？
    • 我愿意！ {'description': [], 'name': [u'营销白带'], 'url': [u'christopherspenn.com/buy-the-marketing-white-belt-book/']}
    • @ChristopherPenn 好的，主要问题解决了。现在，您想在description 字段中看到什么？谢谢。
    • 我其实并不关心它——问题是我认为这个蜘蛛会遍历我的整个网站
    猜你喜欢
    • 2019-09-09
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2012-09-20
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode