如何在scrapy中挖掘到带有链接的站点

Question

我为scrapy写了一个python类，如下：

from scrapy.item import Item, Field
from scrapy.spider import Spider
from scrapy.selector import Selector

class MyItem(Item):
    content = Field()

class TestSpider(Spider):
    name = 'test_spider'
    allowed_domains = ['www.hamshahrionline.ir']
    start_urls = ['http://www.hamshahrionline.ir/']

    def parse(self, response):
        sel = Selector(response)
        h4 = sel.xpath("//h4/a/text()").extract()

    for t4 in h4:
            title4 = MyItem()
            title4['content'] = t4
            yield title4

我想知道如何挖掘到该内容的链接并浏览其他页面？

第二个问题：

您能告诉我如何从网站上逐页查看链接的内容吗？

Answer 1

您需要使用 CrawlSpider 而不是常规的 Spider 类。它支持Rules和LinkExtractors的思想，可以提取链接并关注它们。

示例（按照所有包含service/\w+ 的链接）：

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.item import Item, Field


class MyItem(Item):
    content = Field()


class TestSpider(CrawlSpider):
    name = 'test_spider'
    allowed_domains = ['hamshahrionline.ir']
    start_urls = ['http://www.hamshahrionline.ir']

    rules = (
        Rule(SgmlLinkExtractor(allow=('service/\w+', ), ), callback='parse_item'),
    )

    def parse_item(self, response):
        print response.url

        item = MyItem()
        item['content'] = response.body
        return item