如何评估提取的链接是否是子路径

Question

我正在使用 scrapy 抓取一些页面。我正在使用python 2.7。 蜘蛛返回响应对象，我正在检查页面上找到的 URL。我想限制蜘蛛只跟随我指定位置的子路径的 URL。

例如，我想指定蜘蛛只应遵循 www.google.com/policies/privacy/ 下方的链接

从响应对象中提取的链接遵循许多不同的约定。

例如

• ../../policies/privacy/
• ../../policies/privacy/example/collect-information.html
• #infocollect
• /intl/en_uk/policies/privacy/google_privacy_policy_en_uk.pdf
• //myaccount.google.com/
• https://support.google.com/policies/troubleshooter/2990837?hl=en-GB

我不知道该怎么做。我刚刚在字符串上使用了一个简单的查找方法。在我看来，它并不强大或那么聪明。

import scrapy

class googleSpider(scrapy.Spider):
    name = "google"
    allowed_domains = ["google.co.uk"]
    start_urls = [
        "http://www.google.co.uk/intl/en/policies/privacy/"
    ]

    def parse(self, response):
        for href in response.xpath('//a/@href').extract():
            if href.find('/policies/privacy/') != -1:
                yield scrapy.Request(response.urljoin(href), callback=self.parse_dir_contents)

    def parse_dir_contents(self, response):
        pass

Answer 1

您可以为此使用LinkExtractor。默认的规范化链接。

然后，问题是从您从.extract_links(response) 获得的链接中构建Requests

检查此scrapy shell 示例：

$ scrapy shell https://www.google.com/policies/privacy/
2016-06-22 18:03:19 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot)
(...edited...)
2016-06-22 18:03:20 [scrapy] DEBUG: Crawled (200) <GET https://www.google.com/policies/privacy/> (referer: None)
(...edited...)

>>> from scrapy.linkextractors import LinkExtractor
>>> for l in LinkExtractor().extract_links(response):
...     print(l.url)
... 
https://www.google.com/
(...edited...)
https://support.google.com/accounts/answer/32046?hl=en
https://www.google.com/trends/
https://www.youtube.com/trendsmap
https://privacy.google.com/?hl=en
https://www.google.com/policies/technologies/location-data/
https://www.google.com/policies/technologies/wallet/
https://www.google.com/policies/technologies/voice/
https://www.google.com/safetycenter/families/start/
https://www.google.com/intl/en/about/
https://www.google.com/intl/en/policies/privacy/
https://www.google.com/intl/en/policies/terms/

>>> for l in LinkExtractor().extract_links(response):
...     if response.url in l.url:
...         print(l.url)
... 
https://www.google.com/policies/privacy/
https://www.google.com/policies/privacy/frameworks/
https://www.google.com/policies/privacy/key-terms/
https://www.google.com/policies/privacy/partners/
https://www.google.com/policies/privacy/archive/
https://www.google.com/policies/privacy/example/more-relevant-search-results.html
https://www.google.com/policies/privacy/example/connect-with-people.html
https://www.google.com/policies/privacy/example/sharing-with-others.html
https://www.google.com/policies/privacy/example/ads-youll-find-most-useful.html
https://www.google.com/policies/privacy/example/the-people-who-matter-most.html
https://www.google.com/policies/privacy/example/credit-card.html
https://www.google.com/policies/privacy/example/collect-information.html
https://www.google.com/policies/privacy/example/view-and-interact-with-our-ads.html
https://www.google.com/policies/privacy/example/device-specific-information.html
https://www.google.com/policies/privacy/example/device-identifiers.html
https://www.google.com/policies/privacy/example/phone-number.html
https://www.google.com/policies/privacy/example/may-collect-and-process-information.html
https://www.google.com/policies/privacy/example/sensors.html
https://www.google.com/policies/privacy/example/wifi-access-points-and-cell-towers.html
https://www.google.com/policies/privacy/example/our-partners.html
https://www.google.com/policies/privacy/example/advertising-services.html
https://www.google.com/policies/privacy/example/linked-with-information-about-visits-to-multiple-sites.html
https://www.google.com/policies/privacy/example/provide-services.html
https://www.google.com/policies/privacy/example/maintain-services.html
https://www.google.com/policies/privacy/example/protect-services.html
https://www.google.com/policies/privacy/example/develop-new-ones.html
https://www.google.com/policies/privacy/example/protect-google-and-our-users.html
https://www.google.com/policies/privacy/example/limit-sharing-or-visibility-settings.html
https://www.google.com/policies/privacy/example/improve-your-user-experience.html
https://www.google.com/policies/privacy/example/combine-personal-information.html
https://www.google.com/policies/privacy/example/to-make-it-easier-to-share.html
https://www.google.com/policies/privacy/example/may-not-function-properly.html
https://www.google.com/policies/privacy/example/sharing.html
https://www.google.com/policies/privacy/example/removing-your-content.html
https://www.google.com/policies/privacy/example/access-to-your-personal-information.html
https://www.google.com/policies/privacy/example/legal-process.html
https://www.google.com/policies/privacy/example/we-may-share.html
https://www.google.com/policies/privacy/example/to-show-trends.html