将图像 src 包含到 LinkExtractor Scrapy CrawlSpider

Question

我正在对网站进行爬网，我正在使用来自 scrapy 的 LinkExtractor 来爬网链接并确定其响应状态。

此外，我还想使用链接提取器从站点获取图像 src。我有一个代码，它适用于网站网址，但我似乎无法获取图像。因为它不会登录控制台。

handle_httpstatus_list = [404,502]
# allowed_domains = [''mydomain']

start_urls = ['somedomain.com/']

http_user = '###'
http_pass = '#####'

rules = (
    Rule(LinkExtractor(allow=('domain.com',),canonicalize = True, unique = True), process_links='filter_links', follow = False, callback='parse_local_link'),
    Rule(LinkExtractor(allow=('cdn.domain.com'),tags = ('img',),attrs=('src',),canonicalize = True, unique = True), follow = False, callback='parse_image_link'),
)

def filter_links(self,links):
    for link in

def parse_local_link(self, response):
    if response.status != 200:
        item = LinkcheckerItem()
        item['url'] = response.url
        item['status'] = response.status
        item['link_type'] = 'local'
        item['referer'] = response.request.headers.get('Referer',None)
        yield item

def parse_image_link(self, response):
    print "Got image link"
    if response.status != 200:
        item = LinkcheckerItem()
        item['url'] = response.url
        item['status'] = response.status
        item['link_type'] = 'img'
        item['referer'] = response.request.headers.get('Referer',None)
        yield item

Answer 1

如果有人有兴趣继续使用 CrawlSpider 和 LinkExtractors，只需添加 kwarg deny_extensions，即替换：

    Rule(LinkExtractor(allow=('cdn.domain.com'),tags = ('img',),attrs=('src',),canonicalize = True, unique = True), follow = False, callback='parse_image_link'),

与

    Rule(LinkExtractor(allow=('cdn.domain.com'),deny_extensions=set(), tags = ('img',),attrs=('src',),canonicalize = True, unique = True), follow = False, callback='parse_image_link')

不设置此参数时，默认为scrapy.linkextractors.IGNORED_EXTENSIONS，包含jpeg、png等扩展名。这意味着链接提取器会避免找到包含所述扩展名的链接。

Answer 2

我已经使用 Scarpy 2 年多了，我总是使用 start_requests() 方法开始抓取 URL，而不是 start_urls 和 LinkExtractor

不要被上面弄糊涂了，直接用这个

class MySpider(scrapy.Spider):
    name = "myspider"

    def start_requests(self):

        urls_to_scrape = ["abc.com", "abc.com2"]

        for url in urls_to_scrape:

            yield Request(url=url, callback=self.my_callback)


    def my_callback(self, response):

        for img in response.css("img"):

            image_here = img.css("::attr(src)").extract_first()