如何使用scrapy（正确的css选择器）从整个页面获取href？

Question

我正在尝试抓取一个房地产网站：https://www.nepremicnine.net/oglasi-prodaja/slovenija/hisa/。我想获取隐藏在房屋图像标签中的 href：

我想为整个页面（和其他页面）获取此信息。这是我编写的不返回任何内容的代码（例如空字典）：

import scrapy
from ..items import RealEstateSloItem
import time

# first get all the URLs that have more info on the houses
# next crawl those URLs to get the desired information

class RealestateSpider(scrapy.Spider):
    # allowed_domains = ['nepremicnine.net']
    name = 'realestate'
    page_number = 2
    # page 1 url
    start_urls = ['https://www.nepremicnine.net/oglasi-prodaja/slovenija/hisa/1/']

    def parse(self, response):

        items = RealEstateSloItem()  # create it from items class --> need to store it down

        all_links = response.css('a.slika a::attr(href)').extract()

        items['house_links'] = all_links

        yield items

        next_page = 'https://www.nepremicnine.net/oglasi-prodaja/slovenija/hisa/' + str(RealestateSpider.page_number) +  '/'
        #print(next_page)

        # if next_page is not None: # for buttons
        if RealestateSpider.page_number < 180:  # then only make sure to go to the next page
            # if yes then increase it --> for paginations
            time.sleep(1)
            RealestateSpider.page_number += 1
            # parse automatically checks for response.follow if its there when its done with this page
            # this is a recursive function
            # follow next page and where should it after following
            yield response.follow(next_page, self.parse)  # want it to go back to parse

你能告诉我我在这里用 css 选择器做错了什么吗？

Answer 1

您的选择器正在a.slika 内寻找a 元素。这应该可以解决您的问题：

all_links = response.css('a.slika ::attr(href)').extract()

这些将是相对 url，您可以使用 response.urljoin() 以使用您的响应 url 作为基域来构建绝对 url。