如何使用xpath和scrapy提取不同页面（子页面）上的所有图像

Question

我正在尝试从https://www.rawson.co.za/property/for-sale/cape-town 中提取图像的所有 URL 列表。 但是，所有图像都可以在不同的页面上找到，而不是在主页面上。 我一直在使用 xpath 来检索其他所需的字段。

我不太确定如何从这些子页面中检索列表中的所有 URL。这是我尝试过的：

    class PropDataSpider(scrapy.Spider):
        name = "rawson"
        start_urls = ['https://www.rawson.co.za/property/for-sale/cape-town']


        def parse(self, response):
            propertes = response.xpath("//div[@class='card__main']")
            for prop in propertes:
                title = prop.xpath(
                    "./div[@class='card__body']/h3[@class='card__title']/a/text()").extract_first()
                price = prop.xpath(
                    "./div[@class='card__body']/div[@class='card__footer card__footer--primary']/div[@class='card__price']/text()").extract_first()
                description = prop.xpath(
                    "./div[@class='card__body']/div[@class='card__synopsis']/p/text()").extract_first()
                bedrooms = prop.xpath(
                    "./div[@class='card__body']/div[@class='card__footer card__footer--primary']/div[@class='features features--inline']/ol[@class ='features__list']/li[@class ='features__item'][1]/div[@class='features__label']/text()").extract_first()

    ...



                images = ['https://' + img for img in prop.xpath(
                    "main[@class='l-main']/section[@class='l-section']/div[@class='l-wrapper']/div[@class='l-section__main']/div[@class ='content-block content-block--flat']/div[@class ='gallery gallery--flat js-lightbox']/div[@ class ='row row--flat']/div[@class ='col']/a[@class ='gallery__link js-lightbox-image']/img/@src")]

                yield {'title': title, 'price':price, "description": description, 'bedrooms': bedrooms, 'bathrooms': bathrooms, 'garages': garages, 'images':images}

但此代码确实检索到图像的“无”，这是有道理的，但我不知道如何去做。如果有人有建议，将不胜感激。提前谢谢！

Answer 1

你需要使用response.meta:

def parse(self, response):
    propertes = response.xpath("//div[@class='card__main']")
    for prop in propertes:
        property_url = prop.xpath(
            "./div[@class='card__body']/h3[@class='card__title']/a/@href").extract_first()
        title = prop.xpath(
            "./div[@class='card__body']/h3[@class='card__title']/a/text()").extract_first()
        price = prop.xpath(
            "./div[@class='card__body']/div[@class='card__footer card__footer--primary']/div[@class='card__price']/text()").extract_first()
        description = prop.xpath(
            "./div[@class='card__body']/div[@class='card__synopsis']/p/text()").extract_first()
        bedrooms = prop.xpath(
            "./div[@class='card__body']/div[@class='card__footer card__footer--primary']/div[@class='features features--inline']/ol[@class ='features__list']/li[@class ='features__item'][1]/div[@class='features__label']/text()").extract_first()

        yield scrapy.Request(
            url=property_url,
            callback=self.parse_property,
            meta={
                'title': title,
                'price': price,
                'description': description,
                'bedrooms': bedrooms,
            }
        )

def parse_property(self, response):

    title = response.meta["title"]
    price = response.meta["price"]
    description = response.meta["description"]
    bedrooms = response.meta["bedrooms"]

    images = response.xpath('//a[contains(@class, "gallery__link ")]/@href').getall()

    yield {'title': title, 'price':price, "description": description, 'bedrooms': bedrooms, 'bathrooms': bathrooms, 'garages': garages, 'images':images}