使用 python scrapy 抓取动态网站（使用反应构建）的最佳方法

Question

我一直在尝试使用 scrapy 和 scrapy-splash 抓取这个网站 Link。据我所知，这个网站是在反应中开发的。 response.xpath 总是返回带有任何类名的空列表。请建议我一种方法来抓取这个反应网站。我已经使用这个link 设置了splash，并且能够在同一个项目中抓取一些其他网站，但无法抓取这个反应制作的网站。蜘蛛的代码如下：

import scrapy
from scrapy_splash import SplashRequest

class NykaaFashionbrandsSpider(scrapy.Spider):
    name = 'nykaa_fashionbrands'

    start_urls = ["https://www.nykaafashion.com/"]
    custom_settings = {
        'FEED_FORMAT': 'csv',
        'FEED_URI': 'fashion_brands.csv'
    }

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse,
                                endpoint='render.html',
                                args={'wait': 3},
                                )

    def parse(self, response):
        print(response.xpath('//*[@class="br-inner"]/ul/li/text()').extract())
        # I am trying to get the list items

Answer 1

如果您需要抓取特定类别中的所有产品或产品，您可以使用如下 API url：

https://www.nykaafashion.com/rest/appapi/V2/categories/products?categoryId=6151&PageSize=12&sort=popularity&currentPage=2&filter_format=v2

这条回复：

"products": [
            {
                "sku": "CTWK0648",
                "imageUrl": "https://adn-static1.nykaa.com/nykdesignstudio-images/pub/media/catalog/product/3/8/3884c_1.jpg?rnd=20200526195200",
                "isOutOfStock": 0,
                "subTitle": "Black Embellished Sandals",
                "title": "Catwalk",
                "price": 1995,
                "tag": {},
                "offerCount": 0,
                "categoryId": [
                    "102",
                    "3528",
                    "3522",
                    "2",
                    "6151",
                    "6557"
                ],
                "discount": 55,
                "offers": null,
                "discountedPrice": 899,
                "actionUrl": "/catwalk-black-embellished-sandals-3/p/537684",
                "aspectRatio": 0.75,
                "sizeVariation": [
                    {
                        "id": "537678",
                        "title": "4"
                    },
                    {
                        "id": "537679",
                        "title": "5"
                    },
                    {
                        "id": "537680",
                        "title": "6"
                    },
                    {
                        "id": "537681",
                        "title": "7"
                    }
                ],
                "type": "configurable",
                "id": "537684"
            },

此网站不需要Splash