我的代码返回一个正确的评论和 NULL 而不是其他评论

Question

这是代码

import scrapy
   

class YelpscrapeSpider(scrapy.Spider):
    name = 'yelpscrape'
    start_urls = ['https://www.yelp.com/biz/beretta-san-francisco?osq=Restaurants']

    def parse(self, response):

        for review in response.css('ul.undefined.list__373c0__3GI_T'):
            item = {
                
                'review': review.css('p.comment__373c0__1M-px.css-n6i4z7 span').get()
            }
            
            yield item

这是它返回的内容

[
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": "Correct review"},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null},
{"review": null}
]

我只是不太明白要更改什么，所以它会正确返回。我到处玩，得到了 18 份相同的评论，这也不是我想要的。你们能帮我解决这个问题吗？

Answer 1

如果您在浏览器中查看该页面，并在记录浏览器的网络流量时点击评论页面，您将看到每次点击查看其他页面时都会向 REST API 发出 XHR HTTP GET 请求评论。 API 的响应是 JSON，它一次包含十条评论的信息。您所要做的就是模仿该请求 - 要获得更多评论，您必须向同一个 API 端点发出多个请求，但使用不同的查询字符串参数（"start" 键值对在params字典从"0"开始，然后上升到"10"、"20"等）：

def get_reviews(restaurant_id):
    import requests

    url = "https://www.yelp.com/biz/{}/review_feed".format(restaurant_id)

    params = {
        "rl": "en",
        "q": "",
        "sort_by": "relevance_desc",
        "start": "0"
    }

    headers = {
        "accept": "application/json",
        "accept-encoding": "gzip, deflate",
        "user-agent": "Mozilla/5.0",
        "x-requested-by-react": "true",
        "x-requested-with": "XMLHttpRequest"
    }

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()

    yield from (review["comment"]["text"] for review in response.json()["reviews"])
    

def main():

    for review in get_reviews("4KfQnlcSu4bbTqnvGdGptw"):
        print(review, "\n")
    
    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())