抓取vivino.com评论评论返回空数据

Question

使用 Python3，我想从 vivino.com 的葡萄酒页面中抓取评论 cmets。例如，向下滚动this page 会将您带到名为“社区评论”的部分，这就是我想要抓取的内容。但是我的代码返回一个空数据。事实上，看requests.get(url) 的内容，我看不到评论部分。谁能帮我找出原因并建议我一个解决方案？

这是我的尝试：

from bs4 import BeautifulSoup
import requests
import time

url = "https://www.vivino.com/FR/en/dauprat-pauillac/w/3823873?year=2017&price_id=24797287"

headers = headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'}
data = requests.get(url, headers = headers) # this returns data but isn't capturing the section with review comments.

soup = BeautifulSoup(data.text, 'html.parser')
soup.find_all('div', id='all_reviews')

>>>
[]

Answer 1

评论使用 API 动态加载。

response = requests.get('https://www.vivino.com/api/wines/3823873/reviews?per_page=4&year=2017'
reviews = response.json()

Answer 2

评论是从另一个 URL 动态加载的。你可以使用这个例子来引导他们：

import re
import json
import requests

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0",
}


url = "https://www.vivino.com/FR/en/dauprat-pauillac/w/3823873?year=2017&price_id=24797287"
api_url = (
    "https://www.vivino.com/api/wines/{id}/reviews?per_page=9999&year={year}"
) # <-- increased the number of reviews to 9999

id_ = re.search(r"/(\d{5,})", url).group(1)
year = re.search(r"year=(\d+)", url).group(1)

data = requests.get(api_url.format(id=id_, year=year), headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for r in data["reviews"]:
    print(r["note"])
    print("-" * 80)

打印：

Out the bottle - oak barrel bruja and cherry acidity, in decant - smooth toffee, vanilla, dark fruit and church air tannins and that cool mushroom midsection into a wonderful taste bud saturating peppered opacity of big barrelled Bordeaux. Never thought I would “leave” a wine to age,  now want to get a boatload of these to bide a while, that nauseating term of “ageing potential” ?. 
Sits deep and brooding in the decanter, could swear a gator breached briefly in slow lid blink. 
Magical. 
--------------------------------------------------------------------------------
Nice colour, mid depth, purple red at edge, blackcurrant fruit, some spice, little cherry. 
Nice fresh entry; clean and pure fruit characters; bit of graphite and blackcurrant on the palate; nice texture.
--------------------------------------------------------------------------------
Dark ruby color. Blackberry, blueberry and tobacco. Medium body. Subtle and elegant.
--------------------------------------------------------------------------------
huge wine from a great region, palpable potential and structure. elegant tannins 
--------------------------------------------------------------------------------
Lovly Paulliac , great price beautiful Bouquet
--------------------------------------------------------------------------------
A little young but if you decant for 2 hours it shines
--------------------------------------------------------------------------------
Great powerful Pauillac
--------------------------------------------------------------------------------
When I saw the low price, I had low expectations but this was a really powerful wine with a nice Cabernet backbone.
--------------------------------------------------------------------------------
Wonderful nose and still young... with loads of acid & tannins, this should age very well.  Recommend waiting a few years to enjoy more fully.
--------------------------------------------------------------------------------
Volumes of ripe strawberry, tobacco, vanilla, pencil shavings, balsamic
Medium+ body
Intense, dry fruitiness, punchy acidity, powerful tannins
 
--------------------------------------------------------------------------------
 earthy and dark fruits on the nose, purple hue, with dark fruits and oaky flavours. Lunch outside with Mum and Dad to celebrate Suzie and Jeremy’s wedding.
--------------------------------------------------------------------------------
Deep purple. Mint and vanilla nose with touch of mocha. Very attractive. Very good depth on palate with wonderful ripeness and extract. Really nice. Well done.
--------------------------------------------------------------------------------
Such an interesting wine.next door to paulliacs premier grand cru latour and Lynch bages.made by the owners of du glana..50 of a ha in the midst of excellence old vines and excellent itself.but just a paulliac!
--------------------------------------------------------------------------------
Earthy and Cohiba notes 
--------------------------------------------------------------------------------
Fruit could be bolder, but well structured with ample tannins and graphite, mushroom and dark fruit notes. Elegant expression of the terroir.  
--------------------------------------------------------------------------------

...and so on.