使用汤从烂番茄中刮取电影的所有评论答案

【问题标题】：scraping all reviews of a movie from Rotten tomato using soup使用汤从烂番茄中刮取电影的所有评论
【发布时间】：2021-11-14 13:56:34
【问题描述】：

我正在尝试从这里获取电影的所有评论：https://www.rottentomatoes.com/m/interstellar_2014/reviews。但正如您在网页上看到的那样，它们只显示大约 19 条评论。所以我无法获得所有评论我的代码下面只打印 19 条第一评论。

## First we import the module necessary to open URLs (basically websites)
from urllib.request import urlopen
from bs4 import BeautifulSoup
import pandas as pd
def scrapeUrl(URL):
  """ scrape data from url - give url as a parameter """
  page = urlopen(URL)
  html_bytes = page.read()
  html = html_bytes.decode("utf-8")
  #print(HTML)
  soup = BeautifulSoup(html, "html.parser")
  return soup   
def findReviews(soup):
  """ find reviews using class="the_review" """
  NoneType = type(None)
  reviews = []
  for element in soup.find_all("div"):
    i = element.get("class")
    if isinstance(i, NoneType) == False:
      if 'the_review' in i:
        reviews.append(element.text)
  dfrev = pd.DataFrame(reviews, columns= ['reviews'])
  return dfrev
  url = "https://www.rottentomatoes.com/m/interstellar_2014/reviews"
  sc = scrapeUrl(URL)
  t = findReviews(sc)
  print(t)

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

您可以不使用 BeautifulSoup 来执行此操作，因为 rottentomatoes 会从 api 检索评论。所以你可以先用regex从url中提取电影id，然后循环api请求直到最后一页，用pandas加载数据：

import pandas as pd
import requests
import re

headers = {
    'Referer': 'https://www.rottentomatoes.com/m/notebook/reviews?type=user',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36',
    'X-Requested-With': 'XMLHttpRequest',
}

s = requests.Session()
        
def get_reviews(url):
    r = requests.get(url)
    movie_id = re.findall(r'(?<=movieId":")(.*)(?=","type)',r.text)[0]

    api_url = f"https://www.rottentomatoes.com/napi/movie/{movie_id}/criticsReviews/all" #use reviews/userfor user reviews
    
    payload = {
        'direction': 'next',
        'endCursor': '',
        'startCursor': '',
    }
    
    review_data = []
    
    while True:
        r = s.get(api_url, headers=headers, params=payload)
        data = r.json()

        if not data['pageInfo']['hasNextPage']:
            break

        payload['endCursor'] = data['pageInfo']['endCursor']
        payload['startCursor'] = data['pageInfo']['startCursor'] if data['pageInfo'].get('startCursor') else ''

        review_data.extend(data['reviews'])
        time.sleep(1)
    
    return review_data

data = get_reviews('https://www.rottentomatoes.com/m/interstellar_2014/reviews')
df = pd.json_normalize(data)

	creationDate	isFresh	isRotten	isRtUrl	isTop	reviewUrl	quote	reviewId	scoreOri	scoreSentiment	critic.name	critic.criticPictureUrl	critic.vanity	publication.id	publication.name
0	Oct 9, 2021	True	False	False	False	https://www.nerdophiles.com/2014/11/05/interstellar-delivers-beauty-and-complexity-in-typical-nolan-fashion/	The inherent message of the film brings hope, but it can definitely get waterlogged by intellectual speak and long-winded scenes.	2830324	3/5	POSITIVE	Therese Lacson	http://resizing.flixster.com/gGcp41zlZQ3sYdSbQoS8AATHp8Y=/128x128/v1.YzszODg1O2o7MTg5OTA7MjA0ODszMDA7MzAw	therese-lacson	3888	Nerdophiles
1	Aug 10, 2021	True	False	False	False	https://www.centraltrack.com/space-oddity/	The film is indeed a sight to behold -- and one that demands to be seen on the biggest possible screen.	2812665	B	POSITIVE	Kip Mooney	http://resizing.flixster.com/hoYjdO_o-Ip21XnJaWr0C27-nbc=/128x128/v1.YzszOTk2O2o7MTg5OTA7MjA0ODs0MDA7NDAw	kip-mooney	2577	Central Track
2	Feb 2, 2021	True	False	False	False	http://www.richardcrouse.ca/interstellar-3-stars-one-for-each-hour-of-the-movie-sentimental-sic-fi/	Nolan reaches for the stars with beautifully composed shots and some mind-bending special effects, but the dime store philosophy of the story never achieves lift off.	2763105	3/5	POSITIVE	Richard Crouse	http://resizing.flixster.com/Ep5q7RwWq9Ud5KBhnha2sPnsRD0=/128x128/v1.YzszODgxO2o7MTg5OTA7MjA0ODszMDA7MzAw	richard-crouse	3900	Richard Crouse

【讨论】：