如何使用 bs4 从外部链接中抓取评论？

Question

我想为每部电影提取至少 20 条用户评论，但我不知道如何循环进入 IMDb 标题电影，然后使用 beautifulsoup 获取用户评论。

开始链接 = "https://www.imdb.com/search/title/?title_type=feature,tv_movie&release_date=2018-01-01,2019-12-31&count=250";

title_link(1) = "https://www.imdb.com/title/tt7131622/?ref_=adv_li_tt";

user_reviews_link_movie1 = "https://www.imdb.com/title/tt7131622/reviews?ref_=tt_ov_rt" ;

我能够从静态页面中提取列表中每部电影的标题、年份、评分和元分数。

# Import packages and set urls

from requests import get
url = 'https://www.imdb.com/search/title/?title_type=feature,tv_movie&release_date=2018-01-01,2019-12-31&count=250'
response = get(url)
print(response.text[:500])

from bs4 import BeautifulSoup
html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)


movie_containers = html_soup.find_all('div', class_ = 'lister-item mode-advanced')
print(type(movie_containers))
print(len(movie_containers))

# Lists to store the scraped data in

names = []
years = []
imdb_ratings = []
metascores = []
votes = []

# Extract data from individual movie container
for container in movie_containers:
# If the movie has Metascore, then extract:
    if container.find('div', class_ = 'ratings-metascore') is not None:

# The name
        name = container.h3.a.text
        names.append(name)
# The year
        year = container.h3.find('span', class_ = 'lister-item-year').text
        years.append(year)
# The IMDB rating
        imdb = float(container.strong.text)
        imdb_ratings.append(imdb)
# The Metascore
        m_score = container.find('span', class_ = 'metascore').text
        metascores.append(int(m_score))

import pandas as pd
test_df = pd.DataFrame({'movie': names,'year': years,'imdb': imdb_ratings,'metascore': metascores})
test_df

1. 实际结果：

   电影年 imdb 元评分

   从前...在好莱坞 (2019) (8.1) (83)

   恐怖故事 (2019) (6.5) (61)

   速度与激情：霍布斯与肖 (2019) (6.8) (60)

   复仇者联盟：终局之战 (2019) (8.6) (78)

2. 预期：

   movie1 year1 imbd1 metascore1 review1

   movie1 year1 imbd1 metascore1 review2

   ...

   movie1 year1 imbd1 metascore1 review20

   movie2 year2 imbd2 metascore2 review1

   ...

   movie2 year2 imbd2 metascore2 review20

   ...

   movie250 year250 imbd250 metascore250 review20

Answer 1

假设我在 cmets 中的问题的答案是“是”。

以下是您最初请求的解决方案。 检查某部电影是否真的有 20 条评论。如果更少，则收集所有可用的。

技术上解析过程是正确的，我在分配movie_containers = movie_containers[:3]时检查了它。收集所有数据需要一些时间。

更新：刚刚完成了所有 250 部电影的信息收集 - 一切都被无误地抓取，所以解决方案本身只是仅供参考。

此外，如果您想进一步解析，我的意思是为接下来的 250 部电影等收集数据，您可以在此解析器中再添加一个循环级别。该过程类似于“评论提取”部分中的过程。

# Import packages and set urls

from requests import get
from bs4 import BeautifulSoup
import pandas as pd

base_url = 'https://www.imdb.com/search/title/?title_type=feature,tv_movie&release_date=2018-01-01,2019-12-31&count=250'
url_header_for_reviews = 'https://www.imdb.com'
url_tail_for_reviews = 'reviews?ref_=tt_urv'
base_response = get(base_url)
html_soup = BeautifulSoup(base_response.text, 'html.parser')

movie_containers = html_soup.find_all('div', class_ = 'lister-item mode-advanced')

result_df = pd.DataFrame()

# Extract data from individual movie container
for container in movie_containers:
# If the movie has Metascore, then extract:
    if container.find('div', class_ = 'ratings-metascore') is not None:

# Reviews extracting
        num_reviews = 20
        # Getting last piece of link puzzle for a movie reviews` link
        url_middle_for_reviews = container.find('a')['href']
        # Opening reviews page of a concrete movie
        response_reviews = get(url_header_for_reviews + url_middle_for_reviews + url_tail_for_reviews)
        reviews_soup = BeautifulSoup(response_reviews.text, 'html.parser')
        # Searching all reviews
        reviews_containers = reviews_soup.find_all('div', class_ = 'imdb-user-review')
        # Check if actual number of reviews is less than target one
        if len(reviews_containers) < num_reviews:
            num_reviews = len(reviews_containers)
        # Looping through each review and extracting title and body
        reviews_titles = []
        reviews_bodies = []
        for review_index in range(num_reviews):
            review_container = reviews_containers[review_index]
            review_title = review_container.find('a', class_ = 'title').text.strip()
            review_body = review_container.find('div', class_ = 'text').text.strip()
            reviews_titles.append(review_title)
            reviews_bodies.append(review_body)
# The name
        name = container.h3.a.text
        names = [name for i in range(num_reviews)]
# The year
        year = container.h3.find('span', class_ = 'lister-item-year').text
        years = [year for i in range(num_reviews)]
# The IMDB rating
        imdb_rating = float(container.strong.text)
        imdb_ratings = [imdb_rating for i in range(num_reviews)]
# The Metascore
        metascore = container.find('span', class_ = 'metascore').text
        metascores = [metascore for i in range(num_reviews)]

# Gathering up scraped data into result_df
        if result_df.empty:
            result_df = pd.DataFrame({'movie': names,'year': years,'imdb': imdb_ratings,'metascore': metascores,'review_title': reviews_titles,'review_body': reviews_bodies})
        elif num_reviews > 0:
            result_df = result_df.append(pd.DataFrame({'movie': names,'year': years,'imdb': imdb_ratings,'metascore': metascores,'review_title': reviews_titles,'review_body': reviews_bodies}))

顺便说一句，我不确定 IMDB 是否会让您按原样循环收集所有电影的数据。您有可能获得验证码或重定向到其他页面。如果出现这些问题，我会采用一个简单的解决方案 - 暂停抓取和/或更改 user-agents。

暂停（睡眠）可以如下实现：

import time
import numpy as np

time.sleep((30-5)*np.random.random()+5) #from 5 to 30 seconds

在请求中插入user-agent 可以如下完成：

import requests 
from bs4 import BeautifulSoup

url = ('http://www.link_you_want_to_make_request_on.com/bla_bla')
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

谷歌user-agents 的一些其他变体，从中列出并在下次请求中不时更改它们。请注意您使用的是哪个user-agents - 其中一些表示移动或平板设备，对于他们来说，一个网站（不仅是 IMDB）可以提供不同于 PC 格式的响应页面 - 其他标记、其他设计等。所以一般来说，上述算法仅适用于 PC 版页面。