（美丽的soap4，）AttributeError：'NoneType'对象没有属性'get_text'答案

【问题标题】：(Beautiful soap4, )AttributeError: 'NoneType' object has no attribute 'get_text'（美丽的soap4，）AttributeError：'NoneType'对象没有属性'get_text'
【发布时间】：2021-06-16 15:08:43
【问题描述】：

博客：分析 IMDb 的前 250 部电影：第 1 部分；让我们抓取一些数据

详细信息：https://medium.com/analytics-vidhya/analyzing-imdbs-top-250-movies-part-1-let-scrape-some-data-a422adc3eb8d

问题是，每当我想检索 IMDb 的前 250 部电影中的单个页面链接时，都会出现错误：AttributeError: 'NoneType' object has no attribute 'get_text' 这意味着我理解它没有所需的类名或我们的元素在 HTML 中寻找。但是 HTML 由我传递的类名组成。我正在实现与博客相同的内容，但我无法检索单个电影并抓取数据。以下是与博客中相同的代码：

import requests                 # Simpler HTTP requests 
from bs4 import BeautifulSoup   # Python package for pulling data out of HTML and XML files
import pandas as pd             # Python package for data manipulation and analysis
import re                       # regular expressions
import json                     # Python package used to work with JSON data
from tqdm import tqdm           # python for displaying progressbar 
from datetime import datetime

url = 'https://www.imdb.com/chart/top'
url_text = requests.get(url).text
soup = BeautifulSoup(url_text, 'html.parser'
template = 'https://www.imdb.com%s'
title_links = [template % a.attrs.get('href') for a in url_soup.select( 'td.titleColumn a' )]

movie_name = (page_soup.find("div",{ "class":"title_wrapper" }).get_text( strip=True ).split('|')[0]).split('(')[0]

【问题讨论】：

您的代码没有运行，似乎缺少一些变量声明，例如url_soup 和page_soup

标签： python web web-scraping python-requests

【解决方案1】：

你需要的数据可以在td标签中轻松找到，类名titleColumn。您可以从那里提取电影名称和链接。

在这里，我展示了前 10 部电影。您可以修改此代码以满足您的要求。

import requests
import bs4 as bs

url = "https://www.imdb.com/chart/top"
response = requests.get(url)
html = response.text
soup = bs.BeautifulSoup(html, 'lxml')

t = soup.findAll('td', class_='titleColumn')
for i in range(10):
    a_tag = t[i].find('a')
    link = 'https://www.imdb.com/' + a_tag['href']
    title = a_tag.text

    print(f'Link: {link}\nMovie: {title}\n')

Sample Output:

Link: https://www.imdb.com//title/tt0111161/
Movie: The Shawshank Redemption

Link: https://www.imdb.com//title/tt0068646/
Movie: The Godfather

Link: https://www.imdb.com//title/tt0071562/
Movie: The Godfather: Part II

Link: https://www.imdb.com//title/tt0468569/
Movie: The Dark Knight

【讨论】：

【解决方案2】：

有一个更简单的方法来获得标题。每个页面都有一个 <title> 元素，其中包含您需要的信息：

import requests                 # Simpler HTTP requests
from bs4 import BeautifulSoup   # Python package for pulling data out of HTML and XML files
#import pandas as pd             # Python package for data manipulation and analysis
import re                       # regular expressions
import json                     # Python package used to work with JSON data
#from tqdm import tqdm           # python for displaying progressbar
from datetime import datetime

url = 'https://www.imdb.com/chart/top'
url_text = requests.get(url).text
url_soup = BeautifulSoup(url_text, 'html.parser')
template = 'https://www.imdb.com%s'
title_links = [template % a.attrs.get('href') for a in url_soup.select( 'td.titleColumn a' )]

movie_names = []

for title_link in title_links:
    page_soup = BeautifulSoup(requests.get(title_link).text, 'html.parser')

    movie_names.append(page_soup.title.get_text(strip=True).split(' (')[0])


print(movie_names)

【讨论】：