rottentomatoes 的网页抓取出错了

Question

我想从rottentomatoes抓取一个页面。

页面截图是

根据图片span class= descriptor是a class的父类，div class = info director是Directed By的gradparent。

我想刮掉导演的名字

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
url= 'https://editorial.rottentomatoes.com/guide/best-sci-fi-movies-of-all-time/'
r = requests.get(url, headers=headers)#, proxies=proxies)
content = r.content
soup = BeautifulSoup(content)
director = []
people1 = soup.find_all('div',{'class':'info director'})
for d in people1:
    Dir = d.find('a').text
    director.append(Dir)

我收到了这个错误

AttributeError: 'NoneType' object has no attribute 'text'

Answer 1

使用“info director”类定位 div，并使用单行将所有 href 文本转储到列表中

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
url = 'https://editorial.rottentomatoes.com/guide/best-sci-fi-movies-of-all-time/'
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html5lib')
directors = [a.text for a in (d.find('a') for d in soup.find_all('div', attrs={'class': 'info director'})) if a]
for x in range(len(directors)):
    print(directors[x])  # output directors

# alternative loop
directors = []
for d in soup.find_all('div', attrs={'class': 'info director'}):
    for a in d.find('a'):
        directors.append(a)
        print(a)