如何使用 python 和 Beautifulsoup 抓取网页的第二个 <p>

Question

我一直在尝试使用 BeautifulSoup，因为我想尝试抓取网页 (https://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1)。到目前为止，我已经成功地抓取了一些元素，但现在我想抓取电影说明，但我一直在苦苦挣扎。描述在 html 中的位置是这样的：

<div class="lister-item mode-advanced"> 
    <div class="lister-item-content> 
       <p class="muted-text"> paragraph I don't need</p>
       <p class="muted-text"> paragraph I need</p>
    </div>
</div>

我想抓取似乎很容易做到的第二段，但我尝试的所有内容都给了我“无”作为输出。我一直在四处寻找答案。在另一个 stackoverflow 帖子中，我发现

find('p:nth-of-type(1)')  

或

find_elements_by_css_selector('.lister-item-mode >p:nth-child(1)')

可以做到这一点，但它仍然给了我

none #as output

你可以在下面找到我的一段代码，它的等级有点低，因为我只是在尝试学习的东西

 import urllib2
from bs4 import BeautifulSoup
from requests import get

url = 'http://www.imdb.com/search/title? 
release_date=2017&sort=num_votes,desc&page=1'
response = get(url)

html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)
movie_containers = html_soup.find_all('div', class_='lister-item mode- 
advanced')

first_movie = movie_containers[0]

first_title = first_movie.h3.a.text
print first_title

first_year = first_movie.h3.find('span', class_='lister-item-year text-muted unbold')
first_year = first_year.text
print first_year

first_imdb = float(first_movie.strong.text)
print first_imdb

# !!!! problem zone ---------------------------------------------
first_description = first_movie.find('p', class_='muted-text')
#first_description = first_description.text
print first_description

上面的代码给了我这个输出：

$ python scrape.py
Logan
(2017)
8.1
None

我想学习选择html标签的正确方法，因为这对以后的项目很有用。

Answer 1

find_all() 方法查看标签的后代并检索 与您的过滤器匹配的所有后代。

然后您可以使用列表的索引来获取您需要的元素。索引从 0 开始，因此 1 将给出第二项。

将 first_description 更改为此。

first_description = first_movie.find_all('p', {"class":"text-muted"})[1].text.strip()

完整代码

import urllib2
from bs4 import BeautifulSoup
from requests import get

url = 'http://www.imdb.com/search/title?release_date=2017&sort=num_votes,desc&page=1'
response = get(url)

html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)
movie_containers = html_soup.find_all('div', class_='lister-item mode-advanced')

first_movie = movie_containers[0]

first_title = first_movie.h3.a.text
print first_title

first_year = first_movie.h3.find('span', class_='lister-item-year text-muted unbold')
first_year = first_year.text
print first_year

first_imdb = float(first_movie.strong.text)
print first_imdb

# !!!! problem zone ---------------------------------------------
first_description = first_movie.find_all('p', {"class":"text-muted"})[1].text.strip()
#first_description = first_description.text
print first_description

输出

Logan
(2017)
8.1
In the near future, a weary Logan cares for an ailing Professor X. However, Logan's attempts to hide from the world and his legacy are upended when a young mutant arrives, pursued by dark forces.

阅读Documentation，了解选择html标签的正确方法。

还可以考虑迁移到 python 3。

Answer 2

随便玩.next_sibling 就能搞定。不过，可能还有一种更优雅的方式。至少可以给你一个开始/一些方向

from bs4 import BeautifulSoup


html = '''<div class="lister-item mode-advanced"> 
    <div class="lister-item-content> 
       <p class="muted-text"> paragraph I don't need</p>
       <p class="muted-text"> paragraph I need</p>
    </div>
</div>'''


soup = BeautifulSoup(html, 'html.parser')


first_p = soup.find('div',{'class':'lister-item mode-advanced'}).text.strip()
second_p = soup.find('div',{'class':'lister-item mode-advanced'}).next_sibling.next_sibling.text.strip()



print (second_p)

输出：

print (second_p)
paragraph I need

Answer 3

BeautifulSoup 4.71 支持 :nth-child() 或任何 CSS4 选择器

first_description = soup.select_one('.lister-item-content p:nth-child(4)')
# or 
#first_description = soup.select_one('.lister-item-content p:nth-of-type(2)')

print(desc)