BeautifulSoup 查找多个类别

Question

我正在尝试 scrape 一些 Wiki 页面，仅用于培训 我被困住了，

我想打印页面的标题、最后修改日期和类别 这是我的代码：

from bs4 import BeautifulSoup
import requests
import pandas as pd


response = requests.get('https://en.wikipedia.org/wiki/Eurovision_Song_Contest') 
soup = BeautifulSoup(response.content, "html.parser") 


head=soup.find(class_='firstHeading').get_text()
print('wikipedia entry: '+head)

foot=soup.find(id='footer-info-lastmod').get_text()
print(foot)

cate=soup.find_all(class_='mw-normal-catlinks')
x=soup.findAll("li",attrs={"title"})
print(x)

但它说： ResultSet 对象没有属性“get_text”。您可能将项目列表视为单个项目。当你打算调用 find() 时，你调用了 find_all() 吗？

我需要打印：类别列表 例如在这个页面上：

Answer 1

您可以通过查找父 div 来解决您的问题：

代码：

from bs4 import BeautifulSoup
 import requests
 import pandas as pd


 response = requests.get('https://en.wikipedia.org/wiki/Eurovision_Song_Contest') 
 soup = BeautifulSoup(response.content, "html.parser") 


 head=soup.find(class_='firstHeading').get_text()
 print('wikipedia entry: '+head)

 foot=soup.find(id='footer-info-lastmod').get_text()
 print(foot)

 cate=soup.find_all(class_='mw-normal-catlinks')
 catdiv = soup.find("div",{"id":"mw-normal-catlinks"})
 categories = catdiv.find("ul").find_all("li")
 for cat in categories:
     print(cat.text)

结果：

wikipedia entry: Eurovision Song Contest
 This page was last edited on 6 December 2019, at 10:20 (UTC).
Eurovision Song Contest
1956 establishments in Europe
Eurovision events
Music television
Pop music festivals
Recurring events established in 1956
Song contests

Answer 2

此脚本打印页眉、页脚和类别列表：

from bs4 import BeautifulSoup
import requests

response = requests.get('https://en.wikipedia.org/wiki/Eurovision_Song_Contest')
soup = BeautifulSoup(response.content, "html.parser")

head=soup.find(class_='firstHeading').get_text()
print('wikipedia entry: {}'.format(head))      # better use str.format()

foot=soup.find(id='footer-info-lastmod').get_text(strip=True)   # use strip=True to strip the text of whitespace characters
print(foot)

categories = [li.get_text() for li in soup.select('#mw-normal-catlinks li')]
print(categories)

打印：

wikipedia entry: Eurovision Song Contest
This page was last edited on 6 December 2019, at 10:20(UTC).
['Eurovision Song Contest', '1956 establishments in Europe', 'Eurovision events', 'Music television', 'Pop music festivals', 'Recurring events established in 1956', 'Song contests']

Answer 3

更简单：

normal=soup.find(class_="mw-normal-catlinks")
categories=normal.find_all("a", )
for category in categories:    
        print(category.text)

Answer 4

您的脚本完美地打印了“头”和“脚”，所以我将专注于打印类别列表。

首先，find_all() 返回标签列表而不是单个标签，因此在标签列表上尝试“get_text()”会出错。

cate=soup.find_all(class_='mw-normal-catlinks')
print(cate.get_text())

AttributeError: ResultSet object has no attribute 'get_text'. You're probably treating a list of items like a single item. Did you call find_all() when you meant to call find()?

在您的情况下，由于 find_all() 仅返回一个标签，因此您可以使用 'find()' 或从返回的列表中提取标签（div）。

cate=soup.find_all(class_='mw-normal-catlinks')[0]

您的类别位于“ul”标签下，它是此处“div”标签的子标签（您使用 find_all() 提取），因此您可以直接访问它们并将它们存储到这样的列表中-

cate=soup.find_all(class_='mw-normal-catlinks')[0]

x=cate.ul.get_text("|")

categoryList = x.split("|")

print(categoryList)

输出： [“欧洲歌唱大赛”、“1956 年欧洲场所”、“欧洲歌唱大赛”、“音乐电视”、“流行音乐节”、“1956 年建立的定期活动”、“歌唱比赛”]