BeautifulSoup：HTML 提取项目符号而不是导航栏

Question

我正在使用 BeautifulSoup4 进行一些 HTML 抓取。 我正在尝试提取重要信息，例如标题、元数据、段落和列出的信息。

我的问题是我可以使用这样的段落：

def main():
    response = urllib.request.urlopen('https://ecir2019.org/industry-day/')
    html = response.read()
    soup = BeautifulSoup(html,features="html.parser")
    text = [e.get_text() for e in soup.find_all('p')]
    article = '\n'.join(text)


    print(article)

main()

但如果我的网站链接在正文中有项目符号点，它将包括导航栏。即如果我将p 更改为li 或ul

例如我想要得到的输出是：

The Industry Day's objectives are three-fold:

The first objective is to present the state of the art in search and search-related areas, delivered as keynote talks by influential technical leaders from the search industry.
The second objective of the Industry Day is the presentation of interesting, novel and innovative ideas related to information retrieval.
Finally, we are looking forward to a highly-interactive discussion involving both industry and academia.

我实际得到的： The Industry Day's objectives are three-fold:

HTML 源代码中的标签：

<p>The Industry Day's objectives are three-fold:</p>
<ol>
<li>The first objective is to present the state of the art in search and search-related areas, delivered as keynote talks by influential technical leaders from the search industry.</li>
<li>The second objective of the Industry Day is the presentation of interesting, novel and innovative ideas related to information retrieval.</li>
<li>Finally, we are looking forward to a highly-interactive discussion involving both industry and academia.</li>
</ol>

Answer 1

您可以使用 Or css 选择器语法，这样您也可以选择 li 元素。

import requests
from bs4 import BeautifulSoup

url = 'https://ecir2019.org/industry-day/'

response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')
items = [item.text for item in soup.select('p, ol li')]

print(items)

---

只是那个部分：

import requests
from bs4 import BeautifulSoup

url = 'https://ecir2019.org/industry-day/'

response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')
items = [item.text for item in soup.select('.kg-card-markdown p:nth-of-type(2), .kg-card-markdown p:nth-of-type(2) + ol li')]

print(items)

---

页面似乎已更改，因此我使用的是缓存版本（这仅在缓存更新之前有效）。您可以使用附加的类选择器限制帖子正文：

import requests
from bs4 import BeautifulSoup

url = 'http://webcache.googleusercontent.com/search?q=cache:https://ecir2019.org/industry-day'

response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')
items = [item.text for item in soup.select('.post-body p, .post-body ol li, .post-body ul li')]

print(items)