使用 BS4 进行网页抓取，如何设置查看范围

Question

我正在尝试抓取此维基百科页面的“事件”部分：https://en.wikipedia.org/wiki/2020。该页面没有最容易导航的 HTML，因为大多数标签不是嵌套的，而是同级的。

我想确保我抓取的唯一数据位于下面显示的两个 h2 标记之间。
这是精简的 HTML：

<h2>                  #I ONLY WANT TO SEARCH BETWEEN HERE
    <span id="Events">Events</span>
</h2>
<h3>...</h3>
<ul>...</ul>
<h3>...</h3>
<ul>
    <li>
        <a title="June 17"</a>   #My code below is looking for this, if not found it jumps to another section
    </li>
</ul>
<h3>...</h3>
<ul>...</ul>
<h2>                 #AND HERE. DON"T WANT TO GO PAST HERE
    <span id="Predicted_and_scheduled_events">Predicted_and_scheduled_events</span>
</h2>

如果不清楚，每个标签（span 除外）都是兄弟。如果两个 h2 标记之间存在日期，我的代码当前可以工作，但是如果日期不存在，它将转到页面的另一个部分以提取数据，这是我不想要的。

这是我的代码：

import sys
import requests
import bs4
res = requests.get('https://en.wikipedia.org/wiki/2020')
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text,"lxml")
todaysNews = soup.find('a', {"title": "June 17"}) #goes to date's stories

Answer 1

BS 有很多有用的函数和参数。值得阅读整个文档。

它具有获取父元素、下一个兄弟元素、具有任何标题的元素等的功能。

---

首先我搜索<span id="Events">Events</span>，然后我得到它的parent 元素<h2>，我有数据的开始。

接下来我可以得到next_siblings 并在for-loop 中运行，直到我得到名称为h2 的项目并得到数据结束。

在for-loop 中，我可以检查所有ul 元素并直接搜索没有嵌套li 元素的li 元素（recursive=False），在li 中，我可以先得到a，它有title 任何文字 ({"title": True})

import requests
import bs4

res = requests.get('https://en.wikipedia.org/wiki/2020')
res.raise_for_status()

soup = bs4.BeautifulSoup(res.text, 'lxml')

# found start of data `h2`
start = soup.find('span', {'id': 'Events'}).parent

# check sibling items
for item in start.next_siblings:

    # found end of data `h2`
    if item.name == 'h2': 
        break

    if item.name == 'ul':

        # only direct `li` without nested `li`
        for li in item.find_all('li', recursive=False): 

            # `a` which have `title`
            a = li.find('a', {'title': True}) 

            if a:
                print(a['title'])

结果：

January 1
January 2
January 3
January 5
January 7
January 8
January 9
January 10
January 12
January 16
January 18
January 28
January 29
January 30
January 31
February 5
February 11
February 13
February 27
February 28
February 29
March 5
March 8
March 9
March 11
March 12
March 13
March 14
March 16
March 17
March 18
March 20
March 23
March 24
March 26
March 27
March 30
April 1
April 2
April 4
April 5
April 6
April 7
April 8
April 9
April 10
April 12
April 14
April 15
April 17
April 18
April 19
April 20
April 21
April 22
April 23
April 25
April 26
April 27
April 28
April 29
April 30
May 1
May 3
May 4
May 5
May 6
May 7
May 9
May 10
May 11
May 12
May 14
May 15
May 16
May 18
May 19
May 21
May 22
May 23
May 24
May 26
May 27
May 28
May 30
May 31
June 1
June 2
June 3
June 4
June 6
June 7
June 8
June 9
June 16

Answer 2

您可以将 CSS 选择器与 "," 一起使用，然后检查标签名称。

CSS 选择器h2:contains("Events") ~ ul > li 将选择所有ul > li 兄弟姐妹到<h2>，其中包含字符串"Events"。

import requests
from bs4 import BeautifulSoup

url = 'https://en.wikipedia.org/wiki/2020'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for tag in soup.select('h2:contains("Events") ~ ul > li, h2:contains("Predicted and scheduled events")'):
    if tag.name == 'li':
        print(tag.a.text)
    else:
        break

打印：

January 1
January 2
January 3
January 5
January 7
January 8
January 9
January 10
January 12
January 16
January 18
January 28
January 29
January 30
January 31
February 5
February 11
February 13
February 27
February 28
February 29
March 5
March 8
March 9
March 11
March 12
March 13
March 14
March 16
March 17
March 18
March 20
March 23
March 24
March 26
March 27
March 30
April 1
April 2
April 4
April 5
April 6
April 7
April 8
April 9
April 10
April 12
April 14
April 15
April 17
April 18
April 19
April 20
April 21
April 22
April 23
April 25
April 26
April 27
April 28
April 29
April 30
May 1
May 3
May 4
May 5
May 6
May 7
May 9
May 10
May 11
May 12
May 14
May 15
May 16
May 18
May 19
May 21
May 22
May 23
May 24
May 26
May 27
May 28
May 30
May 31
June 1
June 2
June 3
June 4
June 6
June 7
June 8
June 9
June 16