使用 BS4 抓取数据给了我意想不到的结果

Question

我不想用我的基本问题惹恼你，但我被困住了，希望你能帮助我。 我已经完成了教程并观看了许多视频，但我无法弄清楚我做错了什么。 我想从这个表中抓取数据：https://www.youpriboo.com/vorher_102_main_nat.php?action=show&liga=2.BL

这是我的代码：

import requests
from bs4 import BeautifulSoup

base_URL = 'https://www.youpriboo.com/vorher_102_main_nat.php?action=show&liga='
liga = '2.BL'
URL = base_URL + liga

headers = {
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36:'}

page = requests.get(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')

for name in soup.find_all("td", class_="hac"):
    name1 = name.parent.find_all('td')[1] # team1 
    name2 = name.parent.find_all('td')[2] # team2
    wahr1 = name.parent.find_all('td')[6] # wahr1
    print(name1.get_text() +' '+ name2.get_text()+' '+ wahr1.get_text())

问题是它给了我 3 次数据，并且在游戏之间列出了 3 个数字。

预期结果如下所示：

Armina Bielefeld VfB Stuttgart 34,43
SV Wehen Wiesbaden VfL Osnabrück 34,51
(and so on)

感谢您的时间和工作！

我也在这里发布了这个：https://www.reddit.com/r/Python/comments/d9km7y/scraping_data_using_bs4_gives_me_unexpected/

Answer 1

您可以通过几种不同的方式将结果抓取并写入 csv 文件。我更喜欢和熊猫一起去。首先尝试使用 :has() 来过滤掉不需要的内容。那就是说以下应该起作用：

import requests
import pandas as pd
from bs4 import BeautifulSoup

base_URL = 'https://www.youpriboo.com/vorher_102_main_nat.php?action=show&liga='
liga = '2.BL'

URL = f"{base_URL}{liga}"

page = requests.get(URL, headers={"User-Agent": 'Mozilla/5.0'})
soup = BeautifulSoup(page.content, 'html.parser')

df = pd.DataFrame(columns=['Name_One','Name_Ano','Wahr'])
for tr in soup.select('.prognose_tab_1 tr:has(.greycell)'):
    name1 = tr.select('.hac')[1].get_text()
    name2 = tr.select('.hac')[2].get_text()
    wahr1 = tr.select('.greycell')[0].get_text()
    df = df.append({'Name_One':name1, 'Name_Ano':name2, 'Wahr':wahr1}, ignore_index=True)

    print(f"{name1} {name2} {wahr1}")

df.to_csv("youpriboo.csv", encoding='utf-8', index=False)

Answer 2

过滤不正确。试试这个方法，

table = soup.find_all("tr")
#print(table)
for row in table:
    data = row.find_all("td", class_="hac")
    if(len(data)>0):
        print(data[1].get_text(),data[2].get_text())

    data = row.find_all("td", class_="greycell")
    if(len(data)>0):
        print(data[0].get_text())

Answer 3

试试下面的代码。这会给你预期的输出。

import requests
from bs4 import BeautifulSoup

base_URL = 'https://www.youpriboo.com/vorher_102_main_nat.php?action=show&liga='
liga = '2.BL'
URL = base_URL + liga

headers = {
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36:'}

page = requests.get(URL, headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')
table=soup.select_one(".prognose_tab_1")
for tr in table.select('tr'):
  if tr.select('.hac') and tr.select('.greycell'):
     name1=tr.select('.hac')[1]
     name2 = tr.select('.hac')[2]
     wahr1 = tr.select('.greycell')[0]

     print(name1.get_text() +' '+ name2.get_text()+' '+ wahr1.get_text())

输出

Arminia Bielefeld VfB Stuttgart 34,43
SV Wehen Wiesbaden VfL Osnabrück 34,51
Jahn Regensburg Hamburger SV 24,18
Karlsruher SC 1. FC Heidenheim 37,70
VfL Bochum SV Darmstadt 98 55,22
Erzgebirge Aue Dynamo Dresden 37,70
FC St. Pauli SV Sandhausen 43,90
SpVgg Greuther Fürth Holstein Kiel 46,23
Hannover 96 1. FC Nürnberg 46,23