【问题标题】:webscraping python - Exception has occurred: InvalidURL Failed to parse: <Response [404]>webscraping python - 发生异常:InvalidURL 无法解析:<Response [404]>
【发布时间】:2021-07-18 15:38:51
【问题描述】:

我试图在 transfermarkt 网站上获取一些球员的比赛、进球和助攻数

在第一个代码中,我可以得到我想要的:

    import requests
from bs4 import BeautifulSoup

# headers for be a nav
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36'}

# url site
site = "https://www.transfermarkt.com/lionel-messi/leistungsdaten/spieler/28003/plus/0?saison=ges"

# download page
objeto_response = requests.get(site, headers=headers)

page = BeautifulSoup(objeto_response.content, 'html.parser')
player_name = []
tag_player = page.find("h1", {"itemprop": "name"})
player_name.append(tag_player.text)
print(player_name)

numbers_player = []
games = page.find_all("td", {"class": "zentriert"})
goals = page.find_all("td", {"class": "zentriert"})
assists = page.find_all("td", {"class": "zentriert"})
numbers_player.append(games[0].text)
numbers_player.append(goals[1].text)
numbers_player.append(assists[2].text)
print(numbers_player)

输出:

['Lionel Messi']
['772', '667', '292']

但我想从更多玩家那里获取这些数据 所以我这样尝试:

import requests
from bs4 import BeautifulSoup

# headers for be a nav
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get("https://www.sikayetvar.com/onedio", headers=headers)

players = ['messi', 'cristiano', 'neymar']

for player in players:
    i = requests.get(
        "https://www.transfermarkt.com/schnellsuche/ergebnis/schnellsuche?query="+player+"&x=0&y=0")
player_page = BeautifulSoup(i.content, features="html.parser")
player_num = player_page.select(
    "td.hauptlink > a.spielprofil_tooltip tooltipstered > href")

# download page
objeto_response = requests.get(i, headers=headers)

page = BeautifulSoup(objeto_response.content, 'html.parser')
player_name = []
tag_player = page.find("h1", {"itemprop": "name"})
player_name.append(tag_player.text)
print(player_name)

numbers_player = []
games = page.find_all("td", {"class": "zentriert"})
goals = page.find_all("td", {"class": "zentriert"})
assists = page.find_all("td", {"class": "zentriert"})
numbers_player.append(games[0].text)
numbers_player.append(goals[1].text)
numbers_player.append(assists[2].text)
print(numbers_player)

和:

Exception has occurred: InvalidURL
Failed to parse: <Response [404]>
  File "C:\Python\Projetos\foot_project\foot2.py", line 18, in <module>
    objeto_response = requests.get(i, headers=headers)

有人可以帮助我吗? 我认为这是非常愚蠢的事情,但我不明白

【问题讨论】:

    标签: python parsing web-scraping python-requests html-parsing


    【解决方案1】:

    我把你的代码改成这个

    import requests
    from bs4 import BeautifulSoup
    
    # headers for be a nav
    headers = {'User-Agent': 'Mozilla/5.0'}
    
    players = ['messi', 'cristiano', 'neymar']
    url = "https://www.transfermarkt.com/schnellsuche/ergebnis/schnellsuche?query="
    urls = []
    for player in players:
        i = requests.get(url+player, headers=headers)
        urls.append(i)  
    player_page = BeautifulSoup(i.content, features="html.parser")
    player_num = player_page.select(
        "td.hauptlink > a.spielprofil_tooltip tooltipstered > href")
    # download page
    # objeto_response = requests.get(i, headers=headers)
    for i in urls:
        page = BeautifulSoup(i.content, 'html.parser')
        player_name = []
        tag_player = page.find("h1", {"itemprop": "name"})
        player_name.append(tag_player.text)
        print(player_name)
    
        numbers_player = []
        games = page.find_all("td", {"class": "zentriert"})
        goals = page.find_all("td", {"class": "zentriert"})
        assists = page.find_all("td", {"class": "zentriert"})
        numbers_player.append(games[0].text)
        numbers_player.append(goals[1].text)
        numbers_player.append(assists[2].text)
        print(numbers_player)
    

    请求问题已解决

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2013-09-12
      • 2012-06-05
      • 1970-01-01
      • 2022-01-03
      • 1970-01-01
      • 1970-01-01
      • 2015-08-04
      • 1970-01-01
      相关资源
      最近更新 更多