如何通过格式化的网络抓取循环过滤列表

Question

我有一个篮球运动员列表，我想通过我已经设置的网络抓取循环传递这些球员。球员名单是2011年NBA选秀名单。我想遍历每个球员并从他们大学最后一年获得他们的大学统计数据。问题是一些被选中的球员没有上过大学，因此没有以他们的名字格式化的网址，所以每次我通过甚至一个没有在大学打球的球员时，整个代码都会出错。我试过包括“通过”和“继续”，但似乎没有任何效果。这是迄今为止我得到的最接近的：

from bs4 import BeautifulSoup
import requests 
import pandas as pd 
headers = {'User Agent':'Mozilla/5.0'}

players = [
   'kyrie-irving','derrick-williams','enes-kanter',
   'tristan-thompson','jonas-valanciunas','jan-vesely',
   'bismack-biyombo','brandon-knight','kemba-walker,
   'jimmer-fredette','klay-thompson'
]
#the full list of players goes on for a total of 60 players, this is just the first handful

player_stats = []

 for player in players:
    url = (f'https://www.sports-reference.com/cbb/players/{player}-1.html')
    res = requests.get(url)
    #if player in url:
        #continue
    #else:
        #print("This player has no college stats")
#Including this if else statement makes the error say header is not defined. When not included, the error says NoneType object is not iterable       
    soup = BeautifulSoup(res.content, 'lxml')
    header = [th.getText() for th in soup.findAll('tr', limit = 2)[0].findAll('th')]
    rows = soup.findAll('tr')
    player_stats.append([td.getText() for td in soup.find('tr', id ='players_per_game.2011')])
    player_stats

graph = pd.DataFrame(player_stats, columns = header)

Answer 1

你可以做两件事中的一件：

1. 检查响应状态代码。 200是成功响应，其他都是错误。问题是某些网站会有一个有效的 html 页面来表示“无效页面”，因此您仍然可以获得成功的 200 响应。

2. 只需使用 try/except。如果失败，继续列表中的下一项

由于选项 1 的问题，请在此处选择选项 2。另外，您是否考虑过使用 pandas 来解析表格？做起来容易一点（并且在后台使用 BeautifulSoup）？

最后，您需要对此进行更多的逻辑处理。有多名大学球员“Derrick William”。我怀疑你的意思不是https://www.sports-reference.com/cbb/players/derrick-williams-1.html。所以你需要弄清楚如何解决这个问题。

from bs4 import BeautifulSoup
import requests 
import pandas as pd 
headers = {'User Agent':'Mozilla/5.0'}

players = [
   'kyrie-irving','derrick-williams','enes-kanter',
   'tristan-thompson','jonas-valanciunas','jan-vesely',
   'bismack-biyombo','brandon-knight','kemba-walker',
   'jimmer-fredette','klay-thompson'
]
#the full list of players goes on for a total of 60 players, this is just the first handful

player_stats = []

for player in players:
    url = (f'https://www.sports-reference.com/cbb/players/{player}-1.html')
    res = requests.get(url)

    try:
        soup = BeautifulSoup(res.content, 'lxml')
        header = [th.getText() for th in soup.findAll('tr', limit = 2)[0].findAll('th')]
        rows = soup.findAll('tr')
        player_stats.append([td.getText() for td in soup.find('tr', id ='players_per_game.2011')])
        player_stats
    except:
        print("%s has no college stats" %player)

graph = pd.DataFrame(player_stats, columns = header)

与熊猫：

graph = pd.DataFrame()
for player in players:
    try:
        url = (f'https://www.sports-reference.com/cbb/players/{player}-1.html')
        df = pd.read_html(url)[0]
        cols = list(df.columns)
        
        df = df.iloc[-2][cols]
        df['Player'] = player
        
        graph = graph.append(df).reset_index(drop=True)
        graph = graph[['Player'] + cols]
    except:
        print("%s has no college stats" %player)