网页抓取数据表到excel

Question

我正在尝试从网站抓取数据以生成 Excel。现在它工作正常，但是当它搜索像 Sergio Rodriguez 这样的人时，会出现多个名称 (https://basketball.realgm.com/search?q=Sergio+Rodriguez)，因此它会跳过名称并抛出“No International table for Sergio Rodriguez”。我如何从该列表中选择一个在 NBA 打球的球员，并继续从每场比赛和高级统计数据表中脱颖而出？在这种情况下，当你搜索他的名字时，rodriguez 是第二个。

import requests
from bs4 import BeautifulSoup
import pandas as pd


playernames=['Carlos Delfino', 'Sergio Rodriguez']

result = pd.DataFrame()
for name in playernames:

    fname=name.split(" ")[0]
    lname=name.split(" ")[1]
    url="https://basketball.realgm.com/search?q={}+{}".format(fname,lname)
    response = requests.get(url)

    soup = BeautifulSoup(response.content, 'html.parser')


    try:
        table1 = soup.find('h2',text='International Regular Season Stats - Per Game').findNext('table')
        table2 = soup.find('h2',text='International Regular Season Stats - Advanced Stats').findNext('table')

        df1 = pd.read_html(str(table1))[0]
        df2 = pd.read_html(str(table2))[0]

        commonCols = list(set(df1.columns) & set(df2.columns))
        df = df1.merge(df2, how='left', on=commonCols)
        df['Player'] = name

    except:
        print ('No international table for %s.' %name)
        df = pd.DataFrame([name], columns=['Player'])

    result = result.append(df, sort=False).reset_index(drop=True)

cols = list(result.columns)
cols = [cols[-1]] + cols[:-1]
result = result[cols]
result.to_csv('international players.csv', index=False)

Answer 1

检查您收到的页面的 URL，搜索结果为单个匹配项会将您定向到

https://basketball.realgm.com/player/{player-name}/Summary/{player-id}

但是当你得到多个结果时

https://basketball.realgm.com/search?q={player-name}

为两个url写一个解析器函数，如（伪代码）

...
for name in playernames:
    fname=name.split(" ")[0]
    lname=name.split(" ")[1]
    url="https://basketball.realgm.com/search?q={}+{}".format(fname,lname)
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    # check the response url
    if (response.url == "https://basketball.realgm.com/search..."):
        # parse the search results, finding the players you want
        ... get urls from the table ...
        soup.table...  # etc.
        foreach url in table:
            response = requests.get(player_url)
            soup = BeautifulSoup(response.content, 'html.parser')
            # call the parse function for a player page
            ...
            parse_player(soup)
    else: # we have a player page
        # call the parse function for a player page, same as above
        ...
        parse_player(soup)
    ...

有少量的代码重复，但是当您了解它并使其正常工作时，不要担心。