我的 len(count) 匹配但我得到 IndexError: list index out of range。我究竟做错了什么？

Question

我对网络抓取非常陌生，我正在尝试不同的方法来运行此代码，该代码适用于同一网站上的相同表格抓取（尽管 URL 不同），但我一无所获。

工作代码：

browser = webdriver.Chrome()

urls = {
    "https://www.oddsportal.com/soccer/england/premier-league/results/"
}
class GameData:

    def __init__(self):
        self.date = []
        self.time = []
        self.game = []
        self.score = []
        self.home_odds = []
        self.draw_odds = []
        self.away_odds = []
        self.country = []
        self.league = []


def parse_data(url):
    browser.get(url)
    df = pd.read_html(browser.page_source, header=0)[0]
    html = browser.page_source
    soup = bs(html, "lxml")
    cont = soup.find('div', {'id': 'wrap'})
    content = cont.find('div', {'id': 'col-content'})
    content = content.find('table', {'class': 'table-main'}, {'id': 'tournamentTable'})
    main = content.find('th', {'class': 'first2 tl'})
    if main is None:
        return None
    count = main.findAll('a')
    country = count[1].text
    league = count[2].text
    game_data = GameData()
    game_date = None
    for row in df.itertuples():
        if not isinstance(row[1], str):
            continue
        elif ':' not in row[1]:
            game_date = row[1].split('-')[0]
            continue
        game_data.date.append(game_date)
        game_data.time.append(row[1])
        game_data.game.append(row[2])
        game_data.score.append(row[3])
        game_data.home_odds.append(row[4])
        game_data.draw_odds.append(row[5])
        game_data.away_odds.append(row[6])
        game_data.country.append(country)
        game_data.league.append(league)
    return game_data




if __name__ == '__main__':

    results = None

    for url in urls:
        try:
            game_data = parse_data(url)
            if game_data is None:
                continue
            result = pd.DataFrame(game_data.__dict__)
            if results is None:
                results = result
            else:
                results = results.append(result, ignore_index=True)
        except ValueError:
            game_data = parse_data(url)
            if game_data is None:
                continue
            result = pd.DataFrame(game_data.__dict__)
            if results is None:
                results = result
        except AttributeError:
            game_data = parse_data(url)
            if game_data is None:
                continue
            result = pd.DataFrame(game_data.__dict__)
            if results is None:
                results = result
            else:
                results = results.append(result, ignore_index=True)

df:

|    | date              | time   | game                             | score   |   home_odds |   draw_odds |   away_odds | country   | league         |
|----|-------------------|--------|----------------------------------|---------|-------------|-------------|-------------|-----------|----------------|
|  0 | Yesterday, 11 May | 19:15  | Southampton - Crystal Palace     | 3:1     |        1.89 |        3.8  |        4.11 | England   | Premier League |
|  1 | Yesterday, 11 May | 17:00  | Manchester Utd - Leicester       | 1:2     |        3.72 |        3.58 |        2.07 | England   | Premier League |
|  2 | 10 May 2021       | 19:00  | Fulham - Burnley                 | 0:2     |        2.24 |        3.44 |        3.38 | England   | Premier League |
|  3 | 09 May 2021       | 18:00  | Arsenal - West Brom              | 3:1     |        1.5  |        4.53 |        6.76 | England   | Premier League |
|  4 | 09 May 2021       | 15:30  | West Ham - Everton               | 0:1     |        2.15 |        3.56 |        3.48 | England   | Premier League |

以下是我发现的差异：

Xpath的工作代码url：

urls = {
    "https://www.oddsportal.com/soccer/england/premier-league/results/"
}

//*[@id="tournamentTable"]/tbody/tr[4]/td[2]/a

所需网址的Xpath：

urls = {
    "https://www.oddsportal.com/matches/soccer/20210515/"
}

//*[@id="table-matches"]/table/tbody/tr[2]/td[2]/a[2]

当我跑步时

if main is None:
    return None
count = main.findAll('a')
print(len(count))

我明白了

2

我曾经问过这个问题并尝试过

content = cont.find('div', {'id': 'col-content'})
content = content.find('table', {'class': 'table-main'}, {'id': 'table-matches'})
main = content.find('th', {'class': 'first2 tl'})

但是我比以前更困惑了。

任何指导将不胜感激。

Answer 1

您的变量count 的len 为2。Python 索引从0 开始，这意味着count[2] 会给您一个错误（列表中只有2 个元素）。

请更改

country = count[1].text
league = count[2].text

到

country = count[0].text
league = count[1].text

Answer 2

因为你还是在使用 selenium 并且网站有 jQuery：

data = driver.execute_script('''
  let [country, league] = $('.bflp + a').get().map(a => a.innerText.trim())
  return $('tr.deactivate').get().map(tr => {
    let tds = $(tr).find('td').get().map(td => td.innerText.trim())
    return {
      date: $(tr).prevAll('tr.center').find('th').first().text().trim(),
      time: tds[0],
      game: tds[1],
      score: tds[2],
      home_odds: tds[3],
      draw_odds: tds[4],
      away_odds: tds[5],
      country,
      league
    }
  })
''')

如果您不喜欢这样，您可以使用获取国家和联赛的位。