当从篮球参考中抓取数据时，某些表格是如何被注释掉的？

Question

我正在尝试使用 BeautifulSoup 收集球员的所有篮球参考数据。让我们以迈克尔乔丹为例：https://www.basketball-reference.com/players/j/jordami01.html。问题是当我抓取 html 页面并解析 html 时，我只能抓取一个数据表，而其他数据似乎被注释掉了。我对 python 很陌生，希望有人能告诉我为什么 html 似乎有某些数据表作为 cmets。有人可以指导我解决方法吗？

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import pandas as pd

MJ_url = 'https://www.basketball-reference.com/players/j/jordami01.html'

uClient = uReq(MJ_url)

MJ_html = uClient.read()

uClient.close()

MJ_soup = soup(MJ_html, "html.parser")

MJ_containers = MJ_soup.findAll("table",{"class":"row_summable sortable 
stats_table"})

Answer 1

试试这个。 cmets中的所有数据现在都通过了：

import requests
from bs4 import BeautifulSoup, Comment

res = requests.get("https://www.basketball-reference.com/players/j/jordami01.html",headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text, 'lxml')
for comment in soup.find_all(string=lambda text:isinstance(text,Comment)):
    data = BeautifulSoup(comment,"lxml")
    for items in data.select("table.row_summable tr"):
        tds = [item.get_text(strip=True) for item in items.select("th,td")]
        print(tds)