用 BeautifulSoup 抓取盒子 ccores 并使用 pandas 导出到 Excel

Question

我一直在尝试弄清楚如何使用 Python 3.6 以及 BeautifulSoup 和 Pandas 模块从 Fangraphs 中抓取棒球盒得分。我的最终目标是将网页的不同部分保存到 Excel 中的不同工作表中。

为了做到这一点，我想我必须通过各自的 id 标签分别拉出每个表。这是为构成第一个 Excel 工作表的四个表格（页面图表下方）执行此操作的代码。运行代码会导致这个错误：

Traceback (most recent call last):

File "Fangraphs Box Score Scraper.py", line 14, in <module>
df1 = pd.read_html(soup,attrs={'id': ['WinsBox1_dghb','WinsBox1_dghp','WinsBox1_dgab','WinsBox1_dgap']})

File "C:\Python36\lib\site-packages\pandas\io\html.py", line 906, in read_html
keep_default_na=keep_default_na)

File "C:\Python36\lib\site-packages\pandas\io\html.py", line 743, in _parse
raise_with_traceback(retained)

File "C:\Python36\lib\site-packages\pandas\compat\__init__.py", line 344, in raise_with_traceback

raise exc.with_traceback(traceback)

TypeError: 'NoneType' object is not callable

---

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'http://www.fangraphs.com/boxscore.aspx?date=2017-09-10&team=Red%20Sox&dh=0&season=2017'
response = requests.get(url)
soup = BeautifulSoup(response.text,"lxml")

df1 = pd.read_html(soup,attrs={'id': ['WinsBox1_dghb','WinsBox1_dghp','WinsBox1_dgab','WinsBox1_dgap']})

writer = pd.ExcelWriter('Box Scores.xlsx')
df1.to_excel(writer,'Traditional Box Scores')

Answer 1

你用错了id，你取自<div>，但需要取自<table>标签read_html attrs，我认为你不需要使用bs，试试看：

import pandas as pd

url = 'http://www.fangraphs.com/boxscore.aspx?date=2017-09-10&team=Red%20Sox&dh=0&season=2017'
df1 = pd.read_html(
    url,
    attrs={'id': ['WinsBox1_dghb_ctl00', 'WinsBox1_dgab_ctl00']}
)

# and now df1 it is list of df
writer = pd.ExcelWriter('Box Scores.xlsx')
row = 0
for df in df1:
    df.to_excel(writer, sheet_name='tables', startrow=row , startcol=0)   
    row = row + len(df.index) + 3

writer.save()