试图从 rotogrinders 中抓取表格数据。任何帮助将不胜感激答案

【问题标题】：Trying to webscrape table data from rotogrinders. Any help would be greatly appreciated试图从 rotogrinders 中抓取表格数据。任何帮助将不胜感激
【发布时间】：2022-01-22 17:22:32
【问题描述】：

到目前为止，我有：

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import pandas as pd
        
driver = webdriver.Firefox()
url = r"https://rotogrinders.com/game-stats/nba-player?site=fanduel&range=yesterday"
driver.get(url)
cookies = driver.find_element_by_xpath(r'//*[@id="bc-close-cookie"]').click()
select = driver.find_element_by_xpath(r'/html/body/div[1]/div/section/div/section/div[2]/div[2]/a[2]').click()

我需要将表格数据抓取到 .csv 文件中。有什么建议吗？

【问题讨论】：

标签： python-3.x web-scraping html-table

【解决方案1】：

因为订阅弹窗：

发生此错误时，请在单击全部按钮之前单击不谢谢。

【讨论】：

是的，我必须接受 cookie。太感谢了！！！。现在我只需要弄清楚如何将数据抓取到 .csv 文件

【解决方案2】：

Selenium 在这里是多余的，因为数据在 <script> 标签内。您可以使用requests 通过一个简单的请求来获取它，然后a）使用BeautifulSoup 提取<script> 标记，然后解析它，或者b）使用正则表达式直接提取它。我选择了后者。

它采用漂亮的 json 格式，这意味着使用pandas 转换成表格相对容易。您还将使用 pnadas 写入 csv。

import requests
import pandas as pd
import re
import json

url = 'https://rotogrinders.com/game-stats/nba-player?site=fanduel&range=yesterday'
response = requests.get(url)

p = re.compile("var data = (\[.*\])")
result = p.search(response.text)

jsonStr = result.group(1)
jsonData = json.loads(jsonStr)

df = pd.DataFrame(jsonData)
df.to_csv('nba.csv', index=False)

输出：

print(df.to_string())
         id                      name   fpts gp fgm fga ftm fta 3pm 3pa 2pm 2pa reb ast stl blk to pts oreb pfoul tfoul ffoul    min fouls dd td    usg   pace   fg%   ft%   3p%   2p% team  pos                    player
0       947          Lance Stephenson   6.70  1   1   1   2   2   0   0   1   1   1   3   0   0  3   4    0     1     0     0  17.80     0  0  0  13.19      5     1     1     0     1  IND  SCW          Lance Stephenson
1      1079             Stephen Curry  58.00  1  12  27   9   9   6  16   6  11   5   8   1   0  2  39    0     3     0     0  43.95     0  0  0  32.40  33.50  0.44     1  0.38  0.55  GSW   CG             Stephen Curry
2      1087                Chris Paul  51.50  1   8  14   2   2   2   6   6   8   5  11   2   1  0  20    0     3     0     0  36.68     0  1  0  20.19     15  0.57     1  0.33  0.75  PHO  DIS                Chris Paul
3      1277                Taj Gibson  12.50  1   1   3   0   0   0   0   1   3   5   1   0   1  0   2    1     4     0     0  17.95     0  0  0   7.42      2  0.33     0     0  0.33  NYK   PB                Taj Gibson
4      1334            Andre Iguodala  24.00  1   1   2   2   2   0   0   1   2   5   4   0   4  4   4    2     2     0     0  31.32     0  0  0  10.47      5  0.50     1     0  0.50  GSW  3DW            Andre Iguodala
5      1485              JaVale McGee  10.80  1   3   5   2   2   0   0   3   5   4   0   0   0  2   8    1     3     0     0  17.82     0  0  0  17.69      7  0.60     1     0  0.60  PHO   PB              JaVale McGee
6     13301         Jonas Valanciunas  39.00  1   8  11   1   2   1   4   7   7  10   2   1   2  3  18    1     5     0     0  33.03     0  1  0  18.82     14  0.73  0.50  0.25     1  NOP   PB         Jonas Valanciunas
7     13312           Bismack Biyombo  19.30  1   2   3   5   7   0   0   2   3   4   1   2   0  2   9    2     3     0     0  27.90     0  0  0  12.06   6.50  0.67  0.71     0  0.67  PHO   PB           Bismack Biyombo
8     13315             Klay Thompson  13.90  1   6  17   0   0   0   7   6  10   2   1   0   0  2  12    1     1     0     0  23.10     0  0  0  33.47     18  0.35     0     0  0.60  GSW  3DW             Klay Thompson
9     13335              Kemba Walker  11.40  1   1   5   3   4   0   3   1   2   2   2   1   0  2   5    0     2     1     0  21.17     0  0  0  17.80      9  0.20  0.75     0  0.50  NYK   CG              Kemba Walker
10    13353                Alec Burks  19.60  1   3   9   5   5   2   6   1   3   3   2   1   0  3  13    1     3     0     0  22.58     0  0  0  26.32  13.50  0.33     1  0.33  0.33  NYK   CG                Alec Burks
11    13913             Evan Fournier  10.10  1   2   8   1   2   1   6   1   2   3   1   0   0  1   6    1     1     0     0  25.15     0  0  0  16.24      9  0.25  0.50  0.17  0.50  NYK  SHW             Evan Fournier
12    13942               Jae Crowder  21.60  1   4   9   2   2   3   6   1   3   3   0   2   0  1  13    0     3     0     0  32.65     0  0  0  13.33     11  0.44     1  0.50  0.33  PHO  3DW               Jae Crowder
13    13955               Jeremy Lamb  27.40  1   2   5   8  10   2   3   0   2   2   2   2   1  1  14    1     2     0     0  18.88     0  0  0  23.43     10  0.40  0.80  0.67     0  IND  SCW               Jeremy Lamb
14    14077            Garrett Temple   0.20  1   0   0   0   0   0   0   0   0   1   0   0   0  1   0    0     0     0     0  10.88     0  0  0   3.68      1     0     0     0     0  NOP  SHW            Garrett Temple
15    14559            Justin Holiday  23.50  1   6  13   0   0   4   9   2   4   5   3   0   0  3  16    1     2     0     0  34.22     0  0  0  19.86     15  0.46     0  0.44  0.50  IND  SHW            Justin Holiday
16    16879          Tim Hardaway Jr.  18.50  1   4  10   0   0   3   7   1   3   5   1   0   0  0  11    0     3     0     0  28.28     0  0  0  14.61     10  0.40     0  0.43  0.33  DAL  SHW          Tim Hardaway Jr.
17    16943            Reggie Bullock  11.70  1   1   3   0   0   1   3   0   0   6   1   0   0  0   3    1     1     0     0  20.18     0  0  0   6.60      2  0.33     0  0.33     0  DAL  3DW            Reggie Bullock
18    18566             Dwight Powell   8.60  1   2   3   2   3   0   0   2   3   3   0   0   0  1   6    2     1     0     0  14.35     0  0  0  14.83   3.50  0.67  0.67     0  0.67  DAL   VB             Dwight Powell
19    18620            Andrew Wiggins  22.80  1   5  15   0   1   1   6   4   9   4   2   1   1  2  11    2     5     0     0  38.03     0  0  0  19.04  15.50  0.33     0  0.17  0.44  GSW  SCW            Andrew Wiggins
20    18632             Julius Randle  24.40  1   1   9   2   4   0   2   1   7   7   6   1   1  3   4    2     1     1     0  29.48     0  0  0  21.36     12  0.11  0.50     0  0.14  NYK   VB             Julius Randle
21    18899        Kristaps Porzingis  37.70  1   7  15   2   2   2   4   5  11  11   1   0   2  1  18    2     5     0     0  32.28     0  1  0  21.33     15  0.47     1  0.50  0.45  DAL   VB        Kristaps Porzingis
22    18941             Cameron Payne  18.60  1   4  10   3   3   1   4   3   6   3   0   1   0  0  12    0     1     1     0  18.93     0  0  0  23.92  11.50  0.40     1  0.25  0.50  PHO  DIS             Cameron Payne
23    18945              Kevon Looney  36.50  1   5   6   3   4   0   0   5   6  15   3   2   0  5  13    6     3     0     1  28.17     0  1  0  19.52      7  0.83  0.75     0  0.83  GSW   PB              Kevon Looney
24    18949              Devin Booker  41.00  1  11  25   5   5   1   8  10  17   5   6   0   0  2  28    0     2     1     0  38.30     0  0  0  32.56  29.50  0.44     1  0.13  0.59  PHO  SCW              Devin Booker
25    31814           Nemanja Bjelica  22.20  1   4   5   0   0   0   1   4   4   6   2   1   1  2   8    1     1     0     0  14.13     0  0  0  21.68      6  0.80     0     0     1  GSW   VF           Nemanja Bjelica
26    35227            Brandon Ingram  30.00  1   4  10   6   8   1   1   3   9   5   6   1   0  3  15    0     2     0     0  26.88     0  0  0  27.53     17  0.40  0.75     1  0.33  NOP  SCW            Brandon Ingram
27    35995                Damion Lee  14.00  1   3   5   2   2   1   2   2   3   0   0   2   0  1   9    0     3     0     0     14     0  0  0  19.66      7  0.60     1  0.50  0.67  GSW   CG                Damion Lee
28    36032       Dorian Finney-Smith  14.20  1   2   6   0   0   1   5   1   1   6   2   0   0  1   5    1     4     0     0     32     0  0  0   9.57      6  0.33     0  0.20     1  DAL  3DW       Dorian Finney-Smith
29    36041            Gary Payton II  18.00  1   3   5   0   1   0   2   3   3   5   0   2   0  0   6    1     1     1     0  17.40     0  0  0  12.51   4.50  0.60     0     0     1  GSW   CG            Gary Payton II
30    37796           Frank Ntilikina      0  1   0   0   0   0   0   0   0   0   0   0   0   0  0   0    0     1     0     0   1.70     0  0  0   0.00      0     0     0     0     0  DAL  DIS           Frank Ntilikina
31    37845               Maxi Kleber  22.00  1   3   9   2   2   1   7   2   2   5   4   0   1  2   9    1     2     0     0  27.13     0  0  0  19.46     11  0.33     1  0.14     1  DAL   VB               Maxi Kleber
32    37861              Torrey Craig  32.90  1   5  10   0   0   2   5   3   5   7   1   3   1  1  12    0     3     0     0  34.22     0  0  0  13.24     11  0.50     0  0.40  0.60  IND  3DW              Torrey Craig
33    37889                 Josh Hart  34.40  1   5  10   5   9   2   4   3   6   7   4   1   0  0  17    1     4     0     0  35.28     0  0  0  17.32  13.50  0.50  0.56  0.50  0.50  NOP  SHW                 Josh Hart
34   408821         Mitchell Robinson  37.00  1   6   7   5  10   0   0   6   7  15   0   0   1  1  17    7     3     0     0  30.05     0  1  0  16.51      6  0.86  0.50     0  0.86  NYK   PB         Mitchell Robinson
35   408841             Landry Shamet   4.20  1   0   2   0   0   0   2   0   0   1   0   1   0  0   0    0     3     0     0  10.07     0  0  0   7.94      2     0     0     0     0  PHO  SHW             Landry Shamet
36   408842             Mikal Bridges  42.60  1   5  11   2   2   0   4   5   7   8   6   4   0  0  12    2     1     0     0  37.18     0  0  0  14.91     10  0.45     1     0  0.71  PHO  3DW             Mikal Bridges
37   408968           Devonte' Graham  19.10  1   5  16   1   1   4  10   1   6   3   1   0   0  1  15    0     1     0     0  28.03     0  0  0  25.36  17.50  0.31     1  0.40  0.17  NOP   CG           Devonte' Graham
38   408971               Luka Doncic  44.60  1   9  23   8  11   2   9   7  14   8   8   1   0  8  28    1     1     0     0  38.13     0  0  0  40.37  35.50  0.39  0.73  0.22  0.50  DAL   CG               Luka Doncic
39   409062             Jalen Brunson  20.50  1   8  13   2   2   1   2   7  11   5   1   0   0  6  19    2     3     0     0  34.75     0  0  0  23.26     18  0.62     1  0.50  0.64  DAL  DIS             Jalen Brunson
40   570141                Gary Clark   3.50  1   1   5   0   0   1   4   0   1   0   1   0   0  1   3    0     0     0     0   7.95     0  0  0  31.85      6  0.20     0  0.25     0  NOP   VF                Gary Clark
41  1115150                RJ Barrett  30.20  1   6  13   4   7   1   5   5   8   6   2   0   1  0  17    1     1     0     0  27.98     0  0  0  23.93  15.50  0.46  0.57  0.20  0.63  NYK  SCW                RJ Barrett
42  1115261              Goga Bitadze  32.30  1   5  12   2   5   1   4   4   8   9   5   0   1  2  13    4     2     2     0  31.35     0  0  0  22.78  12.50  0.42  0.40  0.25  0.50  IND   VB              Goga Bitadze
43  1115443              Jaxson Hayes   8.90  1   3   4   0   0   0   0   3   4   2   1   0   0  1   6    0     5     0     0  16.47     0  0  0  12.94      5  0.75     0     0  0.75  NOP   PB              Jaxson Hayes
44  1115445  Nickeil Alexander-Walker  13.60  1   1   4   2   2   0   0   1   4   3   4   0   0  0   4    0     3     0     0  24.42     0  0  0  10.16      5  0.25     1     0  0.25  NOP   CG  Nickeil Alexander-Walker
45  1115568              Jordan Poole  13.40  1   1   7   3   4   0   5   1   2   2   4   0   0  0   5    0     2     0     0  24.67     0  0  0  16.34      9  0.14  0.75     0  0.50  GSW   CG              Jordan Poole
46  1115625               Cam Johnson  19.90  1   2   7   2   2   1   6   1   1   7   1   1   0  0   7    0     2     0     0  20.47     0  0  0  16.04      8  0.29     1  0.17     1  PHO  SHW               Cam Johnson
47  1311751            Oshae Brissett  12.70  1   0   7   2   4   0   3   0   4   6   1   1   0  1   2    0     1     0     0  21.98     0  0  0  18.36     10     0  0.50     0     0  IND  SHW            Oshae Brissett
48  1333886     Juan Toscano-Anderson  15.30  1   2   5   0   0   1   3   1   2   4   1   2   0  2   5    1     4     0     0  14.87     0  0  0  19.72      6  0.40     0  0.33  0.50  GSW  SHW     Juan Toscano-Anderson
49  2439136                Obi Toppin   5.40  1   0   1   0   0   0   1   0   0   2   0   0   1  0   0    0     1     0     0  18.52     0  0  0   2.16      1     0     0     0     0  NYK   VB                Obi Toppin
50  2439366                Josh Green   4.70  1   1   3   0   0   0   0   1   3   1   1   0   0  0   2    0     1     0     0  11.18     0  0  0  11.91      3  0.33     0     0  0.33  DAL  3DW                Josh Green
51  2439514         Immanuel Quickley  31.30  1   4  13   4   4   2   9   2   4   4   5   3   0  4  14    0     1     0     0  23.28     0  0  0  35.07     19  0.31     1  0.22  0.50  NYK  SHW         Immanuel Quickley
52  3005116            Quentin Grimes  16.20  1   5   9   0   0   3   6   2   3   1   2   0   0  1  13    1     4     0     0  23.83     0  0  0  17.89      9  0.56     0  0.50  0.67  NYK  3DW            Quentin Grimes
53  3005227            Isaiah Jackson  33.90  1   5  12   5   8   0   1   5  11   7   1   3   0  0  15    3     5     0     0  18.63     0  0  0  34.03     13  0.42  0.63     0  0.45  IND   VF            Isaiah Jackson
54  3005228              Chris Duarte  48.90  1  10  16   5   5   2   3   8  13   7   3   3   0  0  27    2     3     0     0  38.53     0  0  0  19.92  16.50  0.63     1  0.67  0.62  IND  SCW              Chris Duarte
55  3005417                Herb Jones  28.80  1   5   7   0   0   1   3   4   4   4   4   3   0  2  11    1     3     0     0  37.08     0  0  0  11.13      8  0.71     0  0.33     1  NOP  3DW                Herb Jones
56  3007111          Jonathan Kuminga  14.50  1   0   0   5   6   0   0   0   0   5   3   0   0  1   5    1     3     1     0  15.37     0  0  0  12.05      3     0  0.83     0     0  GSW   VF          Jonathan Kuminga
57  3014026              Keifer Sykes  17.30  1   4  12   0   0   2   5   2   7   4   3   0   0  2  10    2     3     0     0  31.02     0  0  0  19.33     12  0.33     0  0.40  0.29  IND  DIS              Keifer Sykes
58  3015595          Duane Washington   8.50  1   3   7   0   0   2   2   1   5   0   1   0   0  1   8    0     1     0     0  18.37     0  0  0  18.14      8  0.43     0     1  0.20  IND  SHW          Duane Washington
59  3015773             Jose Alvarado  31.00  1   6   9   0   0   1   2   5   7   0   4   4   0  0  13    0     3     0     0  19.97     0  0  0  20.67      9  0.67     0  0.50  0.71  NOP  DIS             Jose Alvarado

【讨论】：