我可以看到表格，但无法使用 BS4 提取答案

【问题标题】：I can see the table but cannot extract using BS4我可以看到表格，但无法使用 BS4 提取
【发布时间】：2021-05-05 13:19:32
【问题描述】：

我一直在尝试从https://www.zacks.com/stock/research/MMM/earnings-announcements 中提取表格。我尽力了，我无法提取表格。有一个类似的帖子BeautifulSoup parse table data that doesn't load immediately，但由于该帖子已有 4 年历史，我无法复制它。我想从那以后网站已经改变了。

这是我的代码：

import requests
from bs4 import BeautifulSoup
header = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:32.0) Gecko/20100101 Firefox/32.0',}
url = 'https://www.zacks.com/stock/research/MMM/earnings-announcements'
r = requests.get(url, headers=header, verify=False)
dfs = pd.read_html(r.text)

我得到一个写着“正在加载”的空表，而不是表。但是，如果我这样做soup=BeautifulSoup(r.text,"lxml") 我可以看到表格数据在那里，但我无法提取它。我将不胜感激任何帮助

【问题讨论】：

要真正查看页面上的内容，请关闭JavaScript，然后您将看到没有桌子。换句话说，你的表是动态添加的，bs4 不会看到这个。这是一个提示。看一下页面源代码。
@baduker 谢谢，那很不幸，那我应该如何使用 JavaScript，我是编码新手，如果您能提供一些从哪里开始的信息，我将不胜感激

标签： python pandas beautifulsoup

【解决方案1】：

数据嵌入在 Javascript 的页面中。您可以使用此示例如何加载它：

import re
import json
import requests
from bs4 import BeautifulSoup


headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0"
}

url = "https://www.zacks.com/stock/research/MMM/earnings-announcements"
html_doc = requests.get(url, headers=headers).text

m = re.search(
    r"document.obj_data = (\{.*?\});",
    html_doc,
    flags=re.S,
)
data = json.loads(m.group(1))

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

# print some data:
for row in data["earnings_announcements_earnings_table"]:
    print(
        ("{:<15}" * 7).format(
            *[
                BeautifulSoup(cell, "html.parser").get_text(strip=True)
                for cell in row
            ]
        )
    )

打印：

4/27/2021      3/2021         $2.25          $2.77          +0.52          +23.11%        Before Open    
1/26/2021      12/2020        $2.19          $2.38          +0.19          +8.68%         Before Open    
10/27/2020     9/2020         $2.27          $2.43          +0.16          +7.05%         Before Open    
7/28/2020      6/2020         $1.77          $1.78          +0.01          +0.56%         Before Open    
4/28/2020      3/2020         $2.02          $2.16          +0.14          +6.93%         Before Open    
1/28/2020      12/2019        $2.10          $1.95          -0.15          -7.14%         Before Open    
10/24/2019     9/2019         $2.47          $2.58          +0.11          +4.45%         Before Open    
7/25/2019      6/2019         $2.04          $2.20          +0.16          +7.84%         Before Open    
4/25/2019      3/2019         $2.50          $2.23          -0.27          -10.80%        Before Open    
1/29/2019      12/2018        $2.27          $2.31          +0.04          +1.76%         Before Open    
10/23/2018     9/2018         $2.70          $2.58          -0.12          -4.44%         Before Open    
7/24/2018      6/2018         $2.59          $2.59          0.00           0.00%          Before Open    
4/24/2018      3/2018         $2.50          $2.50          0.00           0.00%          Before Open    
1/25/2018      12/2017        $2.03          $2.10          +0.07          +3.45%         Before Open    
10/24/2017     9/2017         $2.21          $2.33          +0.12          +5.43%         Before Open    
7/25/2017      6/2017         $2.59          $2.58          -0.01          -0.39%         Before Open    
4/25/2017      3/2017         $2.07          $2.16          +0.09          +4.35%         Before Open    
1/24/2017      12/2016        $1.87          $1.88          +0.01          +0.53%         Before Open    
10/25/2016     9/2016         $2.14          $2.15          +0.01          +0.47%         Before Open    
7/26/2016      6/2016         $2.08          $2.08          0.00           0.00%          Before Open    
4/26/2016      3/2016         $1.92          $2.05          +0.13          +6.77%         Before Open    
1/26/2016      12/2015        $1.62          $1.80          +0.18          +11.11%        Before Open    
10/22/2015     9/2015         $2.01          $2.05          +0.04          +1.99%         Before Open    
7/23/2015      6/2015         $1.99          $2.02          +0.03          +1.51%         Before Open    
4/23/2015      3/2015         $1.92          $1.85          -0.07          -3.65%         Before Open    
1/27/2015      12/2014        $1.79          $1.81          +0.02          +1.12%         Before Open    
10/23/2014     9/2014         $1.96          $1.98          +0.02          +1.02%         Before Open    
7/24/2014      6/2014         $1.91          $1.91          0.00           0.00%          Before Open    
4/24/2014      3/2014         $1.79          $1.79          0.00           0.00%          Before Open    
1/30/2014      12/2013        $1.61          $1.62          +0.01          +0.62%         Before Open    
10/24/2013     9/2013         $1.76          $1.78          +0.02          +1.14%         Before Open    
7/25/2013      6/2013         $1.70          $1.71          +0.01          +0.59%         Before Open    
4/25/2013      3/2013         $1.65          $1.61          -0.04          -2.42%         Before Open    
1/24/2013      12/2012        $1.41          $1.41          0.00           0.00%          Before Open    
10/23/2012     9/2012         $1.65          $1.65          0.00           0.00%          Before Open    
7/26/2012      6/2012         --             --             --             --             Before Open    
4/24/2012      3/2012         --             --             --             --             Before Open    
1/26/2012      12/2011        --             --             --             --             Before Open    
10/25/2011     9/2011         --             --             --             --             Before Open    
7/26/2011      6/2011         --             --             --             --             Before Open    
4/26/2011      3/2011         --             --             --             --             Before Open    
1/25/2011      12/2010        --             --             --             --             Before Open    
10/28/2010     9/2010         --             --             --             --             Before Open    
7/22/2010      6/2010         --             --             --             --             --             
4/27/2010      3/2010         --             --             --             --             Before Open    
1/28/2010      12/2009        --             --             --             --             Before Open    
10/22/2009     9/2009         --             --             --             --             Before Open    
7/23/2009      6/2009         --             --             --             --             Before Open

编辑：创建数据框：

all_rows = []
for row in data["earnings_announcements_earnings_table"]:
    r = [
        BeautifulSoup(cell, "html.parser").get_text(strip=True) for cell in row
    ]
    all_rows.append(r)

df = pd.DataFrame(all_rows)
print(df)

【讨论】：

我有最后一个简单的问题，如何将最后一部分转换为数据框而不是打印。
@BogdanTitomir 不打印该行，而是将该行附加到一个列表（名称all_rows），然后创建一个数据框df = pd.DataFrame(all_rows)。查看我的编辑。
漂亮！谢谢你，就像一个魅力！