尝试在 python 中使用 selenium 抓取 etf.com [重复]答案

【问题标题】：Trying to scrape etf.com using selenium in pythin [duplicate]尝试在 python 中使用 selenium 抓取 etf.com [重复]
【发布时间】：2021-05-13 11:25:05
【问题描述】：

我试图抓取https://www.etf.com/KJUL 以获取页面上的表格>

我用python写了代码。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys    
path = "C:\Program Files (x86)\chromedriver.exe"
driver =webdriver.Chrome(path)
url ="https://www.etf.com/KJUL#overview"
driver.get("https://www.etf.com/KJUL") 
print(driver.page_source)
search =driver.find_elements_by_tag_name("rowText")

我无法从解析的内容中获取任何内容，因为我无法获取与表格相关的任何标签。我怎样才能拿到桌子？

【问题讨论】：

目标页面严重依赖 JavaScript 来填充其内容，包括您尝试定位的内容。 BeautifulSoup 是一个 HTML 解析器，不评估 JavaScript。检查r 的内容，这将证实这一点。使用 Selenium 或 Puppeteer 之类的浏览器控制实用程序，它会像您自己的浏览器一样评估页面上的 JavaScript。
你到底想要哪张桌子？
重复Web-scraping JavaScript page with Python
@baduker KJUL 汇总表如图所示

标签： python html selenium

【解决方案1】：

rowText 不是标签名称。您需要编写一个有效的定位器来抓取 rowText。

代码：

driver.get("https://www.etf.com/KJUL")
driver.execute_script("window.scrollTo(0, 1000)")
KJUL_list = driver.find_elements(By.CSS_SELECTOR, "section.generalDataBox div.rowtext")
for kjul in KJUL_list:
    print(kjul.text)

O/P：

Issuer
Innovator
Inception Date
07/01/20
Legal Structure
Open-Ended Fund
Expense Ratio
0.79%
Assets Under Management
$17.25M
Average Daily $ Volume
$182.82K
Average Spread (%)
0.23%
Competing ETFs
KJAN, KAPR, KOCT
Fund Home Page
Weighted Average Market Cap
--
Price / Earnings Ratio
--
Price / Book Ratio
--
Distribution Yield
--
Next Ex-Dividend Date
N/A
Number of Holdings
--
Index Tracked
No Underlying Index
Index Weighting Methodology
Fixed
Index Selection Methodology
Fixed
Segment Benchmark
MSCI USA Small Cap Index
Expense Ratio
0.79%
Median Tracking Difference (12 Mo)
--
Max. Upside Deviation (12 Mo)
--
Max. Downside Deviation (12 Mo)
--
Max LT/ST Capital Gains Rate
20.00% / 39.60%
Capital Gains Distributions (3 Year)
--
Tax on Distributions
Qualified dividends
Distributes K1
No
Legal Structure
Open-Ended Fund
OTC Derivative Use
Yes
Securities Lending Active
No
Securities Lending Split (Fund/Issuer)
No Policy
ETN Counterparty
N/A
ETN Counterparty Risk
N/A
Fund Closure Risk
High
Portfolio Disclosure
Daily
Avg. Daily Share Volume
6,924
Average Daily $ Volume
$182.82K
Median Daily Share Volume
1,125
Median Daily Volume ($)
$29.66K
Average Spread (%)
0.23%
Average Spread ($)
$0.06
Median Premium / Discount (12 Mo)
-0.05%
Max. Premium / Discount (12 Mo)
0.93% / -0.85%
Impediment to Creations
None
Market Hours Overlap
100.00%
Creation Unit Size (Shares)
25,000
Creation Unit/Day (45 Day Average)
0.04
Creation Unit Cost (%)
0.04%
Underlying Volume / Unit
--
Open Interest on ETF Options
0
Net Asset Value (Yesterday)
$26.47
ETF.com Implied Liquidity
N/A
Goodness of Fit (R2)
0.45
Beta
0.29
Up Beta
0.27
Down Beta
0.29
Downside Standard Deviation
1.05%
Segment Benchmark
MSCI USA Small Cap Index
KJUL Number of Holdings
--
Benchmark Constituents
--
Shared Holdings
--
Shared Holdings Weight
--

【讨论】：