我正在尝试抓取一些财务数据，但每当我使用硒或美汤时，我都会得到空列表
这是我与 selenium 一起使用的代码
driver = webdriver.Chrome(os.getcwd() + '/chromedriver')
driver.get("https://www.amarkets.com/research-education/sentiment-widget/")
time.sleep(3)
real_soup = BeautifulSoup(driver.page_source, 'html.parser')
sentiments = real_soup.findAll("TD")
print(sentiments)
我不断收到一个空列表。
【问题讨论】：
标签： python selenium web-scraping beautifulsoup
不要抓取它 - 使用 api（您可以通过检查 network activity 找到）：
import requests
import pandas as pd
r = requests.get('https://api.prod.amarkets.dev/v1/sentiment-widget-dev/?request=%7B%22Symbols%22%3A%5B%22EURUSD%22%2C%22GBPUSD%22%2C%22USDJPY%22%2C%22AUDUSD%22%2C%22NZDUSD%22%2C%22USDCAD%22%2C%22USDCHF%22%2C%22EURJPY%22%2C%22EURGBP%22%2C%22XAUUSD%22%2C%22XAGUSD%22%2C%22DowJones30%22%2C%22S%26P500%22%2C%22Nasdaq100%22%2C%22BTCUSD%22%2C%22BRENT%22%5D%2C%22NodesType%22%3A%22Real%22%2C%22CollectSuffixes%22%3Atrue%2C%22RequestName%22%3A%22RequestWidgetSentiments%22%7D')
df = pd.DataFrame(r.json()['Result'])
结果：
| Symbol | Time | Bid | Ask | Digits | Spread | MinSpread | Buyers | Sellers | VolumeBuyInUsd | VolumeSellInUsd | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | AUDUSD | 2021-11-10 17:57:39:216 | 0.73826 | 0.73852 | 5 | 2.6 | 2.1 | 127 | 126 | 901424 | 1.00069e+06 |
| 1 | BRENT | 2021-11-10 17:57:39:216 | 84.88 | 84.91 | 2 | 3 | 3 | 84 | 159 | 1.1943e+06 | 1.82476e+06 |
| 2 | BTCUSD | 2021-11-10 17:57:39:216 | 68344 | 68374.3 | 2 | 3024 | 1243 | 86 | 87 | 189964 | 341095 |
| 3 | DowJones30 | 2021-11-10 17:57:39:216 | 36215 | 36223 | 0 | 8 | 7 | 40 | 75 | 262533 | 2.2185e+06 |
| 4 | EURGBP | 2021-11-10 17:57:39:216 | 0.85477 | 0.8549 | 5 | 1.3 | 1.3 | 65 | 89 | 454269 | 890184 |
【讨论】：