beautifulsoup 抓取实时值

Question

我正在尝试获取个人项目的汇率，我使用 css 选择器来获取值所在的类。网站上有一个 javascript 提供这些值，似乎我对开发人员控制台不太熟悉，我检查了它，但在网络部分看不到任何实时运行的东西。这是我写的代码，到目前为止，它带出了一长串破折号。令人惊讶的是，破折号与应该显示的那些部分的源代码相匹配。

from bs4 import BeautifulSoup
import requests
r = requests.get("https://www.ig.com/en/forex/markets-forex")
soup = BeautifulSoup(r.content, "html.parser")
results = soup.findAll("span",attrs={"data-field": "CPT"})
for span in results:
    print(span.text)

Answer 1

通过 JS 填充的跨度元素，动态值。在开始时，每个跨度元素都包含“-”。 您需要 js 驱动程序等待填充元素，然后从 span 中获取值。

使用硒：

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome('./chromedriver')
driver.get('https://www.ig.com/en/forex/markets-forex')

for elm in driver.find_elements(By.CSS_SELECTOR, "span[data-field=CPT]"):
    print(elm, elm.text)

从https://sites.google.com/a/chromium.org/chromedriver/home下载chromedriver

另外，dryscrape + bs4，但 dryscrape 似乎已经过时了。示例here

修改：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome('./chromedriver')
driver.get('https://www.ig.com/en/forex/markets-forex')

time.sleep(2) # Maybe more or less, how much faster page load

for elm in driver.find_elements(By.CSS_SELECTOR, "span[data-field=CPT]"):
    if elm.text:
        print(elm, elm.text)

或

import time
from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome('./chromedriver')
driver.get('https://www.ig.com/en/forex/markets-forex')

data = []
while not data:
    for elm in driver.find_elements(By.CSS_SELECTOR, "span[data-field=CPT]"):
        if elm.text and elm.text != '-': # Maybe check on contains digit
            data.append(elm.text)
    time.sleep(1)
print(data)