使用 Selenium 抓取时未加载 Javascript

Question

我正在尝试抓取 steamdb.info 以下载他们制作的应用程序的图表。当我使用普通浏览器访问游戏图表时，图表将加载。但是，当我使用 selenium 时，图表不会加载，并且他们说“必须启用 Javascript”。我认为这个网站可以告诉我正在使用机器人并且不会加载 javascript。这是我的代码：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
import pandas as pd
import time
from random import randint

fp = webdriver.FirefoxProfile()
#fp.set_preference("browser.download.folderList",2);
#fp.set_preference("browser.download.manager.showWhenStarting",False)
#fp.set_preference("browser.download.dir", "/home/brianebrahimi/Desktop/steamdb_info_scraper/output/")
#fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/pdf")
#fp.set_preference("browser.download.useDownloadDir", True)
#fp.set_preference("pdfjs.disabled", True)
fp.set_preference("javascript.enabled", True)
fp.set_preference("general.useragent.override", "brianebrahimi")

driver = webdriver.Firefox(firefox_profile = fp)
driver.maximize_window()



start_url = "https://steamdb.info/apps/"

driver.get(start_url)

games_set = set()
while driver.find_element_by_xpath("//a[@aria-label='Next Page']"):
    for game in driver.find_elements_by_xpath("/html/body/div[1]/div[1]/div[3]/table/tbody/tr"):
        time.sleep(randint(5,8))
        if game.find_element_by_xpath(".//td[2]/a").text not in games_set:
            game_app_id = game.find_element_by_xpath(".//td[2]/a").text
            games_set.add(game_app_id)
            game.find_element_by_xpath("//td[2]/a").click()
            time.sleep(randint(5, 8))
            try:
                #driver.find_element_by_xpath("//a[@id = 'tab-graphs']")
                driver.find_element_by_xpath("//a[@id = 'tab-graphs']").click()
                time.sleep(randint(5,20))
                for download in driver.find_elements_by_xpath("//li[@class='highcharts-menu-item']"):
                    if download.text == "Download CSV":
                        download.click()
            except:
                driver.get(start_url)

    driver.find_element_by_xpath("//a[@aria-label='Next Page']").click()

我做错了什么？如何加载这些图表？

Answer 1

如果您认为自己被检测为机器人，请尝试undetected_chromedriver。