遍历 driver.get 链接 Selenium Python

Question

我在遍历 driver.get url 以从后续页面中抓取信息时遇到问题。

背景：

我的目标是获取给定页面上所有股票的股票代码名称，然后转到下一页并再次执行此操作。

到目前为止，我已经能够在任何给定页面上获取代码名称信息作为列表（请参见下面的代码和图片）。

我还能够将 url 从第 1 页一直更改到第 5 页（如下面代码中的打印语句所示）。

我卡住的地方正在尝试遍历 driver.get url。

我也尝试过使用请求，但我得到一个空列表

url = https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page="
stock_tickers = []

for page in range(1,10):
    page_url = url+str(page)
    
    res = requests.get(page_url)
    
    soup = bs4.BeautifulSoup(res.text,"lxml")
    
    for name in soup.find_all('a'):
        name.get('href')
        stock_tickers.append(name.text)

有没有办法遍历 driver.get url？

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import bs4

PATH = "C:\\Users\\MYUSERNAME\\Webdrivers\\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1")

string = "https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1]"
new_character = '1'
position = -2
new_character_int = int(new_character)
new_string = string[:position] + str(new_character_int) + string[position+1:]

while new_character_int <=5:
    
    new_string = string[:position] + str(new_character_int) + string[position+1:]
    
    html = driver.page_source 

    soup = bs4.BeautifulSoup(html, 'html.parser')
    
    stock_tickers = []

    for name in soup.find_all('a'):
        name.get('href')
        stock_tickers.append(name.text)
        
    
    new_character_int +=1
    print(new_string)
    
stock_tickers[27:67:2]

Output from code above

Answer 1

如果您尝试循环并且您的新链接是 new_string，您必须在驱动程序中获取该链接才能处理 html。

目前您打印您的链接，但您还需要获取这些链接，请参阅下面的修复

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import bs4

PATH = "C:\\Users\\MYUSERNAME\\Webdrivers\\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1")

string = "https://swingtradebot.com/equities?min_vol=250000&min_price=3&max_price=999999.0&adx_trend=&grade=&include_etfs=0&html_button=as_html&page=1]"
new_character = '1'
position = -2
new_character_int = int(new_character)
new_string = string[:position] + str(new_character_int) + string[position+1:]

while new_character_int <=5:
    
    new_string = string[:position] + str(new_character_int) + string[position+1:]
    driver.get(new_string)
    html = driver.page_source 

    soup = bs4.BeautifulSoup(html, 'html.parser')
    
    stock_tickers = []

    for name in soup.find_all('a'):
        name.get('href')
        stock_tickers.append(name.text)
        
    
    new_character_int +=1
    print(new_string)
    
stock_tickers[27:67:2]