用 selenium 抓取网页到 txt

Question

我会从这个页面上刮掉 ID https://www.flashscore.co.uk/football/russia/premier-league/results/ 然后将 g_1_替换为https://www.flashscore.com/match/ 并将这些url导入txt文件。

我用过这段代码

matches=WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[starts-with(@id,'g_1_')]")))

for match in matches:
    g1 = matches.replace("g_1_", "https://www.flashscore.com/match/")
    print(g1)

但我得到了这个错误

AttributeError: 'list' object has no attribute 'replace'

id that i want to scrape

Answer 1

此错误消息...

AttributeError: 'list' object has no attribute 'replace'

...表示在您的程序中您已在 list 上调用了 replace() 方法，其中 replace() 方法将 specified 短语替换为另一个指定短语.

您需要对 list 中每个元素的文本调用 replace() 方法。

---

解决方案

您可以从元素中收集文本/短语并创建列表，而不是收集元素。实际上，您的代码块将是：

match_texts = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[starts-with(@id,'g_1_')]")))]
for match_text in match_texts:
    g1 = match_text.replace("g_1_", "https://www.flashscore.com/match/")
    print(g1)

Answer 2

首先，如 cmets 中所述，.replace() 是一种应用于字符串的方法。你有matches，它是一个列表对象（WebElements），它会抛出错误'list' object has no attribute 'replace''你需要遍历你用for match in matches:定义的WebElements列表，然后获取id属性作为带有.get_attribute() 的字符串，以便使用replace() 方法。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


#Initializing the webdriver
options = webdriver.ChromeOptions()

#Uncomment the line below if you'd like to scrape without a new Chrome window every time.
#options.add_argument('headless')

#Change the path to where chromedriver is in your home folder.
driver = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe', options=options)
driver.maximize_window()

url = 'https://www.flashscore.co.uk/football/russia/premier-league/results/'
driver.get(url)
matches=WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[starts-with(@id,'g_1_')]")))

for match in matches:
    g1 = match.get_attribute('id')
    g1 = g1.replace("g_1_", "https://www.flashscore.com/match/")
    print(g1)
    
driver.close()

你也可以把它组合成一个单行

g1 = match.get_attribute('id').replace("g_1_", "https://www.flashscore.com/match/")

输出：

https://www.flashscore.com/match/hWhb9Uyh
https://www.flashscore.com/match/rLoB6SLA
https://www.flashscore.com/match/zer38lib
https://www.flashscore.com/match/Eos77864
https://www.flashscore.com/match/4zzK46jN
https://www.flashscore.com/match/tdkfAAMo
https://www.flashscore.com/match/MBpF5nyH
https://www.flashscore.com/match/IwvO3Q5T
https://www.flashscore.com/match/nysS6yGg
https://www.flashscore.com/match/f1pz5Fp6
https://www.flashscore.com/match/jTwq3gFI
https://www.flashscore.com/match/QLhJ8cos
https://www.flashscore.com/match/0voW5eVa
https://www.flashscore.com/match/Yiqv4ZaC
https://www.flashscore.com/match/4CiN7H0m
https://www.flashscore.com/match/Sh1CoRqo