无法从具有复杂布局的表中抓取三个字段

Question

我在 python 和 selenium 中创建了一个脚本来解析网站中可用表中的三个字段 franking credit、gross divident 和 further information。只有当浏览器点击带有加号的黄色圆形按钮时，最后两个字段才会显示。

但是，当单击按钮时，它们会变成红色，表示信息已显示。

我的脚本可以点击所有按钮，但无法从该表中抓取三个字段。

我附上了一张图片，向您展示它的真实外观。

我知道如果我向 https://www.sharedividends.com.au/wp-content/custom/ajaxfile.php?code=MLT 发送带有相关负载的 post http 请求，我可以获取 json 中的所有表格字段，但这不是我想要解决的方法。

Website link

我试过了：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.sharedividends.com.au/mlt-dividend-history/"

driver = webdriver.Chrome()

driver.get(url)

table = driver.find_element_by_css_selector("#divTable")
driver.execute_script("arguments[0].scrollIntoView();",table)

for items in driver.find_elements_by_css_selector("td.sorting_1"):
    driver.execute_script("arguments[0].scrollIntoView();",items)
    items.click()

for elems in driver.find_elements_by_css_selector("#divTable tbody tr"):
    franking_credit = elems.find_elements_by_css_selector("td")[5].text
    gross_divident = elems.find_elements_by_css_selector("td")[6].text
    further_info = elems.find_elements_by_css_selector("td")[7].text
    print(franking_credit,gross_divident,further_info)

driver.quit()

当我运行上面的脚本时，它会抛出这个错误 IndexError: list index out of range 指向 franking_credit = 这一行。

这就是这张桌子的样子。我已在下图中的该表中标记了我感兴趣的三个字段。

Image link

如何解析该表中的三个字段？

Answer 1

您收到以下错误，因为当运行自动化脚本时，它显示 20 行和一些其他属性而不是 10 行。试试下面的代码。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.sharedividends.com.au/mlt-dividend-history/"

driver = webdriver.Chrome()

driver.get(url)

table = driver.find_element_by_css_selector("#divTable")
driver.execute_script("arguments[0].scrollIntoView();",table)

for items in driver.find_elements_by_css_selector("td.sorting_1"):
    driver.execute_script("arguments[0].scrollIntoView();",items)
    items.click()

for elems in driver.find_elements_by_css_selector("#divTable tbody tr[role='row']"):
    franking_credit = elems.find_elements_by_css_selector("td")[5].text
    gross_divident = elems.find_elements_by_css_selector("td")[6].get_attribute('textContent')
    further_info = elems.find_elements_by_css_selector("td")[7].get_attribute('textContent')
    print(franking_credit, gross_divident,further_info)

控制台输出：

$ 0.0446 $ 0.1486 10.4C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0107 $ 0.0357 2.5C FRANKED@30%; SP ECIAL; DRP SUSP

$ 0.0386 $ 0.1286 9C FRANKED @ 30%; DR P NIL DISCOUNT

$ 0.0437 $ 0.1457 10.2C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0377 $ 0.1257 8.8C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0429 $ 0.1429 10C FRANKED @ 30%; D RP NIL DISCOUNT

$ 0.0373 $ 0.1243 8.7C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0424 $ 0.1414 9.9C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0373 $ 0.1243 8.7C FRANKED @ 30%; DRP

$ 0.0441 $ 0.1471 10.3C FR@30%;0.4C SP ECIAL;DRP;NIL DIS

Answer 2

这应该可以解决问题！

from selenium import webdriver

driver = webdriver.Chrome('chromedriver/chromedriver.exe')

driver.get("https://www.sharedividends.com.au/mlt-dividend-history/")

for button in driver.find_elements_by_class_name("sorting_1"):
    button.click()

# Returns first part of the info
for item in driver.find_elements_by_xpath("//tr[@role='row']/td"):
    print(item.text)

# Returns second part of info
for a in driver.find_elements_by_xpath("//ul[@class='dtr-details']/li"):
        print(a.text)

输出； this

Answer 3

要从 Franking Credit、Gross Divident 和 Further Information 三个字段中提取数据，您必须诱导 WebDriverWait em> 对于visibility_of_all_elements_located()，您可以使用以下Locator Strategies：

• 代码块：

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  chrome_options = webdriver.ChromeOptions() 
  chrome_options.add_argument("start-maximized")
  chrome_options.add_argument('disable-infobars')
  driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.sharedividends.com.au/mlt-dividend-history/")
  driver.execute_script("arguments[0].scrollIntoView();", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table#divTable"))))
  for elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@aria-describedby='divTable_info']//tbody//tr/td[@class='sorting_1']"))):
      elem.click()
  all_fc = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@aria-describedby='divTable_info']//tbody//tr//td[position()=6]")))]
  all_gd = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@aria-describedby='divTable_info']//tbody//tr//td[position()=7]")))]
  all_fi = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@aria-describedby='divTable_info']//tbody//tr[@class='child']//li//span[@class='dtr-data']")))]
  for x,y,z in zip(all_fc, all_gd, all_fi):
      print(x,y,z)
  

• 控制台输出：

  $ 0.0446 $ 0.1486 10.4C FRANKED @ 30%; DRP NIL DISCOUNT
  
  $ 0.0107 $ 0.0357 2.5C FRANKED@30%; SP ECIAL; DRP SUSP
  
  $ 0.0386 $ 0.1286 9C FRANKED @ 30%; DR P NIL DISCOUNT
  
  $ 0.0437 $ 0.1457 10.2C FRANKED @ 30%; DRP NIL DISCOUNT
  
  $ 0.0377 $ 0.1257 8.8C FRANKED @ 30%; DRP NIL DISCOUNT
  
  $ 0.0429 $ 0.1429 10C FRANKED @ 30%; D RP NIL DISCOUNT
  
  $ 0.0373 $ 0.1243 8.7C FRANKED @ 30%; DRP NIL DISCOUNT
  
  $ 0.0424 $ 0.1414 9.9C FRANKED @ 30%; DRP NIL DISCOUNT
  
  $ 0.0373 $ 0.1243 8.7C FRANKED @ 30%; DRP
  
  $ 0.0441 $ 0.1471 10.3C FR@30%;0.4C SP ECIAL;DRP;NIL DIS