无法通过 Selenium 从网页获取所有必要的链接

Question

我目前正在尝试在执行专利检索任务时使用一些自动化。我想获取与搜索查询结果对应的所有链接。特别是，我对从 2015 年开始的 Apple 专利感兴趣。所以代码是下一个 -

import selenium
from selenium import webdriver
from selenium.webdriver.firefox.options import Options as options
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.by import By

new_driver_path = r"C:/Users/alexe/Desktop/Apple/PatentSearch/geckodriver-v0.30.0-win64/geckodriver.exe"

ops = options()
serv = Service(new_driver_path)
browser1 = selenium.webdriver.Firefox(service=serv, options=ops)
browser1.get("https://patents.google.com/?assignee=apple&after=priority:20150101&sort=new")

elements = browser1.find_elements(By.CLASS_NAME, "search-result-item")

links = []
for elem in elements:
    href = elem.get_attribute('href')
    if href:
        links.append(href)

links = set(links)
for href in links:
    print(href)

而输出是下一个-

https://patentimages.storage.googleapis.com/ed/06/50/67e30960a7f68d/JP2021152951A.pdf
https://patentimages.storage.googleapis.com/86/30/47/7bc39ddf0e1ea7/KR20210106968A.pdf
https://patentimages.storage.googleapis.com/ca/2a/bc/9380e1657c2767/US20210318798A1.pdf
https://patentimages.storage.googleapis.com/c1/1a/c6/024f785fd5ea10/AU2021204695A1.pdf
https://patentimages.storage.googleapis.com/b3/19/cc/8dc1fae714194f/US20210312694A1.pdf
https://patentimages.storage.googleapis.com/e6/16/c0/292a198e6f1197/AU2021218193A1.pdf
https://patentimages.storage.googleapis.com/3e/77/e0/b59cf47c2b30a1/AU2021212005A1.pdf
https://patentimages.storage.googleapis.com/1b/3d/c2/ad77a8c9724fbc/AU2021204422A1.pdf
https://patentimages.storage.googleapis.com/ad/bc/0f/d1fcc65e53963e/US20210314041A1.pdf

这里的问题是我缺少 1 个链接 -

result item and the missing link

所以我尝试了不同的选择器，但仍然得到相同的结果 - 缺少一个链接。我也尝试使用不同的参数进行搜索，模式是下一个 - 所有丢失的链接都没有与 pdf 输出链接。我花了很多时间试图弄清楚是什么原因，所以如果你能提供关于此事的任何线索，我将不胜感激。提前致谢！

Answer 1

突出显示的选项没有带有class pdflink 的a 标签。将提取链接的代码行放在try 块中。如果未找到所需的元素，请搜索该文章可用的 a 标签。

尝试如下一次：

driver.get("https://patents.google.com/?assignee=apple&after=priority:20150101&sort=new")

articles = driver.find_elements_by_tag_name("article")

print(len(articles))

for article in articles:
    try:
        link = article.find_element_by_xpath(".//a[contains(@class,'pdfLink')]").get_attribute("href") # Use a dot in the xpath to find an element with in an element.
        print(link)
    except:
        print("Exception")
        link = article.find_element_by_xpath(".//a").get_attribute("href")
        print(link)

10
https://patentimages.storage.googleapis.com/86/30/47/7bc39ddf0e1ea7/KR20210106968A.pdf
https://patentimages.storage.googleapis.com/e6/16/c0/292a198e6f1197/AU2021218193A1.pdf
https://patentimages.storage.googleapis.com/3e/77/e0/b59cf47c2b30a1/AU2021212005A1.pdf
https://patentimages.storage.googleapis.com/c1/1a/c6/024f785fd5ea10/AU2021204695A1.pdf
https://patentimages.storage.googleapis.com/1b/3d/c2/ad77a8c9724fbc/AU2021204422A1.pdf
https://patentimages.storage.googleapis.com/ca/2a/bc/9380e1657c2767/US20210318798A1.pdf
Exception
https://patents.google.com/?assignee=apple&after=priority:20150101&sort=new#
https://patentimages.storage.googleapis.com/b3/19/cc/8dc1fae714194f/US20210312694A1.pdf
https://patentimages.storage.googleapis.com/ed/06/50/67e30960a7f68d/JP2021152951A.pdf
https://patentimages.storage.googleapis.com/ad/bc/0f/d1fcc65e53963e/US20210314041A1.pdf

Answer 2

要使用Selenium 和python 提取pdfs 的所有href 属性，您必须为visibility_of_all_elements_located() 诱导WebDriverWait，您可以使用以下任一@987654324 @：

• 使用CSS_SELECTOR：

  print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.search-result-item[href]")))])
  

• 使用XPATH：

  print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[contains(@class, 'search-result-item') and @href]")))])
  

• 控制台输出：

  ['https://patentimages.storage.googleapis.com/86/30/47/7bc39ddf0e1ea7/KR20210106968A.pdf', 'https://patentimages.storage.googleapis.com/e6/16/c0/292a198e6f1197/AU2021218193A1.pdf', 'https://patentimages.storage.googleapis.com/3e/77/e0/b59cf47c2b30a1/AU2021212005A1.pdf', 'https://patentimages.storage.googleapis.com/c1/1a/c6/024f785fd5ea10/AU2021204695A1.pdf', 'https://patentimages.storage.googleapis.com/1b/3d/c2/ad77a8c9724fbc/AU2021204422A1.pdf', 'https://patentimages.storage.googleapis.com/ca/2a/bc/9380e1657c2767/US20210318798A1.pdf', 'https://patentimages.storage.googleapis.com/b3/19/cc/8dc1fae714194f/US20210312694A1.pdf', 'https://patentimages.storage.googleapis.com/ed/06/50/67e30960a7f68d/JP2021152951A.pdf', 'https://patentimages.storage.googleapis.com/ad/bc/0f/d1fcc65e53963e/US20210314041A1.pdf']
  

• 注意：您必须添加以下导入：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

PS：您只能提取九 (9) 个 href 属性，因为其中一个搜索项是 <span> 元素并且不是链接，即没有 href 属性