在 Python 中使用 selenium 从具有多个页面的表中提取信息

Question

我正在尝试使用 Python 中的 Selenium 从网站中提取一些文章。但是，我很难做到这一点。

到目前为止，我的代码如下：

import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Open website
os.environ['PATH'] += r"C:\Users\BackUp HDL\AppData\Local\Programs"
driver = webdriver.Chrome()

driver.get("https://watchmedier.dk/latest/filteredsitesFilter=policywatch.dk&sitesFilter=shippingwatch.dk&sitesFilter=mobilitywatch.dk&sitesFilter=energiwatch.dk&sitesFilter=finanswatch.dk&sitesFilter=ejendomswatch.dk&sitesFilter=mediawatch.dk&sitesFilter=agriwatch.dk&sitesFilter=fodevarewatch.dk&sitesFilter=medwatch.dk&sitesFilter=kapwatch.dk&sitesFilter=itwatch.dk&sitesFilter=ctwatch.dk&sitesFilter=watchmedier.dk&sitesFilter=advokatwatch.dk") 
# Accept cookies
WebDriverWait(driver, 20).until(
EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@title='SP Consent 
Message']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, 
"//button[@title='Accepter']"))).click()

我现在想将下表的内容添加到文件中，即创建一个遍历所有页面的循环，并将来自“DATO”、“ARTIKLE”和“SIDE”的信息收集到一个数据框。我尝试按照一些不同的在线指南运行 for 循环，但我总是会遇到不同的错误（无法找到的元素、不可调用的对象等）。

谁能帮帮我，如何继续？

Answer 1

如果您有错误，那么您应该在问题中显示它们。

---

首先它需要回到parent_frame()。但如果您不接受 cookie，代码也可以工作。

您可以使用带有 "&pageNumber=..." 的 url 加载页面 - 如果您想使用 for-loop 仅加载几个页面，它会很有用。

要加载所有页面，您可以使用 onclick="nextPage()" 搜索 button 并单击它。最终你可以直接运行 JavaScript nextPage() 来加载下一页，但这样你不知道是否有下一页的按钮。

您可以找到表格中的所有行，并使用for-loop 单独处理每一行。您可以使用[1:] 跳过带有标题的第一行。

在每一行中，您可以使用相对 xpath（以点 . 开头）仅搜索该行中的单元格。

将所有值放在普通列表中然后将所有值转换为DataFrame会更简单。

import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Open website
#os.environ['PATH'] += r"C:\Users\BackUp HDL\AppData\Local\Programs"
#driver = webdriver.Chrome()
driver = webdriver.Firefox()

# --- accept (but it works also without acceptation) --

print('Load main page')

url = "https://watchmedier.dk/"
driver.get(url)

# Accept cookies
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@title='SP Consent Message']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@title='Accepter']"))).click()
driver.switch_to.parent_frame()

# --- before loops ---

url = "https://watchmedier.dk/latest/filtered?sitesFilter=policywatch.dk&sitesFilter=shippingwatch.dk&sitesFilter=mobilitywatch.dk&sitesFilter=energiwatch.dk&sitesFilter=finanswatch.dk&sitesFilter=ejendomswatch.dk&sitesFilter=mediawatch.dk&sitesFilter=agriwatch.dk&sitesFilter=fodevarewatch.dk&sitesFilter=medwatch.dk&sitesFilter=kapwatch.dk&sitesFilter=itwatch.dk&sitesFilter=ctwatch.dk&sitesFilter=watchmedier.dk&sitesFilter=advokatwatch.dk&startDate=&endDate="

# load first page - if you use `while`-loop
driver.get(url)

all_results = []

# --- loops ---

page = 1

while True:
#for page in range(1, 4):
    print('--- page:', page, '---')
    
    # load page - if you use `for`-loop
    #driver.get(url + "&pageNumber=" + str(page))
               
    # get all rows in table
    all_rows = driver.find_elements_by_xpath('//table[@class="c-latest-news__table"]//tr')
    
    # process all rows except first with headers
    for number, row in enumerate(all_rows[1:], 1):
        
        #print('row:', number)
        print('.', end='')
        
        # get all columns in row - relative xpath 
        all_cells = row.find_elements_by_xpath('.//td')
        
        date = all_cells[0].text
        
        artikel = all_cells[1].find_element_by_xpath('.//a')
        artikel_text = artikel.text
        artikel_link = artikel.get_attribute('href')
        
        # count `span` and convert to `boolean` (`bool(0)` gives `False`)
        key = bool(len(all_cells[2].find_elements_by_xpath('.//span')))
        
        side = all_cells[3].text
        
        # put row data in list
        all_results.append( [date, artikel_text, artikel_link, key, side] )
        
    print() 
    
    # find button to next page
    try:
        driver.find_element_by_xpath('//button[@onclick]').click()
        page += 1
    except Exception as ex:
        print('Exception:', ex)
        break
    
    #if page > 3: break  # stop `while`-loop after 3 pages

# --- after loops ---

import pandas as pd

# convert all data to DataFrame
df = pd.DataFrame(all_results, columns=['Dato', 'Artikel text', 'Artikel link', 'Key', 'Side'])

print(df.to_string())

结果：（当我使用if page > 3: break 3 页后停止。）

Load main page
--- page: 1 ---
..................................................
--- page: 2 ---
..................................................
--- page: 3 ---
..................................................

      Dato                                                                                                                 Artikel text                                                                Artikel link    Key           Side
0    03/12                                                                    Walgreens undersøger muligheder for salg af britisk enhed                  https://medwatch.dk/Medicinal___Biotek/article13527459.ece  False       MEDWATCH
1    03/12                                                          FDA udvider godkendelse af Eli Lillys antistofbehandling yderligere                  https://medwatch.dk/Medicinal___Biotek/article13527436.ece  False       MEDWATCH
2    03/12                                                                           Svensk regering opgiver modstand mod EU-mindsteløn                       https://policywatch.dk/nyheder/eu/article13527396.ece  False    POLICYWATCH
3    03/12                                                       Italien vil bruge 2,8 mia. euro til afskærmning for øgede energipriser     https://energiwatch.dk/Energinyt/Politik___Markeder/article13527349.ece  False    ENERGIWATCH
4    03/12                                                                   Ny sms sår tvivl om topchefs forklaring om slettede sms'er                    https://agriwatch.dk/Nyheder/politik/article13527312.ece   True      AGRIWATCH
5    03/12                                                                              USA: Statsansattes iPhones ramt af hackerangreb             https://itwatch.dk/ITNyt/Brancher/Sikkerhed/article13527166.ece  False        ITWATCH
6    03/12  Trods rekordkøb afviser A.P. Møller Holding investeringer i life science: "Et for langt skridt fra vores kompetencer i dag"            https://medwatch.dk/laboratorie___diagnostik/article13525353.ece   True       MEDWATCH
7    03/12                                                                                                    3i køber AMP ud af Esvagt                https://kapwatch.dk/nyheder/kapitalfonde/article13527079.ece   True       KAPWATCH
8    03/12                                                                                  Mette F. afventer proces om slettede sms'er                    https://agriwatch.dk/Nyheder/politik/article13527026.ece  False      AGRIWATCH
9    03/12                                             Finanstilsynet ser særlig og sjælden kontotype skabe ny debat om negative renter       https://finanswatch.dk/Finansnyt/Pengeinstitutter/article13526203.ece   True    FINANSWATCH
10   03/12                                                                             Vestas vælger tysk logistikpartner i Nordamerika             https://energiwatch.dk/Energinyt/Renewables/article13526999.ece   True    ENERGIWATCH