selenium 使用元素的子元素抓取数据

Question

您好，我正在尝试从实时股票网站上抓取一些数据。我想显示公司名称和股票价格，%change 等。每页显示 25 家公司的详细信息，这些详细信息遵循相同的格式，一行数据位于标签下方，具有相同的 css 选择器，然后是其余元素，例如公司股价名称，%change 等是该标签的子标签。由于所有数据都遵循相同的格式，我的问题是如何过滤显示该行数据的每个标签？

我想过：

使用 for 循环按类名查找元素（因为它们都具有相同的类），然后循环遍历这些元素，但我不知道如何选择该类的子元素：

这里是 html 代码：

这是网站的链接 https://uk.finance.yahoo.com/most-active

Answer 1

来吧，伙计！我在尝试从类似的股票网站抓取数据时遇到了类似的问题

from bs4 import BeautifulSoup
from urllib.request import urlopen as uReq

url = "https://uk.finance.yahoo.com/most-active"
uClient = uReq(url)
html_page = uClient.read()
uClient.close()

soup = BeautifulSoup(html_page, 'html.parser')
full_table = soup.find_all("tr", class_="simpTblRow")

for tr in full_table:
    td = tr.find_all('td')
    row = [i.text for i in td]
    print(row)

您可以创建自己的数据结构来保存每个打印的值，它与此输出一起运行（没有标签，输出看起来令人困惑，但您可以查看该站点以供参考）

['ICON.L', 'Iconic Labs Plc', '0.0132', '0.0000', '0.00%', '1.291B', '412.755M', 
'1.299M', 'N/A', '']
['SYME.L', 'Supply@ME Capital plc', '0.8020', '+0.2320', '+40.70%', '1.145B', 
'173.333M', '262.694M', 'N/A', '']
['UJO.L', 'Union Jack Oil plc', '0.2925', '-0.0075', '-2.50%', '210.666M', '154.613M', 
'45.165M', 'N/A', '']
['TOM.L', 'TomCo Energy Plc', '0.8750', '+0.0750', '+9.37%', '197.389M', '49.702M', 
'5.894M', 'N/A', '']
['PREM.L', 'Premier African Minerals Limited', '0.0680', '-0.0045', '-6.21%', 
'193.547M', '136.558M', '8.542M', 'N/A', '']
['COPL.L', 'Canadian Overseas Petroleum Limited', '0.4000', '-0.0200', '-4.76%', 
'139.094M', '211.933M', '9.549M', 'N/A', '']
['MSMN.L', 'Mosman Oil And Gas Limited', '0.1800', '+0.0200', '+12.50%', '136.04M', 
'51.481M', '2.854M', 'N/A', '']
['VAST.L', 'Vast Resources plc', '0.2100', '-0.0090', '-4.11%', '133.599M', 
'243.076M', '25.673M', 'N/A', '']
['UKOG.L', 'UK Oil & Gas Investments PLC', '0.2250', '+0.0150', '+7.14%', '114.909M', 
'275.501M', '24.974M', 'N/A', '']
['7DIG.L', '7digital Group plc', '2.8000', '-0.3000', '-9.68%', '111.529M', '23.568M', 
'68.752M', 'N/A', '']
['ALBA.L', 'Alba Mineral Resources PLC', '0.0925', '-0.0125', '-11.90%', '99.299M', 
'39.579M', '4.472M', 'N/A', '']
['LLOY.L', 'Lloyds Banking Group plc', '28.18', '-0.44', '-1.54%', '98.424M', 
'263.235M', '19.948B', '70.46', '']
['KOD.L', 'Kodal Minerals Plc', '0.0475', '0.0000', '0.00%', '111.583M', '75.563M', 
'5.278M', 'N/A', '']
['UFO.L', 'Alien Metals Limited', '0.3600', '+0.0500', '+16.13%', '84.677M', 
'52.319M', '8.516M', 'N/A', '']
['EDL.L', 'Edenville Energy Plc', '0.0500', '-0.0025', '-4.76%', '80.171M', '47.879M', 
'4.073M', 'N/A', '']
['EQT.L', 'EQTEC plc', '0.5300', '+0.0650', '+13.98%', '73.296M', '144.02M', 
'36.929M', 'N/A', '']
['BOIL.L', 'Baron Oil Plc', '0.1000', '-0.0025', '-2.44%', '73.45M', '71.853M', 
'4.426M', 'N/A', '']
['BREE.L', 'Breedon Group plc', '83.20', '+2.20', '+2.72%', '63.05M', '4.616M', 
'1.403B', '43.79', '']
['AMGO.L', 'Amigo Holdings PLC', '9.30', '+0.55', '+6.29%', '62.892M', '20.841M', 
'44.206M', 'N/A', '']
['INSP.L', 'Inspirit Energy Holdings Plc', '0.0800', '+0.0025', '+3.23%', '61.622M', 
'73.243M', '2.323M', 'N/A', '']
['TRAF.L', 'Trafalgar Property Group plc', '0.2000', '0.0000', '0.00%', '55.959M', 
'40.707M', '2.85M', 'N/A', '']
['RBD.L', 'Reabold Resources plc', '0.7300', '-0.0350', '-4.58%', '51.697M', 
'23.499M', '51.808M', 'N/A', '']
['CPI.L', 'Capita plc', '28.70', '-7.18', '-20.01%', '48.494M', '12.711M', '478.994M', 
'N/A', '']
['GGP.L', 'Greatland Gold plc', '12.20', '-0.80', '-6.15%', '44.829M', '33.176M', 
'461.41M', 'N/A', '']
['SRES.L', 'Sunrise Resources plc', '0.3000', '-0.0100', '-3.23%', '39.45M', 
'15.895M', '9.963M', 'N/A', '']

Answer 2

table =[]
#Get the total number of rows for your table on page
tableRows = driver.find_elements_by_xpath("//div[@id='scr-res-table']//tr")
for i in range (2, len(tableRows)+1): #Starting from 2 as 1st row is column caption
    rowList  = []
    #Title scrapping is different from others as it is under tag a under td
    titlexPath = "//div[@id='scr-res-table']//tr["+str(i)+"]//td[1]/a"
    rowList.append(driver.find_element_by_xpath(titlexPath).text) 

    #Assuming you need data from column Name to PE ratio.
    for j in range(2,10):
        dataXpath = "//div[@id='scr-res-table']//tr["+str(i)+"]//td["+str(j)+"]"
        rowList.append(driver.find_element_by_xpath(dataXpath).text)

    table.append(rowList)
print(table)

注意：我使用 List 来存储网页表格中的数据。您可以使用 Panda 库来清洁和轻松地存储和操作数据