RSelenium 删除雅虎财经新闻头条答案

【问题标题】：RSelenium To Scrape Yahoo Financial News HeadlinesRSelenium 删除雅虎财经新闻头条
【发布时间】：2021-10-27 09:00:33
【问题描述】：

我想从雅虎获取一家公司的新闻标题。我使用RSelenium 启动远程浏览器并接受cookie。我找到了环绕的 css 类“StretchedBox”，我可以通过浏览器检查看到标题。如何存储这些标题？接下来，我想使用RSelenium 向下滚动并保存更多这些元素（比如几天）。

library('RSelenium')

# Start Remote Browser
rD <- rsDriver(port = 4840L, browser = c("firefox")) 
remDr <- rD[["client"]]

# Navigate to Yahoo Finance News for Specific Company
# This takes unusual long time 
remDr$navigate("https://finance.yahoo.com/quote/AAPL/news?p=AAPL")

# Get "accept all cookies" botton
webElems <- remDr$findElements(using = "xpath", "//button[starts-with(@class, 'btn primary')]") 

# We can check if we did get the proper button by checking the text of the element:
unlist(lapply(webElems, function(x) {x$getElementText()}))

# We found the two button, and we want to click the first one:
webElems[[1]]$clickElement()

# wait for page loading
Sys.sleep(5) 

# I am looking for news headline in or after the StretchedBox
boxes <- remDr$findElements(using = "class", "StretchedBox")
boxes[1] # empty

boxes[[1]]$browserName

【问题讨论】：

标签： web-scraping xpath css-selectors rselenium

【解决方案1】：

最后，我找到了一个 xpath，我可以从中 getElementText 新闻文章的标题。

library('RSelenium')

# Start Browser
rD <- rsDriver(port = 4835L, browser = c("firefox")) 
remDr <- rD[["client"]]

# Navigate to Yahoo Financial News
remDr$navigate("https://finance.yahoo.com/quote/AAPL/news?p=AAPL")

# Click Accept Cookies
webElems <- remDr$findElements(using = "xpath", "//button[starts-with(@class, 'btn primary')]")
unlist(lapply(webElems, function(x) {x$getElementText()}))
webElems[[1]]$clickElement()

# extract headlines from html/css by xpath 
headlines <- remDr$findElements(using = "xpath", "//h3[@class = 'Mb(5px)']//a")

# extract headline text
headlines <- sapply(headlines, function(x){x$getElementText()})
headlines[1]

[[1]]
[1] "What Kind Of Investors Own Most Of Apple Inc. (NASDAQ:AAPL)?"

【讨论】：