使用 R 跨多个链接进行网络抓取答案

【问题标题】：Web-scraping across multiple links using R使用 R 跨多个链接进行网络抓取
【发布时间】：2021-04-25 07:53:17
【问题描述】：

我正在尝试为多个网站创建一个包含一些新闻稿的整洁数据框。大多数网站的结构都是这样的，所以有一个带有简短简介的标题主页，然后是一个指向主要文章的链接。我想从主页上抓取所有主要文章。这是我的方法。任何帮助将不胜感激。

library(tidyverse)
library(rvest)
library(xml2)

url_1 <- read_html("http://lifepointhealth.net/news")


## seems to grab the lists
url_1 %>% 
  html_nodes("li") %>% 
  html_text() %>% 
  str_squish() %>% 
  str_trim() %>% 
  enframe()

# A tibble: 80 x 2
    name value                      
   <int> <chr>                      
 1     1 Who We Are Our Company Mis…
 2     2 Our Company Mission, Visio…
 3     3 Mission, Vision, Values an…
 4     4 Giving Quality a Voice     
 5     5 How We Operate             
 6     6 Leadership                 
 7     7 Awards                     
 8     8 20th Anniversary           
 9     9 Our Communities Explore Ou…
10    10 Explore Our Communities    
# … with 70 more rows


# this grabs the titles but there should be many more
url_1 %>% 
  html_nodes("li .title") %>% 
  html_text() %>% 
  str_squish() %>% 
  str_trim() %>% 
  enframe() 

# A tibble: 20 x 2
    name value                      
   <int> <chr>                      
 1     1 LifePoint Health Names Elm…
 2     2 David Steitz Named Chief E…
 3     3 LifePoint Health Receives …
 4     4 Thousands of Top U.S. Hosp…
 5     5 Conemaugh Nason Medical Ce…
 6     6 Vicki Parks Named CEO of W…
 7     7 LifePoint Health Honors Ka…
 8     8 Ennis Regional Medical Cen…
 9     9 LifePoint Health Business …
10    10 LifePoint Health and R1 RC…

【问题讨论】：

标签： html r web-scraping rvest

【解决方案1】：

点击开发工具的网络选项卡，您将看到每次单击“加载更多”时页面都会向http://lifepointhealth.net/api/posts发送请求。模仿下面的请求，你就可以把332个帖子的详细信息全部刮掉：

items <- httr::POST(
  "http://lifepointhealth.net/api/posts",
  config = httr::add_headers(`Content-Type` = "application/x-www-form-urlencoded"),
  body = "skip=0&take=332&Type=News&tagFilter=",
  encode = "multipart"
) %>% 
  httr::content() %>%
  .$Items

items <- dplyr::bind_rows(lapply(items, function(f) {
  as.data.frame(Filter(Negate(is.null), f))
}))

【讨论】：