添加 'html_attr' 以使用 rvest 抓取多个 url答案

【问题标题】：Adding 'html_attr' to function scraping multiple urls using rvest添加 'html_attr' 以使用 rvest 抓取多个 url
【发布时间】：2020-08-07 00:14:13
【问题描述】：

我想使用下面的 scrape_change_page 函数将个人请愿书的链接添加到数据框，但我不确定如何调整以包含 html_attr 函数，该函数将从多个页面中抓取 url 链接。非常欢迎任何想法！

library(pacman)
pacman::p_load(rvest, dplyr, stringr, purrr, lubridate, tibble, tidyr, stringi, stringr)

url <- 'https://www.change.org/search?q=ppe'

scrape_change_page <- function(url)
{
  webpage  <- xml2::read_html(url)
  
  get_text <- function(css) 
  {
    vec <- rvest::html_text(rvest::html_nodes(webpage, css), trim = TRUE)
    if(length(vec) < 10) c(vec, rep("", 10 - length(vec))) else vec
  }
  
  dplyr::tibble(
    title         = get_text('.xs-mbs'),
    date          = gsub("Created", "", get_text('.symbol-clock+ span')),
    supporters    = gsub(" supporters", "", get_text('.symbol-supporters+ span')),
    addressee     = gsub("Petition to ", "", get_text('.xs-mbn .type-s')),
    location      = get_text('.plxxs'),
    )
}

scrape_change_page(url)

#select number of pages(test on 3 pages)
n_pages <- 3
urls    <- paste0(url, "&offset=", 10 * (seq(n_pages)) - 1)
result  <- do.call(rbind, lapply(urls, scrape_change_page))

这是抓取链接的基本 html_attr：

page <- read_html("https://www.change.org/search?q=ppe")
page %>%
  html_nodes(".search-results .list-rule") %>%
  html_nodes("a") %>% 
  html_attr("href")

【问题讨论】：

标签： r rvest

【解决方案1】：

通过更新函数来解决这个问题。下面更新了代码。

scrape_change_page <- function(url)
    {
      webpage  <- xml2::read_html(url)
      
      get_text <- function(css) 
      {
        vec <- rvest::html_text(rvest::html_nodes(webpage, css), trim = TRUE)
        if(length(vec) < 10) c(vec, rep("", 10 - length(vec))) else vec
      }
    
      get_attr <- function(css, attr)
      {
        vec <- rvest::html_attr(rvest::html_nodes(webpage, css), attr)
        if(length(vec) < 10) c(vec, rep("", 10 - length(vec))) else vec
      }
      
      dplyr::tibble(
        title         = get_text('.xs-mbs'),
        date          = gsub("Created", "", get_text('.symbol-clock+ span')),
        supporters    = gsub(" supporters", "", get_text('.symbol-supporters+ span')),
        addressee     = gsub("Petition to ", "", get_text('.xs-mbn .type-s')),
        location      = get_text('.plxxs'),
        link          = get_attr('.search-results .list-rule a.link-block.js-click-search-result', 'href')
        )
    }

【讨论】：