在 R 中使用 map() 将列表应用于网络抓取功能

Question

这是我的问题：我生成的这个列表包含大量链接，我想获取这个列表并对其应用一个函数来从所有这些链接中抓取一些数据；但是，当我运行程序时，它只从该元素的第一个链接获取数据，重新打印该信息以获得正确的迭代次数。到目前为止，这是我的所有代码：

library(tidyverse)
library(rvest)

source_link<-"http://www.ufcstats.com/statistics/fighters?char=a&page=all"
source_link_html<-read_html(source_link)

#This scrapes all the links for the pages of all the fighters
links_vector<-source_link_html%>%
  html_nodes("div ul li a")%>%
  html_attr("href")%>%
  #This seq selects the 26 needed links, i.e. from a-z
  .[1:26]

#Modifies the pulled data so the links become useable and contain all the fighers instead of just some
links_vector_modded<-str_c("http://www.ufcstats.com", links_vector,"&page=all")

fighter_links<-sapply(links_vector_modded, function(links_vector_modded){
  read_html(links_vector_modded[])%>%
  html_nodes("tr td a")%>%
  html_attr("href")%>%
  .[seq(1,length(.),3)]%>%
  na.omit(fighter_links)
})

###Next Portion: Using the above links to further harvest

#Take all the links within an element of fighter_links and run it through the function career_data to scrape all the statistics from said pages.
fighter_profiles_a<-map(fighter_links$`http://www.ufcstats.com/statistics/fighters?char=a&page=all`, function(career_data){
  #Below is where I believe my problem lies
  read_html()%>%
  html_nodes("div ul li")%>%
  html_text() 
})

我遇到的问题在代码的最后一部分，read_html()。我不知道如何将列表中元素中的每个链接应用于该函数。另外，有没有办法调用fighter_links 的所有元素，而不是一次调用一个元素？

感谢您的任何建议和帮助！

Answer 1

您可以unlist 将所有fighter_links 放在一起并将其传递给map 函数以提取相关文本。

library(rvest)
library(purrr)

fighter_profiles_a<-map(unlist(fighter_links), function(career_data){
  read_html(career_data)%>%
    html_nodes("div ul li")%>%
    html_text() 
})

fighter_profiles_a 捕获的文本可能需要额外清理。

Answer 2

挑战在于fighter_links 是一个向量列表。将map 应用于每个列表元素会留下一个 URL 向量，并且您希望从每个 URL 获取信息。

如果保留fighter_links 的结构很重要 - 意思是，您不会丢失属于每个战斗机的 URL - 您可以将调用嵌套到 map，如下所示：

fighter_profiles <- 
  fighter_links %>%
    map(function(url_list) {
      map(url_list,
           function(url) read_html("http://www.ufcstats.com/fighter-details/93fe7332d16c6ad9"[]) %>% 
             html_nodes("div ul li") %>% 
             html_text() %>%
             str_replace_all(., "\n\\s+\n\\s+", "")) # a little clean up
    })

这会产生嵌套输出，您可以使用它来跟踪它来自哪个fighter_links 条目：

[[1]]
[[1]][[1]]
 [1] "Height:--\n    "         "Weight:155 lbs.\n    "   "Reach:--\n    "          "STANCE:"                
 [5] "DOB:Jul 13, 1978"        "SLpM:0.00\n\n        "   "Str. Acc.:0%\n        "  "SApM:0.00\n        "    
 [9] "Str. Def:0%\n        "   ""                        "TD Avg.:0.00\n        "  "TD Acc.:0%\n        "   
[13] "TD Def.:0%\n        "    "Sub. Avg.:0.0\n        " "Events & Fights"         "Fighters"               
[17] "Stat Leaders"           

[[1]][[2]]
 [1] "Height:--\n    "         "Weight:155 lbs.\n    "   "Reach:--\n    "          "STANCE:"                
 [5] "DOB:Jul 13, 1978"        "SLpM:0.00\n\n        "   "Str. Acc.:0%\n        "  "SApM:0.00\n        "    
 [9] "Str. Def:0%\n        "   ""                        "TD Avg.:0.00\n        "  "TD Acc.:0%\n        "   
[13] "TD Def.:0%\n        "    "Sub. Avg.:0.0\n        " "Events & Fights"         "Fighters"               
[17] "Stat Leaders"           

注意：如果您愿意，也可以使用map 代替初始的sapply：

path <- "http://www.ufcstats.com/statistics/fighters"
query_str <- paste0("?char=", letters, "&page=all")
urls <- paste0(path, query_str)

get_fighter_link <- function(url) {
  read_html(url[])%>%
    html_nodes("tr td a")%>%
    html_attr("href")%>%
    .[seq(1, length(.), 3)]%>%
    na.omit(fighter_links)
}

fighter_links <- map(urls, get_fighter_link)