R：网络从表中抓取多个变量

Question

我正在尝试使用以下代码在 R 中抓取网页 (https://aviation-safety.net/database/dblist.php?Year=1986&lang=&page=1)：

install.packages("rvest")
library(rvest)

aviationurl = "https://aviation-safety.net/database/dblist.php?Year=1986"
webpage = read_html(aviationurl)

# define variables of interest
variables = c("Date","Type","Registration","Operator","Fat","Location","Flag","Picture","Category")

# create sequence of numbers (as CSS for each variable contains a number)
colnums = seq(1,length(variables))

#  write commands for pulling each variable into an R dataframe and executing them
eval(parse(paste(variables," = as.data.frame(html_text(html_nodes(webpage,'td:nth-child(",colnums,")')))",sep="")))

# create final table with all variables
df = cbind(parse(variables))

但是，在 eval 命令之后，我收到以下错误消息：

文件中的错误（文件名，“r”）：无效的“描述”参数 另外：警告信息： 在 if (file == "") { ： 条件的长度 > 1 并且只使用第一个元素

如果我使用不带 eval(parse()) 的粘贴命令，并手动 C+P 生成字符串，它们可以正常工作。那么为什么 R 不能正确评估它们呢？

愿意接受其他建议，但想知道为什么 eval 不起作用，以及我是否可以做些什么来使这项工作正常进行。

谢谢！

乔什

Answer 1

我认为这可以解决您的问题。

library(rvest)

aviationurl = "https://aviation-safety.net/database/dblist.php? 
Year=1986"
webpage = read_html(aviationurl)

 table <- as.data.frame(html_table(html_nodes(webpage, "table"))) 

head(table)
         date                    type registration
1 03-JAN-1986           Antonov An-2T   CCCP-06101
2 13-JAN-1986        BN-2A-6 Islander       C-GTPB
3 15-JAN-1986      Dassault Falcon 10       F-GBTC
4 15-JAN-1986 Boeing 737-2A8 Advanced       VT-EGD
5 16-JAN-1986           Antonov An-2R       SP-WON
6 18-JAN-1986   SE-210 Caravelle VI-N       HC-BAE
                             operator fat.             location
1 Tselinny gorno-khimicheski kombinat    0        near Shantobe
2                Borealis Exploration    0      Caribou Horn...
3                              Air BG    2 near Vatry/Châlon...
4                     Indian Airlines    0      Tiruchirappa...
5                                 ZUA   NA              Un-Sara
6    SAETA, op.for Aerovias Guatemala   94 near Flores-Santa...
  Var.7 pic cat
1    NA  NA  A1
2    NA  NA  A2
3    NA  NA  A1
4    NA  NA  A2
5    NA  NA  A1
6    NA  NA  A1

其他方法

library(rvest)

 aviationurl = "https://aviation-safety.net/database/dblist.php? 
Year=1986"
 webpage = read_html(aviationurl)

 # define variables of interest
 variables =c("Date","Type","Registration","Operator","Fat","Location","Flag","Picture","Category")

# create sequence of numbers (as CSS for each variable contains a number)
colnums = seq(1,length(variables))

library(dplyr)
table <- list()
for(i in 1:length(colnums)){
  table[[i]] <- as.data.frame(html_text(html_nodes(webpage, paste0("td:nth-child(",colnums[i],")"))))
}
table <- bind_cols(table)
names(table) <- variables


head(table)
         Date                    Type Registration
1 03-JAN-1986           Antonov An-2T   CCCP-06101
2 13-JAN-1986        BN-2A-6 Islander       C-GTPB
3 15-JAN-1986      Dassault Falcon 10       F-GBTC
4 15-JAN-1986 Boeing 737-2A8 Advanced       VT-EGD
5 16-JAN-1986           Antonov An-2R       SP-WON
6 18-JAN-1986   SE-210 Caravelle VI-N       HC-BAE
                             Operator Fat             Location Flag
1 Tselinny gorno-khimicheski kombinat   0        near Shantobe     
2                Borealis Exploration   0      Caribou Horn...     
3                              Air BG   2 near Vatry/Châlon...     
4                     Indian Airlines   0      Tiruchirappa...     
5                                 ZUA                  Un-Sara     
6    SAETA, op.for Aerovias Guatemala  94 near Flores-Santa...     
  Picture Category
1               A1
2               A2
3               A1
4               A2
5               A1
6               A1

Answer 2

为了完整性：代码将网站上的三个页面与数据一起抓取，并将它们绑定在一个数据框中。

library(rvest)

url <- "https://aviation-safety.net/database/dblist.php?Year=1986&.  lang=&page="
urls <- paste0(url, 1:3)

scr <- function(url) {
  read_html(url) %>%
   html_nodes("table") %>%
   html_table() %>%
   as.data.frame()
}
df <- do.call(rbind, lapply(urls, scr))