嵌套的 For 循环无法存储上一次迭代的数据

Question

我实际上是网络抓取的新手，昨晚才了解它。

简介：

我正在登录我的帐户时尝试抓取 Science Direct 网页。

我正在尝试存储每次迭代中的所有标题（有三页，即三个迭代），并且对于每次迭代，我必须进行爬网我做了另一个 for 循环来读取每个标题的 25 个唯一 ID在每次迭代中。

但是，它只存储了上一次迭代（第 3 页）的标题。

当我只进行单页抓取时，我知道代码正在工作，但是当我尝试使用第一个 for 循环抓取“下一页”时：

'''
for (i in seq (from = 0, to = 50, by = 25)) {

'''

正如我之前所说，代码只存储了最后一次迭代（即包含 25 篇文章的第 3 页）。

顺便说一下，每个页面都包含一个选项，每页显示几篇文章，可以是 25、50 或 100 篇文章，我选择了 25，因此序列 = 25。

代码如下：

#install.packages("xml2") # required for rvest
library("rvest") # for web scraping
library("dplyr") # for data management

titleNo = c()
name = list()
for(i in seq(from = 0, to = 50, by = 25)) {
  link = paste0("https://www.sciencedirect.com/search?qs=PISA%2C%20Programme%20for%20International%20Student%20Assessment&date=2010-2021&articleTypes=FLA&lastSelectedFacet=subjectAreas&subjectAreas=3300%2C3200%2C2800%2C2000%2C1200%2C1700%2C1400%2C1800%2C2200&offset=",i,"")
  for(j in 1:26) {
    page = read_html(link)
    titleNo[j] = (paste0(".push-m:nth-child(",j,") h2"))
    name[j] <- list(page %>% html_nodes(titleNo[j])%>% html_text())
  }
  print(paste(i))
}

name <- data.frame(unlist(name))

你们能指出我做错了什么吗？

代码成功地遍历所有页面，但是我的问题是，对于每次迭代，代码都会清除 name 变量并存储新变量，直到最后一次迭代。

我认为我的问题在于我的 for 循环。我不确定我做的是否正确。

谢谢

Answer 1

我认为你过于复杂了。您可以使用适当的 css 选择器一次性提取 25 个标题。

然后您可以unlist 将结果作为一个组合向量来获取。

library(rvest)

values <- seq (from = 25, to = 50, by = 25)
link <- paste0("https://www.sciencedirect.com/search?qs=PISA%2C%20Programme%20for%20International%20Student%20Assessment&date=2010-2021&articleTypes=FLA&lastSelectedFacet=subjectAreas&subjectAreas=3300%2C3200%2C2800%2C2000%2C1200%2C1700%2C1400%2C1800%2C2200&offset=", values)

result <- lapply(link, function(x) x %>%
      read_html() %>%
      html_nodes('div.result-item-content h2 span a') %>%
      html_text())
titles <- unlist(result)
titles

 #[1] "Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving"                                                 
 #[2] "The Comparison between Successful and Unsuccessful Countries in PISA, 2009"                                                                                                                   
 #[3] "Educational Data Mining: Identification of factors associated with school effectiveness in PISA assessment"                                                                                   
 #[4] "Curriculum standardization, stratification, and students’ STEM-related occupational expectations: Evidence from PISA 2006"                                                                    
 #[5] "Testing measurement invariance of PISA 2015 mathematics, science, and ICT scales using the alignment method"                                                                                  
 #[6] "Effects of students’ and schools’ characteristics on mathematics achievement: findings from PISA 2006"
#...
#...