rvest 数据抓取用 NA 替换丢失的 html_node答案

【问题标题】：rvest data scraping replacing missing html_node with NArvest 数据抓取用 NA 替换丢失的 html_node
【发布时间】：2020-04-04 05:59:10
【问题描述】：

大家好！

很长一段时间以来，我一直在尝试抓取数据并以某种方式将丢失的 html_nodes 替换为 NA 或其他任何内容。然而，我一直没有成功。

谁能帮我弄清楚怎么做？或者在哪里查看以了解如何操作？

我目前的抓取代码如下：

library('rvest')
header_bind <- c()
page <- 0
price <- c()
ebay <- c()
runtime <- c()
pages <- 2
for (i in 1:pages) {


    page <- page + 1
    link <- paste("https://www.ebay.com/b/Cell-Phones-Smartphones/9355/bn_320094?LH_BIN=1&LH_ItemCondition=1000&rt=nc&_from=R40&_pgn=",page, sep="")
    webpage <- read_html(link)


    #read the name of the item
    header <- html_nodes(webpage, ".s-item__title")
    header_Text <- html_text(header)

      header_bind <- rbind(header_bind,as.data.frame(header_Text))

    #i get the price
    prim_html <- html_nodes(webpage, ".s-item__price")
    text_prim <- html_text(prim_html)

      price <- rbind(price,as.data.frame(text_prim))

    #i get the (amount sold this is missing sometimes)
    runtime_html <- html_nodes(webpage, ".NEGATIVE")
    text_runtime <- html_text(runtime_html)

      runtime <- rbind(runtime,as.data.frame(text_runtime))

    #prints 0 so i know that it went throught the for(){}
    print(0)
}

附注我知道它看起来很糟糕，但我每天都在学习如何更好地编码。

代码输出 48 obs 的价格和产品名称，但是，当涉及到已售或剩余的数量时，它给了我 43。

我试图通过查看类似的堆栈overflow帖子来了解其他人是如何做到的，但是，我不知何故未能掌握他们的想法。我的想法是，我可以将此函数用于缺少节点的元素，但它似乎不起作用：

text_runtime<- webpage %>% 
      html_nodes(".NEGATIVE") %>% 
      html_text() %>% 
      {if(length(.) == 0) NA else .}

这个函数也给了我 43 个元素，并且没有在节点丢失的地方放置任何 NA。

【问题讨论】：

标签： r rvest

【解决方案1】：

您应该首先使用html_nodes 提取.s-item__details，然后使用html_node（不带s）从每个节点提取.NEGATIVE 或.s-item__hotness。

library('rvest')

pages <- 1
page <- 0
output <- data.frame(header = character(), price = character(), runtime = character())
for (i in 1:pages) {

  page <- page + 1
  link <- paste("https://www.ebay.com/b/Cell-Phones-Smartphones/9355/bn_320094?LH_BIN=1&LH_ItemCondition=1000&rt=nc&_from=R40&_pgn=",page, sep="")

  webpage <- read_html(link)


  #read the name of the item
  header <- html_nodes(webpage, ".s-item__title")
  header_text <- html_text(header)

  #i get the price
  prim_html <- html_nodes(webpage, ".s-item__price")
  text_prim <- html_text(prim_html)

  price <- rbind(price,as.data.frame(text_prim))

  #i get the (amount sold this is missing sometimes)

  item <- html_nodes(webpage, ".s-item__details")
  runtime_html <- html_node(item, ".s-item__hotness")
  text_runtime <- html_text(runtime_html)
  text_runtime[is.na(text_runtime)] <- "0"

  # combine
  out <- data.frame(header_text, text_prim, text_runtime)
  output <- rbind(output, out)

  #prints 0 so i know that it went throught the for(){}
  print(0)

}

output

output

#                                                                                  header_text          text_prim text_runtime
# 1                      Google Nexus 5X H791 32GB (FACTORY UNLOCKED) 5.2" HD - Mint Green LG              $44.88      42 sold
# 2                 Motorola Moto Z3 Play 32GB - Unlocked - Deep Indigo - Brand New - XT1929-4            $177.02   7 watching
# 3                                 LG V20 -Brand New - H915 - Unlocked - Ships Express Canada            $162.16   5 watching
# 4           Samsung Galaxy J3 Unlocked 5" 16GB GSM 4G LTE Android Smartphone Black SM-J320W8             $77.89         0
# 5  New ListingSamsung Galaxy A30s SM-A307GN/DS Dual Sim (FACTORY UNLOCKED) 6.4" 64GB 4GB RAM            $212.43         0
# ...
# ...
# 42                Black phone 2 - 32GB - Black (Unlocked) Smartphone (Rest of World Version)            $318.65   5 watching
# 43                                                                               Sagem MC939            $199.00         0
# 44                                                                        Nokia 6220 classic            $199.00         0
# 45   New ListingSmart Mini Wireless HD Dual WiFi Pocket Projector 2G RAM 16G ROM Android 7.1            $353.35         0
# 46                                                                                nokia 7260            $149.25         0
# 47                                                                                smartphone            $250.00         0

【讨论】：