使用 rvest 和 R 抓取 XHR 动态页面

Question

我正在尝试通过 XHR 请求抓取动态网站 Morningstar.com。

我正在抓取的确切网站是：http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY&region=USA&culture=en_US

我要抓取的是季度绩效数字（1 个月）。截至今天，结果应该是 0.64。

try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
                 query = list(
                   t="SPY",
                   region="usa",
                   culture="en-US"
                 )
  ))

  tryCatch(x <- content(res) %>%
             html_nodes(xpath = '//*[@id="tab-quar-end-content"]/table/tbody/tr[1]/td[1]') %>% 
             html_text() %>%
             trimws() %>% 
             as.numeric()
           , error = function(e) x <-NA)

但是，结果是 numeric(0)

知道我做错了什么吗？

苏迪

更新：

我能够使用以下代码获取 html 数据：

 try(res <- GET(url = "http://performance.morningstar.com/fund/performance-return.action",
                 query = list(

                   t = "SPY",
                   region = "usa",
                   culture = "en-US",
                   ops = "clear",
                   s = "0P0000J533",
                   ndec = "2",
                   ep = "true",
                   align = "q",
                   annlz = "true",
                   comparisonRemove = "false"

                 )
  ))

但我仍然无法使用 CSS 选择器或带有 rvest 的 xpath 来指向数据。

你们用什么来查找这些数据点？ SelectorGadget 仍然是首选吗？

干杯，亚伦

Answer 1

library(httr)

GET(
  url = "http://performance.morningstar.com/perform/Performance/cef/trailing-total-returns.action", 
  add_headers(
    Referer = "http://performance.morningstar.com/funds/etf/total-returns.action?t=SPY&region=USA&culture=en_US", 
    `X-Requested-With` = "XMLHttpRequest"
  ),
  query = list(
    t = "ARCX:SPY", region = "usa", culture = "en-US", 
    cur = "", ops = "clear", s = "0P00001MK8", ndec = "2", ep = "true", 
    align = "q", annlz = "true", comparisonRemove = "false", 
    benchmarkSecId = "", benchmarktype = ""
  ),
  verbose()
) -> res

您必须直接定位 XHR。

Answer 2

该表是使用 java 脚本嵌入的，而不是硬编码的。您将无法抓取这些数据。