使用 phantomjs 和 rvest 抓取网页答案

【问题标题】：Scrape webpage with phantomjs and rvest使用 phantomjs 和 rvest 抓取网页
【发布时间】：2016-02-17 01:52:50
【问题描述】：

我正在尝试抓取以下网页： https://www.occ.com.mx/empleos-en-nuevo-leon 使用 rvest 和 selectorgadget，看起来很简单。

但是，它似乎是用 javascript 构建的，所以我按照this 教程，安装了phantomjs并尝试使用以下脚本在本地构建 html 网页。

// scrape_occ.js

var webPage = require('webpage');
var page = webPage.create();

var fs = require('fs');
var path = 'occ.html'

page.open('https://www.occ.com.mx/empleos-en-nuevo-leon', function (status) {
  var content = page.content;
  fs.write(path,content,'w')
  phantom.exit();
});

然后调用 R:system("phantomjs occ.js") 在本地构建 html，但会产生与原始 rvest 尝试类似的结果。我得到了更多节点，但似乎有一个表单在路上......

p <- read_html("occ.html")
p %>% html_structure()
<html>
  <head>
    {text}
    <meta [http-equiv, content]>
    {text}
    <meta [http-equiv, content]>
    {text}
    <meta [http-equiv, content]>
    {text}
    <noscript>
      {text}
    {text}
  <body>
    <apm_do_not_touch>
      {text}
      <script [language]>
        {cdata}
      {text}
    {text}
    <form [action, method]>
      <input [type, name, value]>
      <input [type, name, value]>
      <input [type, name, value]>
      <input [type, name, value]>
      <input [type, name, value]>
      <input [type, name, value]>
      <input [type, name, value]>
      <input [type, name, value]>

这样做的正确方法是什么？我错过了什么？

【问题讨论】：

标签： javascript html r web-scraping rvest

【解决方案1】：

RSelenium（对我来说是最后的选择，因为它正在编写浏览器脚本）在这里可以提供帮助：

library(RSelenium)
library(rvest)
library(dplyr)

checkForServer()
startServer()
remDr <- remoteDriver()
remDr$open()
remDr$navigate("https://www.occ.com.mx/empleos-en-nuevo-leon")
pg <- remDr$getPageSource()
remDr$close()

doc <- read_html(pg[[1]])

trimws(html_text(html_nodes(doc, "div.lineamod_sr")))

##  [1] "Feb 16  \n                Coordinador de Operaciones (Zona Cienega de Flores)"      
##  [2] "Feb 16  \n                Coordinador de Operaciones (Zona Apodaca)"                
##  [3] "Feb 16  \n                Ventas de Tecnologias de informacion (zona Monterrey)"    
##  [4] "Feb 16  \n                Practicante en Capacitación"                              
##  [5] "Feb 16  \n                Home Office: Reclutador Bilingüe"                         
##  [6] "Feb 16  \n                SISTEMAS"                                                 
##  [7] "Feb 16  \n                Asociado de RH"                                           
##  [8] "Feb 16  \n                Practicante Import Export"                                
##  [9] "Feb 16  \n                Comprador de refacciones"                                 
## [10] "Feb 16  \n                Maestra Auxiliar de Preescolar"

在阅读了 Eduardo 的评论后，一时兴起，我在检查网页并使用“复制为 cURL”（请参阅 Chrome 开发人员工具）获取主页资源后使用 rvest 尝试了此操作。我使用了我的 devtools pkg curlconverter (devtools::install_github("hrbrmstr/curlconverter"))

library(httr)
library(rvest)
library(curlconverter)

req <- make_req(straighten("curl 'https://www.occ.com.mx/empleos-en-nuevo-leon' -H 'Pragma: no-cache' -H 'Origin: https://www.occ.com.mx' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.39 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: no-cache' -H 'Referer: https://www.occ.com.mx/empleos-en-nuevo-leon' -H 'Cookie: cookie_home3=240784394.20480.0000; TS01650164_28=011b64c78f24f98a905a1610d142f8f5e8e6de557a270318cd246d8bdad634ad9d692ef9a7e7c5f5ee6125dfa6d154a7ecc9912326; q=; cat=; fn=; salary=; tm=60; loc=; page=1; TS01650164_77=08cb11f491ab28002f9f692817ac18f609a6564679dd056ae514dd5ed8f04683e016c84db8ad4aff81249c3ffa6f4d72086a11c1a482380047cb7d1b1a33ef2b110ce9bf57566934a0ba894dd447b3cf48fdfb5b4e17eb905895ebb597a5e878746dd0fba0f4fbb336e2f855820d002a; JobSeekerInfo=5877fd484ea1405b8e4780c2c0b05365|; searchtype=1; loctemp=MX-NL; fntemp=MX-NL; cattemp=; qtemp=; salarytemp=; tmtemp=; jobids=8687636,8687651,8689022,8688254,8688719,8688496,8687786,8687676,8688424,8654486,8688354,8687735,8687291,8688736,8687917,8688614,8687428,8688586,8688659,8687483,8688163,8654471,8687688,8688932,8689017,8642398,8687594,8688554,8688940,8687254,8688285,8415967,8687500,8688224,8687619,8688640,8688757,8689026,8688796,8702349,8728708,8728866,8728860,8721116,8591064,8729151,8729124,8729136,8729485,8551881; returnurl=/empleos-en-nuevo-leon; Culture=es-MX; TS01650164=01b518c62e8b19f2612cfbd7378cc0b73784ab414f17fdcc566da5c243d4a31e6ef629c80498c411d543fa42cb92729a648fc738579c3333ce81ff4708684a7e9f18f32db913b5b26edaa3bb4a0ad4d5ee740b16ab26bce774e39df018e046ddcf1c6529ce4b2194e780f9f039d729db6d958e850ead9570fc4748aa65cd7f85c0b79744cbf998c055a8314887942f194106c71e4cc2f736e495d21f228e9ceabfd36dc194b5ee20d31a00bc2b6469c8645ed502656b60a0a9b38aaf38a3435488f6e942917750d71f27386a87f3ff25f3bd1d163c0e6bd60212d36a3056fbdd4e055dab8c; sitemgrOCCM=false; sitemgrRUE=false; loaded=1' -H 'Connection: keep-alive' -H 'DNT: 1' --data 'TS01650164_id=3&TS01650164_cr=08cb11f491ab2800f5f07209cf7558bf5a68e875ec0c3a7ca8beb59ff0f5da214e8ca0f0b0d3925c3466838922d655d208203848fa894800ea2891c392679d0ca17d55a2a828b5e4bbe97f5e3310800fb216c3a9821974a3aa20fa64d8feec278da6e1000bf2a8ac7f2533a8b0e5f5424e20ea558b261d881d570b85067b4859&TS01650164_76=0&TS01650164_86=0&TS01650164_md=1&TS01650164_rf=0&TS01650164_ct=0&TS01650164_pd=0' --compressed"))[[1]]
resp <- req()
doc <- content(resp, as="parsed")
trimws(html_text(html_nodes(doc, "div.lineamod_sr")))

而且，它返回相同的结果。

curlconverter 的东西实际上做了一个这样的函数：

httr::VERB(
  verb = "POST", 
  url = "https://www.occ.com.mx/empleos-en-nuevo-leon", 
  add_headers(Pragma = "no-cache", 
              Origin = "https://www.occ.com.mx", 
              `Accept-Encoding` = "gzip, deflate", 
              `Accept-Language` = "en-US,en;q=0.8", 
              `Upgrade-Insecure-Requests` = "1", 
              `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.39 Safari/537.36", 
              Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 
              `Cache-Control` = "no-cache", 
              Referer = "https://www.occ.com.mx/empleos-en-nuevo-leon", 
              Connection = "keep-alive", 
              DNT = "1"), 
  set_cookies(cookie_home3 = "240784394.20480.0000", 
              TS01650164_28 = "011b64c78f24f98a905a1610d142f8f5e8e6de557a270318cd246d8bdad634ad9d692ef9a7e7c5f5ee6125dfa6d154a7ecc9912326", 
              q = "", cat = "", fn = "", 
              salary = "", tm = "60", 
              loc = "", page = "1", TS01650164_77 = "08cb11f491ab28002f9f692817ac18f609a6564679dd056ae514dd5ed8f04683e016c84db8ad4aff81249c3ffa6f4d72086a11c1a482380047cb7d1b1a33ef2b110ce9bf57566934a0ba894dd447b3cf48fdfb5b4e17eb905895ebb597a5e878746dd0fba0f4fbb336e2f855820d002a", 
              JobSeekerInfo = "5877fd484ea1405b8e4780c2c0b05365|", 
              searchtype = "1", loctemp = "MX-NL", 
              fntemp = "MX-NL", cattemp = "", 
              qtemp = "", salarytemp = "", 
              tmtemp = "", jobids = "8687636,8687651,8689022,8688254,8688719,8688496,8687786,8687676,8688424,8654486,8688354,8687735,8687291,8688736,8687917,8688614,8687428,8688586,8688659,8687483,8688163,8654471,8687688,8688932,8689017,8642398,8687594,8688554,8688940,8687254,8688285,8415967,8687500,8688224,8687619,8688640,8688757,8689026,8688796,8702349,8728708,8728866,8728860,8721116,8591064,8729151,8729124,8729136,8729485,8551881", 
              returnurl = "/empleos-en-nuevo-leon", 
              Culture = "es-MX", TS01650164 = "01b518c62e8b19f2612cfbd7378cc0b73784ab414f17fdcc566da5c243d4a31e6ef629c80498c411d543fa42cb92729a648fc738579c3333ce81ff4708684a7e9f18f32db913b5b26edaa3bb4a0ad4d5ee740b16ab26bce774e39df018e046ddcf1c6529ce4b2194e780f9f039d729db6d958e850ead9570fc4748aa65cd7f85c0b79744cbf998c055a8314887942f194106c71e4cc2f736e495d21f228e9ceabfd36dc194b5ee20d31a00bc2b6469c8645ed502656b60a0a9b38aaf38a3435488f6e942917750d71f27386a87f3ff25f3bd1d163c0e6bd60212d36a3056fbdd4e055dab8c", 
              sitemgrOCCM = "false", 
              sitemgrRUE = "false", loaded = "1"), 
  body = list(TS01650164_id = "3", 
              TS01650164_cr = "08cb11f491ab2800f5f07209cf7558bf5a68e875ec0c3a7ca8beb59ff0f5da214e8ca0f0b0d3925c3466838922d655d208203848fa894800ea2891c392679d0ca17d55a2a828b5e4bbe97f5e3310800fb216c3a9821974a3aa20fa64d8feec278da6e1000bf2a8ac7f2533a8b0e5f5424e20ea558b261d881d570b85067b4859", 
              TS01650164_76 = "0", TS01650164_86 = "0", 
              TS01650164_md = "1", TS01650164_rf = "0", 
              TS01650164_ct = "0", TS01650164_pd = "0"), 
  encode = "form")

问题在于它使用那些与时间相关的 cookie 设置发出 POST 请求。因此，如果在从开发人员工具会话中提取 Copy as cURL 命令后相对较快地完成，这将起作用，但我不确定之后多长时间。

【讨论】：

谢谢！尽管如此，这仍然会引发此错误：github.com/ropensci/RSelenium/issues/54（我正在尝试调试）
:-) 这将是我对 Selenium 的第二个问题。让它始终如一地跨平台工作。我仍然无法在 El Capitan 上使用 phantomjson。
我刚刚更新了它，显示普通的 ol rvest 似乎有效。什么没有吸引你？
我昨天浪费了几个小时试图让 RSelenium 工作，但无济于事。我不确定为什么我的rvest 不起作用，但我会在今晚晚些时候尝试你的方法（我现在无法测试）。谢谢。
一秒。我才意识到在那之前我做了一些事情。会更新。