Google 自定义搜索 API 和 Ruby

Question

我想编写一个 Google 搜索抓取器/解析器，以从 Google 的linkedin.com 索引中提取员工。 Linkedin 关闭了他们的 API，所以我先写了一个 Mechanize/Nokogiri 刮板，这让我得到了验证码，所以我使用 Google 搜索 API gem 重写了脚本。

问题是，我不知道从哪里开始让它带回超过第一页的结果，官方文档甚至不能被描述为“稀疏”。

这是只返回第 1 页的代码：

require 'rubygems'
require 'google/api_client'
require 'json'
require 'pp'
puts "What organisation's employees shall we get today?"
organisation = gets.chomp
puts "Harvesting Google Search Results - This may take some time"

apikey = "1234"
cxid = "5678"
client = Google::APIClient.new(:key => apikey, :authorization => nil,          :application_name => "linkedout", :application_version => "beta_0.5")
  search = client.discovered_api('customsearch')
  response = client.execute(
 :api_method => search.cse.list,
 :parameters => {
   'q' => 'current ' + organisation + ' site:linkedin.com',
   'maxResults' => 100,
   'key' => apikey,
   'cx' => cxid
   }
)

status, headers, body = response
jsonresponse = response.body

employees = []
@tags = JSON.parse(jsonresponse)['items']
  @tags.each do |tag|
     x = tag['title']
     x.gsub!(/ \| LinkedIn/, "")
     x.downcase!
     x.gsub!(/ profiles/, "")
        employees << x
       end
    employees = employees.uniq
    puts employees

我们将非常感激任何帮助 - 我仍在学习这些东西。

编辑：

这里是 JSON google 的 API 返回的 sn-p：

"items": [
  {
   "kind": "customsearch#result",
   "title": "Tina Minor - Recruiter, The Walt Disney Company | LinkedIn",
   "htmlTitle": "Tina Minor - Recruiter, The \u003cb\u003eWalt     Disney\u003c/b\u003e Company | LinkedIn",
   "link": "https://www.linkedin.com/pub/tina-minor-recruiter-the-walt-disney-    company/5/849/5a6",
   "displayLink": "www.linkedin.com",
   "snippet": "View Tina Minor - Recruiter, The Walt Disney Company's     professional profile on \n... Current. The Walt Disney Company. Previous. True     Religion Brand Jeans, ...",
   "htmlSnippet": "View Tina Minor - Recruiter, The \u003cb\u003eWalt     Disney\u003c/b\u003e Company's professional profile on \u003cbr\u003e\n...     \u003cb\u003eCurrent\u003c/b\u003e. The \u003cb\u003eWalt Disney\u003c/b\u003e     Company. Previous. True Religion Brand Jeans, ...",
   "formattedUrl": "https://www.linkedin.com/pub/tina-minor-recruiter-the-    walt-   disney.../5a6",
       "htmlFormattedUrl": "https://www.linkedin.com/pub/tina-minor-recruiter-    the-       \u003cb\u003ewalt\u003c/b\u003e-    \u003cb\u003edisney\u003c/b\u003e.../5a6",
       "pagemap": {
        "cse_image": [
         {
           "src":          "https://media.licdn.com/mpr/mpr/shrink_200_200/p/8/005/09b/3f2/1eb6f83.jpg"
      }
    ],
    "person": [
     {
      "location": "Greater Los Angeles Area",
      "role": "Recruiter, Talent Acquisition at The Walt Disney Company"
     }
    ],
    "cse_thumbnail": [
     {
      "width": "160",
      "height": "160",
      "src": "https://encrypted-tbn1.gstatic.com/images?    q=tbn:ANd9GcTbmlbDVBOMKTtOA_D88aFaPuZ9MjABABwumzBPk0F2x2P2-0puaIRlktce"
     }
    ],
    "metatags": [
     {
      "globaltrackingurl": "//www.linkedin.com/mob/tracking",
      "globaltrackingappname": "profile",
      "globaltrackingappid": "webTracking",
      "lnkd-track-json-lib": "https://static.licdn.com/scds/concat/common/js?    h=2jds9coeh4w78ed9wblscv68v-ebbt2vixcc5qz0otts5io08xv&fc=2",
      "treeid": "SnQhTqcr1RNgnKS8RSsAAA==",
      "appname": "profile",
      "pageimpressionid": "29ca4803-0233-4934-955a-1959a37dfbbf",
      "pagekey": "nprofile_v2_public_fs",
      "analyticsurl": "/analytics/noauthtracker",
      "msapplication-tileimage":     "https://static.licdn.com/scds/common/u/images/logos/linkedin/logo-in-win8-tile-    144_v1.png",
      "msapplication-tilecolor": "#0077B5",
      "application-name": "LinkedIn",
      "remote-nav-init-marker": "true"
     }
    ],
    "hcard": [
     {
      "fn": "Tina Minor - Recruiter, The Walt Disney Company",
      "title": "Recruiter, Talent Acquisition at The Walt Disney Company"
     }
    ]
   }
  }
  ...

Answer 1

根据Search request metadata，当有附加结果时，项目旁边应该有一个nextPage值返回。但是它总是显示Note: This API returns up to the first 100 results only.，所以看起来您已经获得了最大数量的结果。