使用 Nokogori 遍历多个 URL 以解析 HTML

Question

我正在尝试使用 Nokogiri 从多个供应商处获取商品的名称和价格。我通过方法参数将 CSS 选择器（查找名称和价格）传递给 Nokogiri。

关于如何将多个 URL 传递给“scrape”方法同时传递其他参数（例如：供应商、i​​tem_path）的任何指导？还是我以完全错误的方式解决这个问题？

代码如下：

require 'rubygems' # Load Ruby Gems
require 'nokogiri' # Load Nokogiri
require 'open-uri' # Load Open-URI

@@collection = Array.new # Array to hold meta hash

def scrape(url, vendor, item_path, name_path, price_path)
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.each do |item| # Iterates through each item on grid
        @@collection << meta = Hash.new # Creates a new hash then add to global array
        meta[:vendor] = vendor
        meta[:name] = item.css(name_path).text.strip
        meta[:price] = item.css(price_path).to_s.scan(/\d+[.]\d+/).join 
    end
end

scrape( "page_a.html", "Sample Vendor A", "#products", ".title", ".prices")
scrape( ["page_a.html", "page_b.html"], "Sample Vendor B",  "#items", ".productname", ".price")

Answer 1

您可以传递多个url's，就像您在第二个示例中所做的那样：

scrape( ["page_a.html", "page_b.html"], "Sample Vendor B",  "#items", ".productname", ".price")

您的scrape 方法必须遍历那些urls，例如：

def scrape(urls, vendor, item_path, name_path, price_path)
  urls.each do |url|
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.each do |item| # Iterates through each item on grid
        @@collection << meta = Hash.new # Creates a new hash then add to global array
        meta[:vendor] = vendor
        meta[:name] = item.css(name_path).text.strip
        meta[:price] = item.css(price_path).to_s.scan(/\d+[.]\d+/).join 
    end 
  end   
end

这也意味着第一个例子也需要作为数组传递：

scrape( ["page_a.html"], "Sample Vendor A", "#products", ".title", ".prices")

Answer 2

仅供参考，使用@@collection 是不合适的。相反，编写您的方法以返回一个值：

def scrape(urls, vendor, item_path, name_path, price_path)
  collection = []
  urls.each do |url|
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.each do |item| # Iterates through each item on grid
      collection << {
        :vendor => vendor,
        :name   => item.css(name_path).text.strip,
        :price  => item.css(price_path).to_s.scan(/\d+[.]\d+/).join
      }
    end 
  end   

  collection
end

可以简化为：

def scrape(urls, vendor, item_path, name_path, price_path)
  urls.map { |url|
    doc = Nokogiri::HTML(open(url)) # Opens URL
    items = doc.css(item_path) # Sets items
    items.map { |item| # Iterates through each item on grid
      {
        :vendor => vendor,
        :name   => item.css(name_path).text.strip,
        :price  => item.css(price_path).to_s.scan(/\d+[.]\d+/).join
      }
    } 
  }
end