【发布时间】:2015-09-20 03:37:57
【问题描述】:
我正在关注“The Bastards Book of Ruby”,我正在尝试使用 nokogiri 构建一个 webscraper,但是当我尝试运行代码时大约四分之一的时间会引发错误:
Crawler.rb:6:in `mkdir': No such file or directory @ dir_s_mkdir - data-hold/nobel (Errno::ENOENT)
from Crawler.rb:6:in `<main>'
我的代码如下:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
DATA_DIR = "data-hold/nobel"
Dir.mkdir(DATA_DIR) unless File.exists?(DATA_DIR)
BASE_WIKIPEDIA_URL = "http://en.wikipedia.org"
LIST_URL = "#{BASE_WIKIPEDIA_URL}/wiki/List_of_Nobel_laureates"
HEADERS_HASH = {"User-Agent" => "Ruby/#{RUBY_VERSION}"}
page = Nokogiri::HTML(open(LIST_URL))
rows = page.css('div.mw-content-ltr table.wikitable tr')
rows[1..-2].each do |row|
hrefs = row.css("td a").map{ |a|
a['href'] if a['href'] =~ /^\/wiki\//
}.compact.uniq
hrefs.each do |href|
remote_url = BASE_WIKIPEDIA_URL + href
local_fname = "#{DATA_DIR}/#{File.basename(href)}.html"
unless File.exists?(local_fname)
puts "Fetching #{remote_url}..."
begin
wiki_content = open(remote_url, HEADERS_HASH).read
rescue Exception=>e
puts "Error: #{e}"
sleep 5
else
File.open(local_fname, 'w'){|file| file.write(wiki_content)}
puts "\t...Success, saved to #{local_fname}"
ensure
sleep 1.0 + rand
end # done: begin/rescue
end # done: unless File.exists?
end # done: hrefs.each
end # done: rows.each
我真的不知道为什么它不创建一个新目录来存储数据。我知道我一定错过了一些非常简单的东西......
【问题讨论】:
标签: ruby web-crawler mkdir