如何阅读别人的论坛

Question

我的朋友有一个论坛，里面到处都是包含信息的帖子。有时她想查看论坛中的帖子并得出结论。目前，她通过点击她的论坛来查看帖子，并生成不一定准确的数据图片（在她的大脑中），她从中得出结论。我今天的想法是，我可能会编写一个快速的 Ruby 脚本来解析必要的 HTML，让她真正了解数据在说什么。

我今天第一次使用Ruby的net/http库，遇到了一个问题。虽然我的浏览器在查看我朋友的论坛时没有问题，但 Net::HTTP.new("forumname.net") 方法似乎产生了以下错误：

由于目标机器主动拒绝，无法建立连接。 - 连接（2）

通过谷歌搜索该错误，我了解到它与 MySQL（或类似的东西）有关，不希望像我这样爱管闲事的人远程在那里闲逛：出于安全原因。这对我来说是有道理的，但它让我想知道：我的浏览器怎么会在我朋友的论坛上闲逛，但我的小 Ruby 脚本却没有任何权限。我的脚本有什么方法可以告诉服务器它不是威胁吗？我只想要阅读权而不是写作权？

谢谢大家，

z.

Answer 1

抓取网站？使用mechanize:

#!/usr/bin/ruby1.8

require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new
page = agent.get("http://xkcd.com")
page = page.link_with(:text=>'Forums').click
page = page.link_with(:text=>'Mathematics').click
page = page.link_with(:text=>'Math Books').click
#puts page.parser.to_html    # If you want to see the html you just got
posts = page.parser.xpath("//div[@class='postbody']")
for post in posts
  title = post.at_xpath('h3//text()').to_s
  author = post.at_xpath("p[@class='author']//a//text()").to_s
  body = post.xpath("div[@class='content']//text()").collect do |div|
    div.to_s
  end.join("\n")
  puts '-' * 40
  puts "title: #{title}"
  puts "author: #{author}"
  puts "body:", body
end

输出的第一部分：

----------------------------------------
title: Math Books
author: Cleverbeans
body:
This is now the official thread for questions about math books at any level, fr\
om high school through advanced college courses.
I'm looking for a good vector calculus text to brush up on what I've forgotten.\
 We used Stewart's Multivariable Calculus as a baseline but I was unable to pur\
chase the text for financial reasons at the time. I figured some things may hav\
e changed in the last 12 years, so if anyone can suggest some good texts on thi\
s subject I'd appreciate it.
----------------------------------------
title: Re: Multivariable Calculus Text?
author: ThomasS
body:
The textbooks go up in price and new pretty pictures appear. However, Calculus \
really hasn't changed all that much.
If you don't mind a certain lack of pretty pictures, you might try something li\
ke Widder's Advanced Calculus from Dover. it is much easier to carry around tha\
n Stewart. It is also written in a style that a mathematician might consider no\
rmal. If you think that you might want to move on to real math at some point, i\
t might serve as an introduction to the associated style of writing.

Answer 2

某些网站只能使用“www”子域访问，因此可能会导致问题。

要创建一个 get 请求，您需要使用 Get 方法：

require 'net/http'

url = URI.parse('http://www.forum.site/')
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) {|http|
  http.request(req)
}
puts res.body

你可能还需要在某个时候设置用户代理作为一个选项：

{'User-Agent' => 'Mozilla/5.0 (Windows; U;
    Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1'})