我想从一个网站获取很多页面，比如
curl "http://farmsubsidy.org/DE/browse?page=[0000-3603]" -o "de.#1"
但在 python 中获取页面数据，而不是磁盘文件。
有人可以张贴pycurl 代码来做到这一点，
或快速urllib2（不是一次一个），如果可能的话，
或者说“算了吧，curl 更快更健壮”？谢谢
【问题讨论】：
标签： python curl urllib2 pycurl
所以你有 2 个问题，让我用一个例子来告诉你。请注意，pycurl 已经完成了多线程/不是一次一个，没有您的辛勤工作。
#! /usr/bin/env python
import sys, select, time
import pycurl,StringIO
c1 = pycurl.Curl()
c2 = pycurl.Curl()
c3 = pycurl.Curl()
c1.setopt(c1.URL, "http://www.python.org")
c2.setopt(c2.URL, "http://curl.haxx.se")
c3.setopt(c3.URL, "http://slashdot.org")
s1 = StringIO.StringIO()
s2 = StringIO.StringIO()
s3 = StringIO.StringIO()
c1.setopt(c1.WRITEFUNCTION, s1.write)
c2.setopt(c2.WRITEFUNCTION, s2.write)
c3.setopt(c3.WRITEFUNCTION, s3.write)
m = pycurl.CurlMulti()
m.add_handle(c1)
m.add_handle(c2)
m.add_handle(c3)
# Number of seconds to wait for a timeout to happen
SELECT_TIMEOUT = 1.0
# Stir the state machine into action
while 1:
ret, num_handles = m.perform()
if ret != pycurl.E_CALL_MULTI_PERFORM:
break
# Keep going until all the connections have terminated
while num_handles:
# The select method uses fdset internally to determine which file descriptors
# to check.
m.select(SELECT_TIMEOUT)
while 1:
ret, num_handles = m.perform()
if ret != pycurl.E_CALL_MULTI_PERFORM:
break
# Cleanup
m.remove_handle(c3)
m.remove_handle(c2)
m.remove_handle(c1)
m.close()
c1.close()
c2.close()
c3.close()
print "http://www.python.org is ",s1.getvalue()
print "http://curl.haxx.se is ",s2.getvalue()
print "http://slashdot.org is ",s3.getvalue()
最后这些代码主要是基于pycurl网站上的一个例子=.=
也许你真的应该阅读文档。人们在上面花费了大量时间。
【讨论】：
这里有一个基于urllib2和线程的解决方案。
import urllib2
from threading import Thread
BASE_URL = 'http://farmsubsidy.org/DE/browse?page='
NUM_RANGE = range(0000, 3603)
THREADS = 2
def main():
for nums in split_seq(NUM_RANGE, THREADS):
t = Spider(BASE_URL, nums)
t.start()
def split_seq(seq, num_pieces):
start = 0
for i in xrange(num_pieces):
stop = start + len(seq[i::num_pieces])
yield seq[start:stop]
start = stop
class Spider(Thread):
def __init__(self, base_url, nums):
Thread.__init__(self)
self.base_url = base_url
self.nums = nums
def run(self):
for num in self.nums:
url = '%s%s' % (self.base_url, num)
data = urllib2.urlopen(url).read()
print data
if __name__ == '__main__':
main()
【讨论】：
您可以将其放入 for 循环内的 bash 脚本中。
但是，您在使用 python 解析每个页面时可能会取得更好的成功。 http://www.securitytube.net/Crawling-the-Web-for-Fun-and-Profit-video.aspx 您将能够获得准确的数据并将其同时保存到数据库中。 http://www.securitytube.net/Storing-Mined-Data-from-the-Web-for-Fun-and-Profit-video.aspx
【讨论】：
如果你想使用python爬取网站，你应该看看scrapy http://scrapy.org
【讨论】：
使用BeautifulSoup4 和requests -
抢首页：
page = Soup(requests.get(url='http://rootpage.htm').text)
创建一个请求数组：
from requests import async
requests = [async.get(url.get('href')) for url in page('a')]
responses = async.map(requests)
[dosomething(response.text) for response in responses]
请求需要 gevent 顺便说一句。
【讨论】：
我可以推荐你使用human_curl的异步模块
看例子：
from urlparse import urljoin
from datetime import datetime
from human_curl.async import AsyncClient
from human_curl.utils import stdout_debug
def success_callback(response, **kwargs):
"""This function call when response successed
"""
print("success callback")
print(response, response.request)
print(response.headers)
print(response.content)
print(kwargs)
def fail_callback(request, opener, **kwargs):
"""Collect errors
"""
print("fail callback")
print(request, opener)
print(kwargs)
with AsyncClient(success_callback=success_callback,
fail_callback=fail_callback) as async_client:
for x in xrange(10000):
async_client.get('http://google.com/', params=(("x", str(x)),)
async_client.get('http://google.com/', params=(("x", str(x)),),
success_callback=success_callback, fail_callback=fail_callback)
用法很简单。然后页面成功加载失败的 async_client 调用你回调。您还可以指定并行连接的数量。
【讨论】：