urllib2.HTTPError - 从 python 上的网站获取数据答案

【问题标题】：urllib2.HTTPError - Getting Data from website on pythonurllib2.HTTPError - 从 python 上的网站获取数据
【发布时间】：2015-11-18 02:44:22
【问题描述】：

我有一个脚本，运行时收到错误消息：

urllib2.HTTPError: HTTP Error 400: Bad Request

你能帮帮我吗？

from lxml import html
import urllib2
import urllib

ip_list = []
port_list = []
protocol_list = []
array = [20, 40]
ck = True
i = 0
while i < len(array) :
    h = urllib2.urlopen('http://proxylist.me/proxys/index/'+ str(array[i]))
    HTML_CODE = h.read()
    tree = html.fromstring(HTML_CODE)
    for block in tree.xpath('//tbody/tr'):
        ip, port, _, protocol, _, _, _, _, _ = [
            x.strip()
            for x in block.xpath('.//text()')
                if x.strip() not in ""
            ]
        ip_l = "{}".format(ip)
        port_l = "{}".format(port)
        protocol_l = "{}".format(protocol)
        if ip_l != {}:
            ck = True
            ip_list.append(ip_l)
            port_list.append(port_l)
            protocol_list.append(protocol_l)
            i = i+1
        else:
            ck = False
    print ip_list

我收到此错误：

Traceback (most recent call last):
  File "C:/Users/PC0308-PC/Desktop/get_data_html.py", line 11, in <module>
    h = urllib2.urlopen('http://proxylist.me/proxys/index/'+str(i))
  File "C:\Python27\lib\urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python27\lib\urllib2.py", line 437, in open
    response = meth(req, response)
  File "C:\Python27\lib\urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python27\lib\urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 400: Bad Request

【问题讨论】：

您尝试访问的实际端点是什么？ i 是一个列表，您将其转换为 str，然后将其附加到 URL。我不认为那是你想要做的。
@dursk，你能帮我编辑一下脚本，让它可以获取前两页的数据吗？
在尝试提出更多问题之前，请阅读How do I ask a good question?。
What does your step debugger tell you?

标签： python parsing html-parsing

【解决方案1】：

array = [0, 20, 40]
ck = True
for item in array:
    h = urllib2.urlopen('http://proxylist.me/proxys/index/%s'%(item))
    HTML_CODE = h.read()
    tree = html.fromstring(HTML_CODE)
    for block in tree.xpath('//tbody/tr'):
        ip, port, _, protocol, _, _, _, _, _ = [
            x.strip()
            for x in block.xpath('.//text()')
                if x.strip() not in ""
            ]
        ip_l = "{}".format(ip)
        port_l = "{}".format(port)
        protocol_l = "{}".format(protocol)
        if ip_l != {}:
            ck = True
            ip_list.append(ip_l)
            port_list.append(port_l)
            protocol_list.append(protocol_l)
        else:
            ck = False
    print ip_list

它可以在我的 Windows 机器上运行，解析来自 http://proxylist.me/proxys/index 的前 3 页

顺便说一句，您的代码从一开始就运行良好，但它只解析了第一页。

【讨论】：