Python urllib.request 和 urllib.error 用法

Question

我有少量 Python 代码使用 urllib.request 和 urllib.error 来检查 URL 的状态。如果出现错误，这应该会触发 SNS 发布到主题。

import boto3
from urllib.request import urlopen
from urllib.error import HTTPError
from os import environ

region = 'eu-west-2'

def get_status_code(url, topic):
    try:
        connection = urlopen(url, timeout=10)
        status = connection.getcode()
        print('%s Status is %s' % (url, status))
        connection.close()
    except HTTPError as err:
        status = err.getcode()
        print('%s: Status is unreachable - %s' % (url, status))
        sns_client = boto3.client('sns', region_name=region)
        message = sns_client.publish(TopicArn=topic,
                                     Message='%s is unreachable - HTTP error code is: %s' % (url, status)
                                     )
        print("Publish to SNS, Lambda update triggered: {}".format(message))

def lambda_handler(event, context):
    dns = environ['dns']
    sns_topic = environ['sns_topic_arn']

    get_status_code(dns, sns_topic)

作为测试的一部分，我经历了不同程度的成功。如果我在 urlopen 中保持超时，则它无法触发 except 处理程序，但确实会转储失败。如果设置超时限制并被认为是错误，那么为什么它不触发异常处理程序？

timed out: timeout
Traceback (most recent call last):
  File "/var/task/lambda_function1.py", line 27, in lambda_handler
    get_status_code(elb_dns, sns_topic)
  File "/var/task/lambda_function1.py", line 10, in get_status_code
    connection = urlopen(url, timeout=10)
  File "/var/lang/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/var/lang/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/var/lang/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/var/lang/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1377, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/var/lang/lib/python3.6/http/client.py", line 1379, in getresponse
    response.begin()
  File "/var/lang/lib/python3.6/http/client.py", line 311, in begin
    version, status, reason = self._read_status()
  File "/var/lang/lib/python3.6/http/client.py", line 272, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/var/lang/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

如果我从 urlopen 中删除超时，则作业在 3 分钟后超时并触发异常处理程序，然后成功完成（我知道它是否在 10 秒后没有工作，它不会去）。

如果我使用 HTTPS URL，还会出现第二个意外错误。它也无法触发异常处理程序。

<urlopen error timed out>: URLError
Traceback (most recent call last):
  File "/var/task/lambda_function1.py", line 24, in lambda_handler
    get_status_code(elb_dns, sns_topic)
  File "/var/task/lambda_function1.py", line 8, in get_status_code
    connection = urlopen(url, timeout=10)
  File "/var/lang/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/var/lang/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/var/lang/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/var/lang/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1392, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error timed out>

Answer 1

因此，当我添加输出的错误时，我注意到它们没有写入异常处理程序所期望的 HTTPError（我预计超时至少是默认超时？？？）：

import boto3
import socket
from socket import AF_INET, SOCK_DGRAM
from urllib.request import urlopen
from urllib.error import HTTPError
from urllib.error import URLError
from os import environ

region = 'eu-west-2'

def get_status_code(url, topic):
    try:
        connection = urlopen(url, timeout=10)
        status = connection.getcode()
        print('%s Status is %s' % (url, status))
        connection.close()
    except (socket.timeout, URLError, HTTPError) as err:
        status = err
        print('%s: Status is unreachable - %s' % (url, status))
        sns_client = boto3.client('sns', region_name=region)
        message = sns_client.publish(TopicArn=topic,
                                     Message='%s is unreachable - HTTP error code is: %s' % (url, status)
                                     )
        print("Publish to SNS, Lambda update triggered: {}".format(message))

def lambda_handler(event, context):
    dns = environ['dns']
    sns_topic = environ['sns_topic_arn']

    get_status_code(dns, sns_topic)

所以我将新错误添加到异常处理程序中，现在它们正在被捕获和处理。它确实破坏了我的“HTTP 错误代码是”，因为我现在只收到错误消息，但它比以前工作得更好。