【发布时间】：2020-12-19 19:18:46
【问题描述】：
我正在尝试使用来自 Docker 容器的 Python 从外部服务获取数据。 我需要到达两个不同的端点，比如说：
endpointA = "https://myhost:1234/resA" # Requests take about 1 minute to complete
endpointB = "https://myhost:1234/resB" # Requests take about 8 minutes to complete
我尝试使用两个不同的库，Requests：
import requests
headers = {'Authorization': 'Basic XXXXXXXXXXXXXXXXXXXXXX', 'Content-Type': 'application/json'}
response = requests.get(endpointA, verify=False, headers=headers)
和 Pycurl：
import pycurl
headers = ['Authorization: Basic XXXXXXXXXXXXXXXXXXXXXX', 'Content-Type: application/json']
buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.VERBOSE, True)
c.setopt(c.URL, endpointA)
c.setopt(c.WRITEDATA, buffer)
c.setopt(c.SSL_VERIFYPEER, False)
c.setopt(c.SSL_VERIFYHOST, False)
c.setopt(c.HTTPHEADER, headers)
c.perform()
c.close()
我的代码在本地运行时没有问题（endpointA 和 endpointB 都返回）。
在 Docker 容器内运行时，对端点 A 的请求有效，但对端点 B 的请求在 300 秒后引发异常：
在请求的情况下：
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
在pycurl的情况下：
Traceback (most recent call last):
File "test.py", line 230, in get_data
c.perform()
pycurl.error: (52, 'Empty reply from server')
这只发生在 Docker 中，因为在我的控制台中一切正常。
我尝试了 python:3.6 和 python:3.8 Docker 镜像。
我还尝试从容器内部对两个端点进行 cURL 并且服务器正确回复。
【问题讨论】：
-
由于端点 B 需要 8 分钟才能运行，我假设它是一个非常耗费资源的过程。我认为发生的事情是端点 B 的 docker 容器超过 100% 的 CPU 或内存利用率，这将导致对 GET 请求没有响应，因此你是错误的。特别是 300 秒，因为这是超过 100% 内存可能需要多长时间。您可以通过运行端点 B 来检查这一点，然后通过 ssh 进入容器并使用
watch htop监视使用情况。如果内存使用率超过 100%，该过程将崩溃。您可以通过为 docker 引擎分配更多资源来解决此问题。 -
感谢您的回复，但这里没有运气。进程在服务器端占用大量资源，我的客户端只等待并获得响应（几 kb）。无论如何，我尝试分配更多资源，但问题仍然存在，监控显示没有资源问题。
标签： python docker curl python-requests pycurl