Peer pymongo 重置连接

Question

我有一些文档必须从 mongodb 中获取并将其设置为 memcache。这是代码

import memcache
from pymongo import MongoClient

db = mongo_client.job_db.JobParsedData
jobs = db.find().sort("JobId", 1)

def set_to_memcache_raw(jobs):
    print("Setting raw message to memcache")
    count = 0
    for item in jobs:
        job_id = item.get('JobId')
        job_details = item.get('JobDetails')
        if job_id.strip():
            count += 1
            memcache_obj.set(job_id, job_details, time=72000)
            if count % 1000 == 0:
                print("Inserted {} keys in memcache".format(count))
            if count >= 1000000:
                break

但是，经过奇数次迭代后，代码会抛出此错误 -

 Traceback (most recent call last):
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/pool.py", line 450, in receive_message
    self.sock, operation, request_id, self.max_message_size)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/network.py", line 137, in receive_message
    header = _receive_data_on_socket(sock, 16)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/network.py", line 164, in _receive_data_on_socket
    chunk = sock.recv(length)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "memcache-poc.py", line 56, in <module>
    elapsed = time.time() - t0
  File "memcache-poc.py", line 52, in main
    jobs = db.find(query)
  File "memcache-poc.py", line 17, in set_to_memcache_raw
    print("Setting raw message to memcache")
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/cursor.py", line 1056, in _refresh
    self.__max_await_time_ms))
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/mongo_client.py", line 905, in _send_message_with_response
    exhaust)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/mongo_client.py", line 916, in _reset_on_error
    return func(*args, **kwargs)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/server.py", line 136, in send_message_with_response
    response_data = sock_info.receive_message(1, request_id)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/pool.py", line 452, in receive_message
    self._raise_connection_failure(error)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/pool.py", line 550, in _raise_connection_failure
    _raise_connection_failure(self.address, error)
  File "/home/dimension/.virtualenvs/docparser/lib/python3.5/site-packages/pymongo/pool.py", line 211, in _raise_connection_failure
    raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: xxx.xxx.xxx.xxx:27017: [Errno 104] Connection reset by peer

我已经浏览了诸如

之类的链接

pymongo-errors

mongodb-TCP keep-alive

why-does-pymongo-throw-autoreconnect

在上面的代码中没有套接字不活动的问题，因为我的作业对象是一个迭代器，每次在这个对象上调用 next() 时，它都会获取下一个文档（来自 mongo 本身）

我在 Azure 云上安装了 mongodb，我的 TCP 保持活动时间为 7200 秒。我通过触发这个命令得到这个数字

sysctl net.ipv4.tcp_keepalive_time
7200

在这种情况下，在 for 循环上设置 try cacth 块会有所帮助

Answer 1

根据pymongo api 文档here 的问题如下信息，该异常需要手动处理。

exception pymongo.errors.AutoReconnect(message='', errors=None)

当与数据库的连接丢失并尝试自动重新连接时引发。

为了自动重新连接，您必须处理这个异常，认识到导致它的操作不一定成功。未来的操作将尝试打开与数据库的新连接（并将继续引发此异常，直到建立第一个成功的连接）。

我搜索了一些现有的解决方案来解决这个问题，请参见下文。

1. 来自here 的代码 sn-p 用于优雅地处理 PyMongo AutoReconnect 异常。
2. 使用名为MongoDBProxy 的中间件自动处理AutoReconnect 异常。

希望对你有帮助。