断开连接时的龙卷风内存泄漏

Question

我有一个设置，其中 Tornado 被用作工人的一种传递方式。 Tornado 收到请求，将请求发送给 N 个工作人员，汇总结果并将其发送回客户端。这工作正常，除非由于某种原因发生超时 - 然后我有内存泄漏。

我有一个类似于这个伪代码的设置：

workers = ["http://worker1.example.com:1234/",
           "http://worker2.example.com:1234/", 
           "http://worker3.example.com:1234/" ...]

class MyHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def post(self):
        responses = []

        def __callback(response):
            responses.append(response)
            if len(responses) == len(workers):
                self._finish_req(responses)

        for url in workers:
            async_client = tornado.httpclient.AsyncHTTPClient()
            request = tornado.httpclient.HTTPRequest(url, method=self.request.method, body=body)
            async_client.fetch(request, __callback) 

    def _finish_req(self, responses):
        good_responses = [r for r in responses if not r.error]
        if not good_responses:
            raise tornado.web.HTTPError(500, "\n".join(str(r.error) for r in responses))
        results = aggregate_results(good_responses)
        self.set_header("Content-Type", "application/json")
        self.write(json.dumps(results))
        self.finish()

application = tornado.web.Application([
    (r"/", MyHandler),
])

if __name__ == "__main__":
    ##.. some locking code 
    application.listen()
    tornado.ioloop.IOLoop.instance().start()

我做错了什么？内存泄漏从何而来？

Answer 1

代码看起来不错。泄漏可能在 Tornado 内部。

我只是偶然发现了这一行：

async_client = tornado.httpclient.AsyncHTTPClient()

你知道这个构造函数中的实例化魔法吗？ 来自文档：

"""
The constructor for this class is magic in several respects:  It actually
creates an instance of an implementation-specific subclass, and instances
are reused as a kind of pseudo-singleton (one per IOLoop).  The keyword
argument force_instance=True can be used to suppress this singleton
behavior.  Constructor arguments other than io_loop and force_instance
are deprecated.  The implementation subclass as well as arguments to
its constructor can be set with the static method configure()
"""

所以实际上，您不需要在循环内执行此操作。 （在另一 手，它不应该造成任何伤害。）但是你是哪个实现 使用 CurlAsyncHTTPClient 还是 SimpleAsyncHTTPClient？

如果是SimpleAsyncHTTPClient，请注意代码中的这条注释：

"""
This class has not been tested extensively in production and
should be considered somewhat experimental as of the release of
tornado 1.2. 
"""

您可以尝试切换到 CurlAsyncHTTPClient。或关注 Nikolay Fominyh 的建议并跟踪对 __callback() 的调用。

Answer 2

我不知道问题的根源，看起来gc应该可以解决它，但是你可以尝试两件事。

第一种方法是简化一些引用（看起来在 RequestHandler 完成时可能还有对responses 的引用）：

class MyHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def post(self):
        self.responses = []

        for url in workers:
            async_client = tornado.httpclient.AsyncHTTPClient()
            request = tornado.httpclient.HTTPRequest(url, method=self.request.method, body=body)
            async_client.fetch(request, self._handle_worker_response) 

    def _handle_worker_response(self, response):
        self.responses.append(response)
        if len(self.responses) == len(workers):
            self._finish_req()

    def _finish_req(self):
        ....

如果这不起作用，您可以随时手动调用垃圾回收：

import gc
class MyHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def post(self):
        ....

    def _finish_req(self):
        ....

    def on_connection_close(self):
        gc.collect()