龙卷风生成器引擎中的内存泄漏，当连接关闭时，try/finally 阻塞答案

【问题标题】：Memory leak in tornado generator engine with try/finally block when connections are closed龙卷风生成器引擎中的内存泄漏，当连接关闭时，try/finally 阻塞
【发布时间】：2012-12-19 05:37:33
【问题描述】：

这个很棒的代码显示了 tornado 的 gen 模块中的内存泄漏，当连接关闭而不读取响应时：

import gc
from tornado import web, ioloop, gen

class MainHandler(web.RequestHandler):
    @web.asynchronous
    @gen.engine
    def get(self):
        gc.collect()
        print len(gc.garbage)  # print zombie objects count
        self.a = '*' * 500000000  # ~500MB data
        CHUNK_COUNT = 100
        try:
            for i in xrange(CHUNK_COUNT):
                self.write('*' * 10000)  # write ~10KB of data
                yield gen.Task(self.flush)  # wait for reciever to recieve
            print 'finished'
        finally:
            print 'finally'

application = web.Application([
    (r"/", MainHandler),
    ])

application.listen(8888)
ioloop.IOLoop.instance().start()

现在，多次运行一个简单的测试客户端

#!/usr/bin/python
import urllib
urlopen('http://127.0.0.1:8888/')  # exit without reading response

现在，服务器输出显示，内存使用量增加：

0
WARNING:root:Write error on 8: [Errno 104] Connection reset by peer
1
WARNING:root:Read error on 8: [Errno 104] Connection reset by peer
WARNING:root:error on read
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 361, in _handle_read
    if self._read_to_buffer() == 0:
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 428, in _read_to_buffer
    chunk = self._read_from_socket()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 409, in _read_from_socket
    chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 104] Connection reset by peer
2
ERROR:root:Uncaught exception GET / (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': '127.0.0.1:8888', 'User-Agent': 'Python-urllib/1.17'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1021, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1139, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 120, in wrapper
    runner.run()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 345, in run
    yielded = self.gen.send(next)
  File "test.py", line 10, in get
    self.a = '*' * 500000000
MemoryError
ERROR:root:500 GET / (127.0.0.1) 3.91ms

如果将CHUNK_COUNT设置为1，则10KB的数据可以写入OS连接缓冲区，并且'finished'和'finally'文本将打印到控制台，并且由于生成器已完成，因此不会发生内存泄漏。

但奇怪的部分是，如果你删除 try/finally 块，问题就消失了！！（即使 CHUNK_COUNT 设置为 100）

这是 CPython 或 tornado 上的错误还是...？！

【问题讨论】：

你用的是什么版本的龙卷风？

标签： python memory-leaks generator tornado try-finally

【解决方案1】：

此错误使用 Tornado 2.4.1（提出此问题时的最新版本）进行测试，并在 https://github.com/facebook/tornado/issues/660 上报告。

在提交 https://github.com/facebook/tornado/commit/769bc52e11656788782a6e7a922ef646503f9ab0 中修复并包含在 Tornado 3.0 中的问题。

【讨论】：