如何在龙卷风中启动一个不会阻塞请求处理的后台任务？ [复制]

Question

我正在编写一个服务，它消耗磁盘空间并且必须不时清理它。在此之前，清理是在请求处理中执行的。不幸的是，如果需要清理大量磁盘空间，则会消耗大量时间并且服务会挂起。我试图将清洁程序重写为龙卷风的未来，但对从发电机创造未来（或类似的东西）的方式感到困惑。下面是我的代码的简化：

import tornado
import tornado.ioloop
import tornado.web
from tornado.process import Subprocess
#from tornado.stack_context import run_in_stack_context, NullContext
from time import time
import random
import json
import asyncio 

class meta_doc(type):
    def __init__(cls, name, bases, methods):
        super().__init__(name, bases, methods)
        cls.storage_size=0

class Documentation_parsing(tornado.web.RequestHandler, metaclass=meta_doc):
    max_storage_size=200
    optimal_storage_size=100
    cleaning=False
    @classmethod
    @tornado.gen.coroutine
    def _collect_old_folders(cls):
        print('start cleaning')
        for d in subfolders:
            if cls.storage_size<cls.optimal_storage_size:
                break
            delta=random.randint(5, 15)
            time.sleep(random.uniform(0.5, 3))
            cls.storage_size-=delta
            print('Folder have been deleted. Folder size:', cls.storage_size)
            yield None
        cls.cleaning=False  
        print('finish cleaning')
    @classmethod
    def collect_old_folders(cls):
        if not cls.cleaning:
            cls.cleaning=True
            tornado.ioloop.IOLoop.current().add_future(cls._collect_old_folders, lambda f: f.result())
    @tornado.gen.coroutine
    def post(self):
        request_id=self.get_body_argument("request_id", default='')
        self.__class__.storage_size+=random.randint(5, 15)
        if self.storage_size>self.max_storage_size:
            self.collect_old_folders()
        self.write(json.dumps({'request_id': request_id, 'storage_size': self.storage_size}))
        print('process request: request_id {0}, storage size {1}'.format(request_id, self.storage_size))

ApplicationSuffixes=[(r'/main_parsing.*', Documentation_parsing)]

if __name__=='__main__':
    app = tornado.web.Application(ApplicationSuffixes)
    app.listen(9999)
    tornado.ioloop.IOLoop.current().start()

此代码返回未来断言错误。我寻找了另一个从生成器或协程中创造未来的装饰器，但没有找到。请帮我解决这个问题。 UPD。我已经尝试过上述问题的解决方案，但收到“无法导入 run_in_stack_context”。这就是为什么在我的代码中注释了这种导入

Answer 1

解决它：

import tornado
import tornado.ioloop
import tornado.web
from tornado.process import Subprocess
#from tornado.stack_context import run_in_stack_context, NullContext
import time
import random
import json
import asyncio 

class meta_doc(type):
    def __init__(cls, name, bases, methods):
        super().__init__(name, bases, methods)
        cls.storage_size=0

class Documentation_parsing(tornado.web.RequestHandler, metaclass=meta_doc):
    max_storage_size=200
    optimal_storage_size=100
    cleaning=False
    @classmethod
    @tornado.gen.coroutine
    def _collect_old_folders(cls):
        print('start cleaning')
        while True:
            if cls.storage_size<cls.optimal_storage_size:
                break
            delta=random.randint(5, 15)
            time.sleep(random.uniform(0.5, 3))
            cls.storage_size-=delta
            print('Folder have been deleted. Folder size:', cls.storage_size)
            yield None
        cls.cleaning=False  
        print('finish cleaning')
    @classmethod
    def collect_old_folders(cls):
        def func(inp):
            print('called', inp)
        if not cls.cleaning:
            cls.cleaning=True
            future=cls._collect_old_folders()
            print(type(future))
            tornado.ioloop.IOLoop.current().add_future(future, func)
    @tornado.gen.coroutine
    def post(self):
        request_id=self.get_body_argument("request_id", default='')
        self.__class__.storage_size+=random.randint(5, 15)
        if self.storage_size>self.max_storage_size:
            self.collect_old_folders()
        self.write(json.dumps({'request_id': request_id, 'storage_size': self.storage_size}))
        print('process request: request_id {0}, storage size {1}'.format(request_id, self.storage_size))

ApplicationSuffixes=[(r'/main_parsing.*', Documentation_parsing)]

if __name__=='__main__':
    app = tornado.web.Application(ApplicationSuffixes)
    app.listen(8999)
    tornado.ioloop.IOLoop.current().start()

评论：装饰器 tornado.gen.coroutine 返回一个返回未来的函数。所以我只需要调用方法来获得未来。在 Tornado 的 5.0.2 版本中，我可以直接向 IOLoop 添加未来。唯一的想法是，除了未来我必须传递一个函数作为第二个参数。该函数接收异常或未来性能的结果。