tornado使用了单进程(当然也可以多进程) + 协程 + I/O多路复用的机制,解决了C10K中因为过多的线程(进程)的上下文切换 而导致的cpu资源的浪费。
tornado中的I/O多路复用前面已经讲过了。本文不做详细解释。
来看一下tornado中的协程模块:tornado.gen:
tornado.gen是根据生成器(generator)实现的,用来更加简单的实现异步。
先来说一下tornado.gen.coroutine的实现思路:
我们知道generator中的yield语句可以使函数暂停执行,而send()方法则可以恢复函数的执行。
tornado将那些异步操作放置到yield语句后,当这些异步操作完成后,tornado会将结果send()至generator中恢复函数执行。
在tornado的官方文档中有这么一句话:
Most asynchronous functions in Tornado return a Future; yielding this object returns its result.
就是说:
在tornado中大多数的异步操作返回一个Future对象
yield Future对象 会返回该异步操作的结果,这句话的意思就是说 假如 ret = yield some_future_obj 当some_future_obj所对应的异步操作完成后会自动的将该异步操作的结果赋值给 ret
那么,Future对象到底是什么?
一 Future对象
先来说说Future对象:
Future对象可以概括为: 一个异步操作的占位符,当然这个占位符有些特殊,它特殊在:
1 这个占位符是一个对象
2 这个对象包含了很多属性,包括_result 以及 _callbacks,分别用来存储异步操作的结果以及回调函数
3 这个对象包含了很多方法,比如添加回调函数,设置异步操作结果等。
4 当这个对象对应的异步操作完成后,该对象会被set_done,然后遍历并运行_callbacks中的回调函数
来看一下Future的简化版:
class Future(object): ''' Future对象主要保存一个回调函数列表_callbacks与一个执行结果_result,当我们set_result时,就会执行_callbacks中的函数 如果set_result或者set_done,就会遍历_callbacks列表并执行callback(self)函数 ''' def __init__(self): self._result = None # 执行的结果 self._callbacks = [] # 用来保存该future对象的回调函数 def result(self, timeout=None): # 如果操作成功,返回结果。如果失败则抛出异常 self._clear_tb_log() if self._result is not None: return self._result if self._exc_info is not None: raise_exc_info(self._exc_info) self._check_done() return self._result def add_done_callback(self, fn): if self._done: fn(self) else: self._callbacks.append(fn) def set_result(self, result): self._result = result self._set_done() def _set_done(self): # 执行结束(成功)后的操作。 self._done = True for cb in self._callbacks: try: cb(self) except Exception: app_log.exception('Exception in callback %r for %r', cb, self) self._callbacks = None
完整源码:
class Future(object): ''' Future对象主要保存一个回调函数列表_callbacks与一个执行结果_result,当我们set_result时,就会执行_callbacks中的函数 ''' def __init__(self): self._done = False # 是否执行完成 self._result = None # 执行的结果 self._exc_info = None # 执行的异常信息 self._log_traceback = False # Used for Python >= 3.4 self._tb_logger = None # Used for Python <= 3.3 self._callbacks = [] # 用来保存该future对象的回调函数 # Implement the Python 3.5 Awaitable protocol if possible # (we can't use return and yield together until py33). if sys.version_info >= (3, 3): exec(textwrap.dedent(""" def __await__(self): return (yield self) """)) else: # Py2-compatible version for use with cython. def __await__(self): result = yield self # StopIteration doesn't take args before py33, # but Cython recognizes the args tuple. e = StopIteration() e.args = (result,) raise e def cancel(self): """Cancel the operation, if possible. 如果可能的话取消操作 tornado对象不支持取消操作,所以总是返回False """ return False def cancelled(self): # 同上 return False def running(self): """Returns True if this operation is currently running.""" return not self._done def done(self): """Returns True if the future has finished running.""" return self._done def _clear_tb_log(self): self._log_traceback = False if self._tb_logger is not None: self._tb_logger.clear() self._tb_logger = None def result(self, timeout=None): """If the operation succeeded, return its result. If it failed, re-raise its exception. 如果操作成功,返回结果。如果失败则抛出异常 This method takes a ``timeout`` argument for compatibility with `concurrent.futures.Future` but it is an error to call it before the `Future` is done, so the ``timeout`` is never used. """ self._clear_tb_log() if self._result is not None: return self._result if self._exc_info is not None: raise_exc_info(self._exc_info) self._check_done() return self._result def exception(self, timeout=None): """If the operation raised an exception, return the `Exception` object. Otherwise returns None. This method takes a ``timeout`` argument for compatibility with `concurrent.futures.Future` but it is an error to call it before the `Future` is done, so the ``timeout`` is never used. """ self._clear_tb_log() if self._exc_info is not None: return self._exc_info[1] else: self._check_done() return None def add_done_callback(self, fn): """Attaches the given callback to the `Future`. 将callback附加到 It will be invoked with the `Future` as its argument when the Future has finished running and its result is available. In Tornado consider using `.IOLoop.add_future` instead of calling `add_done_callback` directly. """ if self._done: fn(self) else: self._callbacks.append(fn) def set_result(self, result): """Sets the result of a ``Future``. 将 result 设置为该future对象的结果 It is undefined to call any of the ``set`` methods more than once on the same object. """ self._result = result self._set_done() def set_exception(self, exception): """Sets the exception of a ``Future.``""" self.set_exc_info( (exception.__class__, exception, getattr(exception, '__traceback__', None))) def exc_info(self): """Returns a tuple in the same format as `sys.exc_info` or None. .. versionadded:: 4.0 """ self._clear_tb_log() return self._exc_info def set_exc_info(self, exc_info): """Sets the exception information of a ``Future.`` Preserves tracebacks on Python 2. .. versionadded:: 4.0 """ self._exc_info = exc_info self._log_traceback = True if not _GC_CYCLE_FINALIZERS: self._tb_logger = _TracebackLogger(exc_info) try: self._set_done() finally: # Activate the logger after all callbacks have had a # chance to call result() or exception(). if self._log_traceback and self._tb_logger is not None: self._tb_logger.activate() self._exc_info = exc_info def _check_done(self): if not self._done: raise Exception("DummyFuture does not support blocking for results") def _set_done(self): # 执行结束(成功)后的操作。 self._done = True for cb in self._callbacks: try: cb(self) except Exception: app_log.exception('Exception in callback %r for %r', cb, self) self._callbacks = None # On Python 3.3 or older, objects with a destructor part of a reference # cycle are never destroyed. It's no longer the case on Python 3.4 thanks to # the PEP 442. if _GC_CYCLE_FINALIZERS: def __del__(self): if not self._log_traceback: # set_exception() was not called, or result() or exception() # has consumed the exception return tb = traceback.format_exception(*self._exc_info) app_log.error('Future %r exception was never retrieved: %s', self, ''.join(tb).rstrip())