Python3 functools lru_cache RuntimeError

Question

from functool import lru_cache


@lru_cache
def fibonacci(n):
    """0, 1, 1, 2, 3, 5, 8, 13, 21, 34
    """

    if n == 0:
        yield 0
    elif n == 1:
        yield 1
    else:
        yield next(fibonacci(n - 1)) + next(fibonacci(n - 2))

如果我像这样使用 @lru_cache 装饰器调用此函数：

for x in range(10):
    print(next(fibonacci(x)))

我明白了：

StopIteration

The above exception was the direct cause of the following exception:

RuntimeError: generator raised StopIteration

我已经做了很多搜索，但我不知道如何解决这个问题。没有装饰器，一切正常。

Answer 1

如果您确实想要缓存并因此重用生成器迭代器，请确保它们确实支持这一点。也就是说，让他们产生他们的结果，而不是一次，而是重复。例如：

@lru_cache
def fibonacci(n):
    """0, 1, 1, 2, 3, 5, 8, 13, 21, 34
    """

    if n == 0:
        while True:
            yield 0
    elif n == 1:
        while True:
            yield 1
    else:
        result = next(fibonacci(n - 1)) + next(fibonacci(n - 2))
        while True:
            yield result

测试：

>>> for x in range(10):
        print(next(fibonacci(x)))

0
1
1
2
3
5
8
13
21
34

Answer 2

你可以使用记忆装饰器

参考：Can I memoize a Python generator? Jasmijn 的回答

代码

from itertools import tee
from types import GeneratorType

Tee = tee([], 1)[0].__class__

def memoized(f):
    cache={}
    def ret(*args):
        if args not in cache:
            cache[args]=f(*args)
        if isinstance(cache[args], (GeneratorType, Tee)):
            # the original can't be used any more,
            # so we need to change the cache as well
            cache[args], r = tee(cache[args])
            return r
        return cache[args]
    return ret

@memoized
def Fibonacci(n):
    """0, 1, 1, 2, 3, 5, 8, 13, 21, 34
    """

    if n == 0:
        yield 0
    elif n == 1:
        yield 1
    else:
        yield next(fibonacci_mem(n - 1)) + next(fibonacci_mem(n - 2))

时序测试

总结

测试 n 从 1 到 20 orig：原始代码 lru：使用 lru 缓存 mem：使用记忆装饰器

每种算法运行 3 次的时间以秒为单位

结果表明 lru_cache 技术提供了最快的运行时间（即更短的时间）

n: 1 orig: 0.000008, lru 0.000006, mem: 0.000015
n: 10 orig: 0.000521, lru 0.000024, mem: 0.000057
n: 15 orig: 0.005718, lru 0.000013, mem: 0.000035
n: 20 orig: 0.110947, lru 0.000014, mem: 0.000040
n: 25 orig: 1.503879, lru 0.000018, mem: 0.000042

定时测试代码

from itertools import tee
from types import GeneratorType
from functools import lru_cache

Tee = tee([], 1)[0].__class__

def memoized(f):
    cache={}
    def ret(*args):
        if args not in cache:
            cache[args]=f(*args)
        if isinstance(cache[args], (GeneratorType, Tee)):
            # the original can't be used any more,
            # so we need to change the cache as well
            cache[args], r = tee(cache[args])
            return r
        return cache[args]
    return ret
    
def fibonacci(n):
    """0, 1, 1, 2, 3, 5, 8, 13, 21, 34
    """

    if n == 0:
        yield 0
    elif n == 1:
        yield 1
    else:
        yield next(fibonacci(n - 1)) + next(fibonacci(n - 2))

@memoized
def fibonacci_mem(n):
    """0, 1, 1, 2, 3, 5, 8, 13, 21, 34
    """

    if n == 0:
        yield 0
    elif n == 1:
        yield 1
    else:
        yield next(fibonacci_mem(n - 1)) + next(fibonacci_mem(n - 2))

@lru_cache
def fibonacci_cache(n):
    """0, 1, 1, 2, 3, 5, 8, 13, 21, 34
    """

    if n == 0:
        while True:
            yield 0
    elif n == 1:
        while True:
            yield 1
    else:
        result = next(fibonacci_cache(n - 1)) + next(fibonacci_cache(n - 2))
        while True:
            yield result

from timeit import timeit

cnt = 3
for n in [1, 10, 15, 20, 25]:
  t_orig = timeit(lambda:next(fibonacci(n)), number = cnt)
  t_mem = timeit(lambda:next(fibonacci_mem(n)), number = cnt)
  t_cache = timeit(lambda:next(fibonacci_cache(n)), number = cnt)
  print(f'n: {n} orig: {t_orig:.6f}, lru {t_cache:.6f}, mem: {t_mem:.6f}')