任何加速 itertool.product 的方法

Question

我正在使用 itertools.product 来查找资产可以采用的可能权重，因为所有权重的总和为 100。

min_wt = 10
max_wt = 50
step = 10
nb_Assets = 5

weight_mat = []
for i in itertools.product(range(min_wt, (max_wt+1), step), repeat = nb_Assets):
    if sum(i) == 100:
        weight = [i]
        if np.shape(weight_mat)[0] == 0:
            weight_mat = weight
        else:
            weight_mat = np.concatenate((weight_mat, weight), axis = 0)

上面的代码可以工作，但是它太慢了，因为它通过了不可接受的组合，例如 [50,50,50,50,50] 最终测试了 3125 个组合而不是 121 个可能的组合。有什么办法可以在循环中添加“和”条件来加快速度？

Answer 1

许多改进是可能的。

对于初学者，可以使用 itertools.combinations_with_replacement() 减少搜索空间，因为求和是可交换的。

另外，最后一个加数应该计算而不是测试。例如，如果t[:4] 是(10, 20, 30, 35)，您可以将t[4] 计算为1 - sum(t)，给出值5。这将比在(10, 20, 30, 35, x) 中尝试一百个 x 值快 100 倍。

Answer 2

您可以编写一个递归算法来尽早修剪所有不可能的选项：

def make_weight_combs(min_wt, max_wt, step, nb_assets, req_wt):
    weights = range(min_wt, max_wt + 1, step)
    current = []
    yield from _make_weight_combs_rec(weights, nb_assets, req_wt, current)

def _make_weight_combs_rec(weights, nb_assets, req_wt, current):
    if nb_assets <= 0:
        yield tuple(current)
    else:
        # Discard weights that cannot possibly be used
        while weights and weights[0] + weights[-1] * (nb_assets - 1) < req_wt:
            weights = weights[1:]
        while weights and weights[-1] + weights[0] * (nb_assets - 1) > req_wt:
            weights = weights[:-1]
        # Add all possible weights
        for w in weights:
            current.append(w)
            yield from _make_weight_combs_rec(weights, nb_assets - 1, req_wt - w, current)
            current.pop()

min_wt = 10
max_wt = 50
step = 10
nb_assets = 5
req_wt = 100
for comb in make_weight_combs(min_wt, max_wt, step, nb_assets, req_wt):
    print(comb, sum(comb))

输出：

(10, 10, 10, 20, 50) 100
(10, 10, 10, 30, 40) 100
(10, 10, 10, 40, 30) 100
(10, 10, 10, 50, 20) 100
(10, 10, 20, 10, 50) 100
(10, 10, 20, 20, 40) 100
(10, 10, 20, 30, 30) 100
(10, 10, 20, 40, 20) 100
...

如果权重的顺序无关紧要（例如，(10, 10, 10, 20, 50) 和 (50, 20, 10, 10, 10) 相同），那么您可以修改 for 循环如下：

for i, w in enumerate(weights):
    current.append(w)
    yield from _make_weight_combs_rec(weights[i:], nb_assets - 1, req_wt - w, current)
    current.pop()

它给出了输出：

(10, 10, 10, 20, 50) 100
(10, 10, 10, 30, 40) 100
(10, 10, 20, 20, 40) 100
(10, 10, 20, 30, 30) 100
(10, 20, 20, 20, 30) 100
(20, 20, 20, 20, 20) 100

Answer 3

比较所提供解决方案的性能：

import itertools
import timeit
import numpy as np


# original code from question
def f1():
    min_wt = 10
    max_wt = 50
    step = 10
    nb_assets = 5

    weight_mat = []
    for i in itertools.product(range(min_wt, (max_wt+1), step), repeat=nb_assets):
        if sum(i) == 100:
            weight = [i, ]
            if np.shape(weight_mat)[0] == 0:
                weight_mat = weight
            else:
                weight_mat = np.concatenate((weight_mat, weight), axis=0)

    return weight_mat


# code from question using list instead of numpy array
def f1b():
    min_wt = 10
    max_wt = 50
    step = 10
    nb_assets = 5

    weight_list = []
    for i in itertools.product(range(min_wt, (max_wt+1), step), repeat=nb_assets):
        if sum(i) == 100:
            weight_list.append(i)

    return weight_list


# calculating the last element of each tuple
def f2():
    min_wt = 10
    max_wt = 50
    step = 10
    nb_assets = 5

    weight_list = []
    for i in itertools.product(range(min_wt, (max_wt+1), step), repeat=nb_assets-1):
        the_sum = sum(i)
        if the_sum < 100:
            last_elem = 100 - the_sum
            if min_wt <= last_elem <= max_wt:
                weight_list.append(i + (last_elem, ))

    return weight_list


# recursive solution from user kaya3 (https://stackoverflow.com/a/58823843/9225671)
def constrained_partitions(n, k, min_w, max_w, w_step=1):
    if k < 0:
        raise ValueError('Number of parts must be at least 0')
    elif k == 0:
        if n == 0:
            yield ()
    else:
        for w in range(min_w, max_w+1, w_step):
            for p in constrained_partitions(n-w, k-1, min_w, max_w, w_step):
                yield (w,) + p

def f3():
    return list(constrained_partitions(100, 5, 10, 50, 10))


# recursive solution from user jdehesa (https://stackoverflow.com/a/58823990/9225671)
def make_weight_combs(min_wt, max_wt, step, nb_assets, req_wt):
    weights = range(min_wt, max_wt + 1, step)
    current = []
    yield from _make_weight_combs_rec(weights, nb_assets, req_wt, current)

def _make_weight_combs_rec(weights, nb_assets, req_wt, current):
    if nb_assets <= 0:
        yield tuple(current)
    else:
        # Discard weights that cannot possibly be used
        while weights and weights[0] + weights[-1] * (nb_assets - 1) < req_wt:
            weights = weights[1:]
        while weights and weights[-1] + weights[0] * (nb_assets - 1) > req_wt:
            weights = weights[:-1]
        # Add all possible weights
        for w in weights:
            current.append(w)
            yield from _make_weight_combs_rec(weights, nb_assets - 1, req_wt - w, current)
            current.pop()

def f4():
    return list(make_weight_combs(10, 50, 10, 5, 100))

我使用timeit 测试了这些函数，如下所示：

print(timeit.timeit('f()', 'from __main__ import f1 as f', number=100))

使用问题中的参数的结果：

# min_wt = 10
# max_wt = 50
# step = 10
# nb_assets = 5
0.07021828400320373       # f1 - original code from question
0.041302188008558005      # f1b - code from question using list instead of numpy array
0.009902548001264222      # f2 - calculating the last element of each tuple
0.10601829699589871       # f3 - recursive solution from user kaya3
0.03329997700348031       # f4 - recursive solution from user jdehesa

如果我扩大搜索空间（减少步骤和增加资产）：

# min_wt = 10
# max_wt = 50
# step = 5
# nb_assets = 6
7.6620834979985375        # f1 - original code from question
7.31425816299452          # f1b - code from question using list instead of numpy array
0.809070186005556         # f2 - calculating the last element of each tuple
14.88188026699936         # f3 - recursive solution from user kaya3
0.39385621099791024       # f4 - recursive solution from user jdehesa

似乎f2 和f4 是最快的（对于经过测试的数据大小）。

Answer 4

让我们概括一下这个问题；您想遍历总和为 n 且其元素在 range(min_w, max_w+1, w_step) 内的 k 元组。这是一种integer partitioning problem，对分区的大小及其组件的大小有一些额外的限制。

为此，我们可以编写一个递归生成器函数；对于范围内的每个w，元组的其余部分是一个 (k - 1) 元组，其和为 (n - w)。基本情况是一个 0 元组，只有当所需的总和为 0 时才有可能。

正如 Raymond Hettinger 所说，您还可以通过测试所需的总和是否是允许的权重之一来提高 k = 1 时的效率。

def constrained_partitions(n, k, min_w, max_w, w_step=1):
    if k < 0:
        raise ValueError('Number of parts must be at least 0')
    elif k == 0:
        if n == 0:
            yield ()
    elif k == 1:
        if n in range(min_w, max_w+1, w_step):
            yield (n,)
    elif min_w*k <= n <= max_w*k:
        for w in range(min_w, max_w+1, w_step):
            for p in constrained_partitions(n-w, k-1, min_w, max_w, w_step):
                yield (w,) + p

用法：

>>> for p in constrained_partitions(5, 3, 1, 5, 1):
...     print(p)
...
(1, 1, 3)
(1, 2, 2)
(1, 3, 1)
(2, 1, 2)
(2, 2, 1)
(3, 1, 1)
>>> len(list(constrained_partitions(100, 5, 10, 50, 10)))
121

每当您对某种组合问题的所有解决方案进行迭代时，通常最好直接生成实际解决方案，而不是生成超出您需要的解决方案（例如使用product 或combinations_with_replacement）并拒绝您的解决方案不想要。对于较大的输入，由于combinatorial explosion，绝大多数时间将用于生成将被拒绝的解决方案。

请注意，如果您不希望以不同的顺序重复（例如 1, 1, 3 和 1, 3, 1），您可以将递归调用更改为 constrained_partitions(n-w, k-1, min_w, w, w_step) 以仅生成权重非递增顺序的分区。

Answer 5

请注意，当您的 N 个权重总和为 100，并且您选择了 N - 1 个权重时，剩余的权重已定义为 100 - 已选择的权重，应该是正数。同样的限制适用于任何数量的已选择权重。

接下来，您不希望组合只是相同权重的排列。这就是为什么您可以按值排序重量，并选择组合中的下一个重量低于或等于前一个重量。

这会立即使搜索空间变得更小，并且您可以更早地中断特定的搜索分支。

可能首先使用显式循环编写它，或者作为递归算法，应该更容易理解和实现。