查找同时出现在多个列表中的项目

Question

我正在尝试编写一些高效的 p​​ython 代码来识别同时出现在多个列表中的项目。例如，在列表字典中

list_of_lists = {'lista':list('abcdefhmqr'),
                 'listb':list('abdgklmr'),
                 'listc':list('abcdgjkmr'),
                 'listd':list('abcdglmrt'),
                 'liste':list('admoprst')}

“adrm”一起出现在所有五个列表中，而“abdm”、“abdr”、“abmr”和“bdmr”一起出现在四个列表中，四个字母的许多组合出现在三个或两个列表中。

代码如下：

def make_dict(lists):
    # creates a dictionary with each unique item as the key,
    # and a set of lists the item appears on as the value
    letter_dict={}
    for item in lists.items():
        for letter in item[1]:
            if letter in letter_dict:
                letter_dict[letter].add(item[0])
            else:
                letter_dict[letter] = set([item[0]])
    return OrderedDict(sorted(letter_dict.items(),key=lambda x:x[0]))

def find_matches(dictionary):
    # takes a dictionary with tuples of list elements as keys
    # and lists they appear on as values, and finds the intersection with 
    # the master list of elements and their lists
    matches={}
    for key in dictionary.keys():
        index_of_key = index_of_attr_keys.index(key[-1])
        for next_key in islice(master_list,index_of_key+1,None):
            intersection = dictionary[key] & master_list[next_key]
            if len(intersection)>1:
                new_key = set(key)
                new_key.add(next_key[0])
                new_key = tuple(sorted(new_key))
                matches[new_key] = intersection
    return matches

master_list = make_dict(list_of_lists)
index_of_attr_keys = sorted(master_list.keys())

我可以迭代地制作带有两个、三个、四个等项的元组键的字典

doubles = find_matches(master_list)
triples = find_matches(doubles)
quads = find_matches(triples)

我的代码适用于这个玩具示例，但是当我在我的实际数据集上运行它时它并不是特别快，该数据集包含出现在数百个列表中的 84,000 多个唯一元素。从我的 84,000 多个独特元素列表开始，生成一个包含 120 万对的列表需要一个小时，这些对一起出现在多个列表中，并且事情会变得更长。我想知道是否有更快的方法来做到这一点。

Answer 1

转换为set 并在您想要的列表之间进行交集。例如查找出现在所有列表中的所有元素：

set.intersection(*[set(x) for x in list_of_lists.values()])

输出：

 {'a', 'd', 'm', 'r'}

Answer 2

我将稍微改一下你的问题。我对此的解读是，您在键上的 powerset 中寻找数据集的交集。

下面的代码生成你的集合的幂集，在你的数据上运行交叉点，当它遇到一个空交叉点时在那个路径上跳伞。你的 powerset 中的叶子总数是 2^n，所以如果你有大量重叠的集合，它会变得粗糙。另一方面，您的交叉点通常很小，或者很快变得很小，那么只需要访问整个 2^n 宽树的一小部分。

您还可以通过表达您想要的最小集合大小并将其用作“if 0==len(intersection)”行上的保释选项来显着缩小搜索空间，这可能会有很大帮助，但这又取决于在您的基础数据上。

list_of_lists = {'lista':list('abcdefhmqr'),
                 'listb':list('abdgklmr'),
                 'listc':list('abcdgjkmr'),
                 'listd':list('abcdglmrt'),
                 'liste':list('admoprst'),
                 'listz':list('z'),
                 'listza':list('az')}

dict_of_sets = {k: set(v) for k, v in list_of_lists.items()}


def do_something_with_intersection(dict_of_sets, curr_keyset, curr_intersection):
    print("Intersection: " + str(curr_intersection) + " in sets: " + str(curr_keyset))

# Remaining_keys is the list of keys in dict_of_sets that are still unvisited
# dict_of_sets is all the underlying data
# curr_keyset is the current set of keys [drawn on dict_of_sets] - function visits the power set of those keys
# curr_intersection is the intersections of sets from dict_of_sets whose keys are in curr_set
def do_powerset_intersection(remaining_keys, dict_of_sets, curr_keyset=set(), curr_intersection=None):
    # Hit the end of the recursive adventure
    if 0 == len(remaining_keys):
        if 0 != len(curr_keyset):
            print("Found a Leaf: " + str(curr_keyset))
            do_something_with_intersection(dict_of_sets, curr_keyset, curr_intersection)
            print()
        return
    q = remaining_keys.pop()
    if curr_intersection is None:
        new_intersection = dict_of_sets[q]
    else:
        new_intersection = curr_intersection.intersection(dict_of_sets[q])

    if 0 == len(new_intersection):
        # Ran out of stuff to intersect - curr_intersection is best we can do on this path
        print("Didn't make it to a leaf, best we can do")
        do_something_with_intersection(dict_of_sets, curr_keyset, curr_intersection)
        print()
        return

    new_set = curr_keyset.copy()
    new_set.add(q)
    do_powerset_intersection(remaining_keys, dict_of_sets, curr_keyset, curr_intersection)
    do_powerset_intersection(remaining_keys, dict_of_sets, new_set, new_intersection)
    remaining_keys.append(q)


do_powerset_intersection(list(dict_of_sets.keys()), dict_of_sets)