如何对元组列表（标签，概率）的列表进行聚类？ - Python答案

【问题标题】：How can I cluster a list of a list of tuple (tag, probability)? - python如何对元组列表（标签，概率）的列表进行聚类？ - Python
【发布时间】：2014-01-26 06:02:32
【问题描述】：

我有一堆文本，它们被分类，然后每个文档被标记为 0、1 或 2，每个标记的概率。

[ "this is a foo bar",
  "bar bar black sheep",
  "sheep is an animal"
  "foo foo bar bar"
  "bar bar sheep sheep" ]

管道中的前一个工具返回一个元组列表列表，外部列表中的每个元素都是一个文档。我只能处理这样一个事实，即我知道每个文档都被标记为 0、1 或 2 以及它们的概率：

[ [(0,0.3), (1,0.5), (2,0.1)],
  [(0,0.5), (1,0.3), (2,0.3)],
  [(0,0.4), (1,0.4), (2,0.5)],
  [(0,0.3), (1,0.7), (2,0.2)],
  [(0,0.2), (1,0.6), (2,0.1)] ]

我需要它来查看每个元组列表中的哪个标签最有可能并实现：

[ [[(0,0.5), (1,0.3), (2,0.3)], [(0,0.4), (1,0.4), (2,0.5)]] ,
  [[(0,0.3), (1,0.7), (2,0.2)], [(0,0.2), (1,0.6), (2,0.1)]] ,
  [[(0,0.4), (1,0.4), (2,0.5)]] ]

再举一个例子：

[in]:

[ [(0,0.7), (1,0.2), (2,0.4)],
  [(0,0.5), (1,0.9), (2,0.3)],
  [(0,0.3), (1,0.8), (2,0.4)],
  [(0,0.8), (1,0.2), (2,0.2)],
  [(0,0.1), (1,0.7), (2,0.5)] ]

[out]:

 [[[(0,0.7), (1,0.2), (2,0.4)],
 [(0,0.8), (1,0.2), (2,0.2)]] ,

 [[(0,0.5), (1,0.9), (2,0.3)],
 [(0,0.1), (1,0.7), (2,0.5)],
 [(0,0.3), (1,0.8), (2,0.4)]] ,

 []]

注意：当数据进入我的管道部分时，我确实不有权访问原始文本。

如何使用标签和概率对元组列表的列表进行聚类？ numpy、scipy、sklearn 或任何支持 python 的 ML 套件中是否有东西可以做到这一点？甚至NLTK。

假设集群的数量是固定的，但集群的大小不是。

我只尝试找到质心的最大值，但这只给了我每个集群中的第一个值：

instream = [ [(0,0.3), (1,0.5), (2,0.1)],
                        [(0,0.5), (1,0.3), (2,0.3)],
                        [(0,0.4), (1,0.4), (2,0.5)],
                        [(0,0.3), (1,0.7), (2,0.2)],
                        [(0,0.2), (1,0.6), (2,0.1)] ]

# Find centroid.  
c1_centroid_value = sorted([i[0] for i in instream], reverse=True)[0]
c2_centroid_value = sorted([i[1] for i in instream], reverse=True)[0]
c3_centroid_value = sorted([i[2] for i in instream], reverse=True)[0]

c1_centroid = [i for i,j in enumerate(instream) if j[0] == c1_centroid_value][0]
c2_centroid = [i for i,j in enumerate(instream) if j[1] == c2_centroid_value][0]
c3_centroid = [i for i,j in enumerate(instream) if j[2] == c3_centroid_value][0]

print instream[c1_centroid]
print instream[c2_centroid]
print instream[c2_centroid]

[out]（每个簇中的顶部元素：

[(0, 0.5), (1, 0.3), (2, 0.3)]
[(0, 0.3), (1, 0.7), (2, 0.2)]
[(0, 0.3), (1, 0.7), (2, 0.2)]

【问题讨论】：

如果您可以展示一些输入/输出的示例，将会有所帮助。并且只是更多地解释你到底想要做什么 - 确保它不是XY Problem。
@InbarRose，我已经编辑了问题以提供更多背景信息。
你out的第三行不应该是[(0,0.4), (1,0.4), (2,0.5)]吗？
错字，它是一个列表输出列表，但很容易操作 =)
所以，如果我理解正确，您的输入是元组列表的列表。每个内部列表有 3 个元组，每个元组有 2 个项目，第一个项目将是 0,1 或 2。第二个项目是概率（浮点数）您想要获得与每个项目的最大概率相对应的列表元组的第一项（0,1 和 2）？

标签： python numpy machine-learning scikit-learn

【解决方案1】：

如果我理解正确，这就是你想要的。

import numpy as np

N_TYPES = 3

instream = [ [(0,0.3), (1,0.5), (2,0.1)],
             [(0,0.5), (1,0.3), (2,0.3)],
             [(0,0.4), (1,0.4), (2,0.5)],
             [(0,0.3), (1,0.7), (2,0.2)],
             [(0,0.2), (1,0.6), (2,0.1)] ]
instream = np.array(instream)

# this removes document tags because we only consider probabilities here
values = [map(lambda x: x[1], doc) for doc in instream]

# determine the cluster of each document by using maximum probability
belongs_to = map(lambda x: np.argmax(x), values)
belongs_to = np.array(belongs_to)

# construct clusters of indices to your instream
cluster_indices = [(belongs_to == k).nonzero()[0] for k in range(N_TYPES)]

# apply the indices to obtain full output
out = [instream[cluster_indices[k]].tolist() for k in range(N_TYPES)]

输出out:

[[[[0.0, 0.5], [1.0, 0.3], [2.0, 0.3]]],

 [[[0.0, 0.3], [1.0, 0.5], [2.0, 0.1]],
  [[0.0, 0.3], [1.0, 0.7], [2.0, 0.2]],
  [[0.0, 0.2], [1.0, 0.6], [2.0, 0.1]]],

 [[[0.0, 0.4], [1.0, 0.4], [2.0, 0.5]]]]

我使用了numpy 数组，因为它们可以很好地搜索和索引。例如，表达式(belongs_to == 1).nonzero()[0] 将索引数组返回到数组belongs_to，其中值为1。索引示例为instream[cluster_indices[2]]。

【讨论】：

【解决方案2】：

为什么要将索引保留在元组中？ 0、1 和 2 是多余的，如果我理解正确，请不要提供任何信息。只需将n_samples x 3 概率数组提供给任何 scikit-learn 算法即可。或者，如果您只想要最有可能的标签分配，请执行np.argmax(X, axis=1)。

【讨论】：