如何使用 Scikit-Learn 和 Python 找到最佳集群数量

Question

我正在使用 Pythons scikit-learn lib 学习集群，但我找不到找到最佳集群数量的方法。我试图列出集群数量并将其传递给for loop，并查看elbow，但我想找到更好的解决方案。这种方法只有在我为range(1,11) 执行此操作时才有效，因为那条线变得非常平滑并且我看不到elbow。 我试过silhouette_score，但我得到的值非常低，有时是负数。

另外，我使用文本数据，我写了几个可以*（比如说）分组的句子，我有关于房子/家庭、学习、聚会、食物的句子......

由于我使用文本数据，我是否有可能获得较低的 silhouette_score 值，我还需要在 cv.fit_transform(doc) 之后缩放数据吗？

有没有更好的方法，也许一些函数会返回最佳集群数的integer 值？例如 1,2,3,4....n

这是我写的代码：

import sklearn.metrics as sm

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.preprocessing import scale
from sklearn.cluster import KMeans, SpectralClustering, MiniBatchKMeans
from sklearn.metrics import silhouette_samples, silhouette_score

import matplotlib.pyplot as plt

doc = ['this is very good show' , 'i had a great time on my school trip', 'such a boring movie', 'Springbreak was amazing', 'You are wrong', 'let s go to the beach', 'how can we do this',
     'i love this product', 'this is an amazing item', 'this food is delicious', 'I had a great time last night', 'thats right', ' lets go to the party', 'we were at the party last night', 
     'this is my favourite restaurant, I love their food, its so good','i love healty food', 'skiing is the best sport', 'what is this', 'this product has a lot of bugs', "i'm on the road again", 
     'I love basketball, its very dynamic' , 'its a shame that you missed the trip, it was amazing', 'Party last night was so boring', 'lets go on road trip', 'this is my home, im living there for 26 years',
     'such a nice song' , 'this is the best movie ever', 'hawaii is the best place for trip','how that happened','This is my favourite band', 'true love', 'party was great','home sweet home',
     'I cant believe that you did that', 'Why are you doing that, I do not get it', 'this is tasty', 'this song is amazing', 'this food is tasty', 'lets go to the cinema', 'lets get together at my house',
     'I need to study for the test', 'I cant go out this weekend', 'I had a great time last night', 'I went out last night and it was amazing', 'you are beautiful', 'we crashed the party',
     'this is the best song i have ever heard', 'i love listening to music', 'music is my life', 'this song is terrible', 'how was your hollyday', 'i do not understand you, I have told you that last night',
      'I know whats best for you', 'I m on collage now', 'this is my favourite subject', 'math is fun', 'i love to study maths', 'programming is my live', 'i need to study, my final exam is tomorrow',
      'i m cooming home', 'i need to clean my house', 'what do you thing about last night', 'lets go out, my house is a mess', 'Im staying at home tonight', 'love is such a beautiful word',
      'i want to buy new house for me and my family', 'im will be home in a couple of hours', 'im working on a science project', 'working is hard and i need to work', 'you need to find a job',
       'this is bad, and we cant do anything about that', 'real estate market is growing', 'im selling my appartment', 'i live at the appartment above', 'i m into real estate', 'prices are going down',
       'i m building house of cards', 'I feel so tired, i was studying all nigh long', 'i was playing piano for more than 10 years and I was pretty good at it','I have never done that in my life',
       'i will buy this product in a couple of days', 'i m buying new phone next month', 'my home is near by', 'i m living in my home', 'i live in my parents house', 'i m living in my appartment',
       'my phone is very slow', 'do you know password for wifi', 'wifi is short for wireless network', 'you are so funny', 'my neighbours are horrible', 'such a nice phone, im glad to have it',
       'last time we went into that club and it was so boring', 'if I were you, i would never said that', 'you done very good work, your boss is very proud of you', 'Overall, I like this place a lot',
       'I was spending money on wrong things', 'whats the price for this item', 'where can I buy it', 'is it for sale', 'This hole in the wall has great Mexican street tacos, and friendly staff'
       'The movie showed a lot of Florida at it s best, made it look very appealing', 'This short film certainly pulls no punches', 'This is the kind of money that is wasted properly',
       'Not only did it only confirm that the film would be unfunny and generic, but it also managed to give away the ENTIRE movie', 'But it s just not funny','you have already done that',
       'I especially liked the non-cliche choices with the parents', 'it was well-paced and suited its relatively short run time']


cv = TfidfVectorizer(analyzer = 'word', max_features = 4000, lowercase=True, preprocessor=None, tokenizer=None, stop_words = 'english')  
x = cv.fit_transform(doc)

my_list = []
for i in range(1,10):

    kmeans = KMeans(n_clusters = i, init = 'k-means++', random_state = 0)
    kmeans.fit(x)
    my_list.append(kmeans.inertia_)
    silhouette_avg = silhouette_score(x, cluster_labels)
    print(silhouette_avg)

plt.plot(range(1,10),my_list)
plt.show()

Answer 1

一般来说，找到最佳聚类数是一个难题，因为它没有唯一的解决方案，而且这个问题也不是确定性的（尤其是对于文本数据）。此外，聚类问题的最优解是在你使用的模型后面优化给定度量的局部最优解，它存在大量clustering models。

因此，自动学习文本数据“正确”数量的聚类的基线是所谓的分层狄利克雷过程 (HDP)，它推广了潜在狄利克雷分配 (LDA) 模型。

您可以在gensim library 中找到 HDP 的示例和用例。