您如何使用 scikit-learn 中的惯性计算差距统计的标准偏差？

Question

我正在尝试使用 scikit-learn 库计算差距统计数据，以确定 k-means 的最佳 k。 为了明确地确定正确k 的值，我相信我需要从每个后续间隙中减去标准偏差并检查以查看if gap(k) >= gap(k+1) - std(k+1)。我不明白如何确定这个标准差的值。

谢谢！

这是我的代码：

import pandas as pd
import numpy as np
from sklearn.cluster import KMeans, MiniBatchKMeans
from numpy.random import random_sample
from math import sqrt, log

# returns series of random values sampled between min and max values of passed col
def get_rand_data(col):
    rng = col.max() - col.min()
    return pd.Series(random_sample(len(col))*rng + col.min())

def iter_kmeans(df, n_clusters, num_iters=10):
    rng =  range(1, num_iters + 1)
    vals = pd.Series(index=rng)
    for i in rng:
        k = KMeans(n_clusters=n_clusters, n_init=3)
        k.fit(df)
        print "Ref k: %s" % k.get_params()['n_clusters']
        vals[i] = k.inertia_
    return vals

def gap_statistic(df, max_k=15):
    gaps = pd.Series(index = range(1, max_k + 1))
    for k in range(1, max_k + 1):
        km_act = KMeans(n_clusters=k, n_init=3)
        km_act.fit(df)

        # get ref dataset
        ref = df.apply(get_rand_data)
        ref_inertia = iter_kmeans(ref, n_clusters=k).mean()

        gap = log(ref_inertia - km_act.inertia_)
        print "Ref: %s   Act: %s  Gap: %s" % ( ref_inertia, km_act.inertia_, gap)
        gaps[k] = gap

    return(gaps)

Answer 1

计算标准差iter_kmeans(ref, n_clusters=k) 并乘以sqrt(1 + 1 / num_iters)。详情见原论文：https://web.stanford.edu/~hastie/Papers/gap.pdf