时间序列组之间的相似性

Question

我有一个如下所示的数据集、多个组、超过 200 列的完整值（表示天数）

输入

Series	1	2	3	4	5	6	7	GROUP
01/08/2021	100%	75%	60%	50%	40%	30%	0%	A
08/08/2021	100%	95%	80%	60%	30%	10%	0%	A
15/08/2021	100%	85%	60%	40%	20%	10%	5%	A
01/08/2021	100%	70%	65%	55%	45%	35%	0%	B
08/08/2021	100%	90%	80%	60%	30%	10%	0%	B
15/08/2021	100%	95%	60%	40%	30%	20%	5%	B

现在，我有一个不完整的数据集，如下所示。我想计算每个组的相似度指标，并说明哪个系列最相似。

出于相似的目的，我目前在 Excel 中使用 CORREL，如果出现平局，我使用的是最新的。为了比较，仅比较两组中的完整值（即，预期输出中的缺失值不用于相似性度量计算）。

这是一个 VBA 宏，我正在转换为 python（pandas 或 pyspark）。

我对如何最好地进行感到困惑。也可以尝试任何其他相似性度量。谢谢

预期输出

Series	1	2	3	4	5	6	7	Similarity_Score	Similarity_Week	Group
01/09/2021					39%	28%	0%	0.99	01/08/2021	A
08/09/2021			62%	44%	21%	12%	7%	0.99	15/08/2021	A
15/09/2021						8%	0%	1.00	08/08/2021	A
15/09/2021					30%	19%	0%	1.00	15/08/2021	B

Answer 1

此解决方案涉及对每个组进行迭代，获取每个数据帧的子集并获取每个数据帧值的乘积，以便可以将每一行与其他每一行进行比较。

我们可以使用一些嵌套的 zip/filter/reverse 技巧来仅保留已填写的列。将其放入包含 dfs 和组的日期的列表中，我们可以创建一个数据框、排序、组，并保持每个的最高分。

将它加入第二个 df 应该会给你想要的输出。

import pandas as pd
import numpy as np
from itertools import product

df = pd.DataFrame({'Series': {0: '01/08/2021',
  1: '08/08/2021',
  2: '15/08/2021',
  3: '01/08/2021',
  4: '08/08/2021',
  5: '15/08/2021'},
 '1': {0: '100%', 1: '100%', 2: '100%', 3: '100%', 4: '100%', 5: '100%'},
 '2': {0: '75%', 1: '95%', 2: '85%', 3: '70%', 4: '90%', 5: '95%'},
 '3': {0: '60%', 1: '80%', 2: '60%', 3: '65%', 4: '80%', 5: '60%'},
 '4': {0: '50%', 1: '60%', 2: '40%', 3: '55%', 4: '60%', 5: '40%'},
 '5': {0: '40%', 1: '30%', 2: '20%', 3: '45%', 4: '30%', 5: '30%'},
 '6': {0: '30%', 1: '10%', 2: '10%', 3: '35%', 4: '10%', 5: '20%'},
 '7': {0: '0%', 1: '0%', 2: '5%', 3: '0%', 4: '0%', 5: '5%'},
 'GROUP': {0: 'A', 1: 'A', 2: 'A', 3: 'B', 4: 'B', 5: 'B'}})

df2 = pd.DataFrame({'Series': {0: '01/09/2021',
  1: '08/09/2021',
  2: '15/09/2021',
  3: '15/09/2021'},
 '1': {0: np.nan, 1: np.nan, 2: np.nan, 3: np.nan},
 '2': {0: np.nan, 1: np.nan, 2: np.nan, 3: np.nan},
 '3': {0: np.nan, 1: '62%', 2: np.nan, 3: np.nan},
 '4': {0: np.nan, 1: '44%', 2: np.nan, 3: np.nan},
 '5': {0: '39%', 1: '21%', 2: np.nan, 3: '30%'},
 '6': {0: '28%', 1: '12%', 2: '8%', 3: '19%'},
 '7': {0: '0%', 1: '7%', 2: '0%', 3: '0%'},
 'Similarity_Score': {0: 0.99, 1: 0.99, 2: 1.0, 3: 1.0},
 'Similarity_Week': {0: '01/08/2021',
  1: '15/08/2021',
  2: '08/08/2021',
  3: '15/08/2021'},
 'Group': {0: 'A', 1: 'A', 2: 'A', 3: 'B'}}
)

df2.drop(columns=['Similarity_Score','Similarity_Week'], inplace=True)


l = []
for g, data in df.groupby('GROUP'):
    x = df2.loc[df2['Group']==g]
    for c in product(data.values,x.values):
        a = c[0][1:-1]
        b = c[1][1:-1]
        a,b = list(zip(*(zip(reversed(a),list(filter(lambda v: v==v, b))))))
        a = [int(x.replace('%',''))/100 for x in a]
        b = list(reversed([int(x.replace('%',''))/100 for x in b]))
        l.append([g,c[0][0],c[1][0], np.corrcoef(a,b)[1,0]])

out = df2.merge(pd.DataFrame(l, columns=['Group','Similarity_Week','Series','Similarity_Score']).sort_values(by=['Similarity_Score', 'Similarity_Week'], ascending=False).groupby(['Group','Series']).head(1), on=['Group','Series'])

输出

       Series   1   2    3    4    5    6   7 Group Similarity_Week  \
0  01/09/2021 NaN NaN  NaN  NaN  39%  28%  0%     A      01/08/2021   
1  08/09/2021 NaN NaN  62%  44%  21%  12%  7%     A      15/08/2021   
2  15/09/2021 NaN NaN  NaN  NaN  NaN   8%  0%     A      01/08/2021   
3  15/09/2021 NaN NaN  NaN  NaN  30%  19%  0%     B      15/08/2021   

   Similarity_Score  
0          0.999405  
1          0.999005  
2          1.000000  
3          0.999286 

我相信 2021 年 9 月 15 日 A 组的分数非常相似，因此，如果您对分数进行四舍五入，您会得到不同的最近日期。您可以通过检查来验证这一点

[x for x in l if x[2]=='15/09/2021' and x[0]=='A']

产量

[['A', '01/08/2021', '15/09/2021', 1.0],
 ['A', '08/08/2021', '15/09/2021', 0.9999999999999998],
 ['A', '15/08/2021', '15/09/2021', 0.9999999999999998]]

所以理论上 15/08/2021 将是日期，如果您四舍五入到小数点后几位，您可以通过将 round() 放在 np.corrcoef 周围来做到这一点

Answer 2

如果您更喜欢没有for 循环的解决方案，您可以merge Group 上的两个数据框，并使用groupby 应用相似度指标。

以@Chris 构建的数据框为基础：

df.rename(columns={"GROUP":"Group"}, inplace=True)

def similarity(arr1, arr2):
        """Similarity between two arrays of percent strings, nans ignored"""

    df = pd.DataFrame({"arr1":arr1, "arr2":arr2}).dropna() \
            .apply(lambda s: s.str.strip("%").astype(float)/100)
    return df.arr1.corr(df.arr2)

# Convert data columns to array in each row.
df_xformed = df.set_index(["Series", "Group"]).apply(pd.Series.to_numpy, axis=1) \
                .reset_index().rename(columns={"Series":"df_Series", 0:"df"})
df2_xformed = df2.set_index(["Series", "Group"]).apply(pd.Series.to_numpy, axis=1) \
                .reset_index().rename(columns={"Series":"df2_Series", 0:"df2"})

# Merge on Group and calculate similarities.
df_combined = df_xformed.merge(df2_xformed, on="Group")
df_combined["similarity"] = df_combined.apply(
    lambda row: similarity(row["df"], row["df2"]), axis=1)

# Find max similarity of each df2_Series within its Group.
df_combined["df2_sim_max"] = df_combined.groupby(\
                                 ["df2_Series", "Group"])["similarity"] \
                                .transform(max)

idx = df_combined["similarity"] == df_combined["df2_sim_max"]
result = df_combined[idx][["df2_Series", "Group", "df2", "df_Series", "similarity"]]
result
#     df2_Series Group  ...   df_Series similarity
# 0   01/09/2021     A  ...  01/08/2021   0.999405
# 2   15/09/2021     A  ...  01/08/2021   1.000000
# 7   08/09/2021     A  ...  15/08/2021   0.999005
# 11  15/09/2021     B  ...  15/08/2021   0.999286