删除两个时间序列中的相同异常值

Question

我有一个关于从两个时间序列中消除异常值的问题。一个时间序列包括现货市场价格，另一个包括电力输出。这两个系列是从 2012 年到 2016 年，都是带有时间戳和值的 CSV 文件。例如功率输出：2012-01-01 00:00:00,2335.2152646951617 和价格：2012-01-01 00:00:00,17.2

因为现货市场价格波动很大，有很多异常值，所以我把它们过滤掉了。对于第二个时间序列，我必须删除具有相同时间戳的值，这些值在价格的时间序列中已被消除。我考虑过生成一个包含已删除值的列表并编写一个循环来删除第二个时间序列中具有相同时间戳的值。但到目前为止，这还没有奏效，我也没有真正开始。有人有想法吗？

我的python代码如下：

import pandas as pd
import matplotlib.pyplot as plt

power_output = pd.read_csv("./data/external/power_output.csv", delimiter=",", parse_dates=[0], index_col=[0])
print(power_output.head())
plt.plot(power_output)

spotmarket = pd.read_csv("./data/external/spotmarket_dhp.csv", delimiter=",", parse_dates=[0], index_col=[0])
print(spotmarket.head())

r = spotmarket['price'].pct_change().dropna() * 100
print(r)
plt.plot(r)

Q1 = r.quantile(.25)
Q3 = r.quantile(.75)
q1 = Q1-2*(Q3-Q1)
q3 = Q3+2*(Q3-Q1)

a = r[r.between(q1, q3)]
print(a)
plt.plot(a)

有人可以帮帮我吗？

Answer 1

如果您的问题是关于如何比较两个时间戳，您可以查看this。

基本上你可以这样做：

out = r[~r.between(q1, q3)] # negation of your between to get the outliers
df=pd.merge(spotmarker,out,on=['date'],how="outer",indicator=True)
df=df[df['_merge']=='left_only']

这是一个合并操作，它只保留那些只存在于左侧数据框中的行

Answer 2

以下建议基于我在previous post 的回答。 您可以通过merging 解决您的两个系列的问题并将它们存储在pandas dataframe 中。然后，您可以使用任何所需的技术来识别和删除异常值。看看上面提到的帖子。

这是我使用可以处理多个系列的 sn-p 对您的特定问题的看法：

---

由于我无权访问您的数据，因此以下 sn-p 将生成两个系列，其中一个具有独特的异常值：

def sample(colname):

    base = 100
    nsample = 20
    sigma = 10

    # Basic df with trend and sinus seasonality 
    trend1 = np.linspace(0,1, nsample)
    y1 = np.sin(trend1)
    dates = pd.date_range(pd.datetime(2016, 1, 1).strftime('%Y-%m-%d'), periods=nsample).tolist()
    df = pd.DataFrame({'dates':dates, 'trend1':trend1, 'y1':y1})
    df = df.set_index(['dates'])
    df.index = pd.to_datetime(df.index)

    # Gaussian Noise with amplitude sigma
    df['y2'] = sigma * np.random.normal(size=nsample)
    df['y3'] = df['y2'] + base + (np.sin(trend1))
    df['trend2'] = 1/(np.cos(trend1)/1.05)
    df['y4'] = df['y3'] * df['trend2']

    df=df['y4'].to_frame()
    df.columns = [colname]

    return(df)

df_sample1 = sample(colname = 'series1')
df_sample2 = sample(colname = 'series2')
df_sample2['series2'].iloc[10] = 800
df_sample1.plot()
df_sample2.plot()

系列 1 - 没有异常值

系列 2 - 一个独特的异常值

现在您可以像这样合并这些系列：

# Merge dataframes
df_merged = pd.merge(df_sample1, df_sample2, how='outer', left_index=True, right_index=True)
df_merged.plot()

什么被认为是异常值将完全取决于您的数据集的性质。在这种情况下，您可以使用sscipy.zscore() 设置识别异常值的级别。在以下情况下，每一个差值超过 3 的观测值都被视为异常值。

# A function for removing outliers
def noSpikes(df, level, keepFirst):

    # 1. Get some info about the original data:

    ##%%
    #df = df_merged
    #level = 3
    #keepFirst = True
    ##%%

    firstVal = df[:1]
    colNames = df.columns
    colNumber = len(df.columns)

    #cleanBy = 'Series1'

    # 2. Take the first difference and 
    df_diff = df.diff()

    # 3. Remove missing values
    df_clean = df_diff.dropna()

    # 4. Select a level for a Z-score to identify and remove outliers
    df_Z = df_clean[(np.abs(stats.zscore(df_clean)) < level).all(axis=1)]
    ix_keep = df_Z.index

    # 5. Subset the raw dataframe with the indexes you'd like to keep
    df_keep = df.loc[ix_keep]

    # 6. 
    # df_keep will be missing some indexes.
    # Do the following if you'd like to keep those indexes
    # and, for example, fill missing values with the previous values
    df_out = pd.merge(df_keep, df, how='outer', left_index=True, right_index=True)

    # 7. Keep only the original columns (drop the diffs)
    df_out = df_out.ix[:,:colNumber]

    # 8. Fill missing values
    df_complete = df_out.fillna(axis=0, method='ffill')

    # 9. Reset column names
    df_complete.columns = colNames

    # Keep the first value
    if keepFirst:
        df_complete.iloc[0] = firstVal.iloc[0]

    return(df_complete)

df_clean = noSpikes(df = df_merged, level = 3, keepFirst = True)
df_clean.plot()

让我知道这对你有什么影响。

---

这是一个简单的复制粘贴的全部内容：

# Imports
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from scipy import stats

np.random.seed(22)

# A function for noisy data with a trend element
def sample(colname):

    base = 100
    nsample = 20
    sigma = 10

    # Basic df with trend and sinus seasonality 
    trend1 = np.linspace(0,1, nsample)
    y1 = np.sin(trend1)
    dates = pd.date_range(pd.datetime(2016, 1, 1).strftime('%Y-%m-%d'), periods=nsample).tolist()
    df = pd.DataFrame({'dates':dates, 'trend1':trend1, 'y1':y1})
    df = df.set_index(['dates'])
    df.index = pd.to_datetime(df.index)

    # Gaussian Noise with amplitude sigma
    df['y2'] = sigma * np.random.normal(size=nsample)
    df['y3'] = df['y2'] + base + (np.sin(trend1))
    df['trend2'] = 1/(np.cos(trend1)/1.05)
    df['y4'] = df['y3'] * df['trend2']

    df=df['y4'].to_frame()
    df.columns = [colname]

    return(df)

df_sample1 = sample(colname = 'series1')
df_sample2 = sample(colname = 'series2')
df_sample2['series2'].iloc[10] = 800
df_sample1.plot()
df_sample2.plot()

# Merge dataframes
df_merged = pd.merge(df_sample1, df_sample2, how='outer', left_index=True, right_index=True)
df_merged.plot()

# A function for removing outliers
def noSpikes(df, level, keepFirst):

    # 1. Get some info about the original data:
    firstVal = df[:1]
    colNames = df.columns
    colNumber = len(df.columns)

    #cleanBy = 'Series1'

    # 2. Take the first difference and 
    df_diff = df.diff()

    # 3. Remove missing values
    df_clean = df_diff.dropna()

    # 4. Select a level for a Z-score to identify and remove outliers
    df_Z = df_clean[(np.abs(stats.zscore(df_clean)) < level).all(axis=1)]
    ix_keep = df_Z.index

    # 5. Subset the raw dataframe with the indexes you'd like to keep
    df_keep = df.loc[ix_keep]

    # 6. 
    # df_keep will be missing some indexes.
    # Do the following if you'd like to keep those indexes
    # and, for example, fill missing values with the previous values
    df_out = pd.merge(df_keep, df, how='outer', left_index=True, right_index=True)

    # 7. Keep only the original columns (drop the diffs)
    df_out = df_out.ix[:,:colNumber]

    # 8. Fill missing values
    df_complete = df_out.fillna(axis=0, method='ffill')

    # 9. Reset column names
    df_complete.columns = colNames

    # Keep the first value
    if keepFirst:
        df_complete.iloc[0] = firstVal.iloc[0]

    return(df_complete)

df_clean = noSpikes(df = df_merged, level = 3, keepFirst = True)
df_clean.plot()