【问题标题】:Average a minute-wise time-series dataset monthly with Panda使用 Panda 每月平均每分钟的时间序列数据集
【发布时间】:2019-10-13 18:09:52
【问题描述】:

我有一个非常大的分钟时间序列数据集(3 个月),格式如下

datetime,val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12
1/06/2017 0:00,0,0,0,0,0,0,0,0,0,0.011,0,0.036
1/06/2017 0:01,0,0,0,0,0,0,0,0,0,0.011,0,0.036
...
1/06/2017 23:59,0,0,0,0,0,0,0,0,0,0.011,0,0.035
2/06/2017 0:00,0,0,0,0,0,0,0,0,0,0.014,0,0.036
2/06/2017 0:01,0,0,0,0,0,0,0,0,0,0.011,0,0.036
...
2/06/2017 23:59,0,0,0,0,0,0,0,0,0,0.011,0,0.035
....
31/08/2017 0:00,0,0.2,0,0,0,0.56,0,0,0,0.014,0,0.036
31/08/2017 0:01,0,0.23,0,0,0,0,0,0,0,0.011,0,0.032
...
31/08/2017 23:59,0,0,0,0,0,0,.55,0,0,0.011,0,0.034

使用 panda 获取每个月每个列的平均值的最有效方法是什么? 预期输出将是

month,val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12
06/2017,0,0,0,0,0,0,0,0,0,0.011,0,0.036
07/2017,0,0,0,0,0,0,0,0,0,0.014,0,0.036
08/2017,0,0,0.21,0,0,0,0,0.52,0,0.011,0,0.036

目前我正在做的是每天读取数据集,然后获取累积的天数数据集,然后将其除以每月的天数。但这非常低效,而且需要很多时间。

【问题讨论】:

    标签: python pandas dataframe time-series


    【解决方案1】:

    Pandas read_csv 和 to_csv 是您所需要的:

    df = pd.read_csv('input.csv', parse_dates=['datetime'])
    df.groupby(df.datetime.dt.strftime('%m/%Y')).mean().rename_axis('month').to_csv(out, float_format='%.06f')
    

    使用您的输入数据(从...过滤)它给出:

    month,val1,val2,val3,val4,val5,val6,val7,val8,val9,val10,val11,val12
    01/2017,0,0.000000,0,0,0,0.000000,0.000000,0,0,0.011000,0,0.035667
    02/2017,0,0.000000,0,0,0,0.000000,0.000000,0,0,0.012000,0,0.035667
    08/2017,0,0.143333,0,0,0,0.186667,0.183333,0,0,0.012000,0,0.034000
    

    【讨论】:

      【解决方案2】:

      首先将列转换为to_datetime,然后将DataFrame.resample 转换为MS 用于月份的开始,最后将DatetimeIndex 的格式更改为MM/YYY 的DatetimeIndex.strftime:

      df['datetime'] = pd.to_datetime(df['datetime'], format='%d/%m/%Y %H:%M')
      
      df = df.resample('MS', on='datetime').mean()
      df.index = df.index.strftime('%m/%Y')
      print (df)
               val1      val2  val3  val4  val5      val6      val7  val8  val9  \
      06/2017   0.0  0.000000   0.0   0.0   0.0  0.000000  0.000000   0.0   0.0   
      07/2017   NaN       NaN   NaN   NaN   NaN       NaN       NaN   NaN   NaN   
      08/2017   0.0  0.143333   0.0   0.0   0.0  0.186667  0.183333   0.0   0.0   
      
                val10  val11     val12  
      06/2017  0.0115    0.0  0.035667  
      07/2017     NaN    NaN       NaN  
      08/2017  0.0120    0.0  0.034000  
      

      或者通过Series.dt.strftime 将转换后的日期时间列传递给groupby 并聚合mean:

      df = df.groupby(df['datetime'].dt.strftime('%m/%Y')).mean()
      print (df)
                val1      val2  val3  val4  val5      val6      val7  val8  val9  \
      datetime                                                                     
      06/2017      0  0.000000     0     0     0  0.000000  0.000000     0     0   
      08/2017      0  0.143333     0     0     0  0.186667  0.183333     0     0   
      
                 val10  val11     val12  
      datetime                           
      06/2017   0.0115      0  0.035667  
      08/2017   0.0120      0  0.034000  
      

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 2016-06-29
        • 2012-09-02
        • 2018-10-15
        • 1970-01-01
        • 2017-11-12
        • 2018-06-02
        • 2019-07-03
        相关资源
        最近更新 更多