使用 Python 读取带有日期对象和浮点数的逗号分隔文件

Question

我有一个文件，其中的条目看起来像

2013-12-11 23:00:27.003293,$PAMWV,291,R,005.8,M,A*36
2013-12-11 23:00:28.000295,$PAMWV,284,R,005.5,M,A*3F
2013-12-11 23:00:29.000295,$PAMWV,273,R,004.0,M,A*33
2013-12-11 23:00:30.003310,$PAMWV,007,R,004.9,M,A*3B

考虑到分隔符实际上是逗号（','），这是一个经典的 CSV 文件。

我试过了：

wind = loadtxt("/disk2/Wind/ws425.log.test", dtype(str,float), delimiter=',', usecols=(0,4))
ts= time.strptime(str(wind[:,0]), '%Y-%m-%d %H:%M:%S.%f')

我得到的是

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-31-484b71dea724> in <module>()
----> 1 ts= time.strptime(str(wind[:,0]), '%Y-%m-%d %H:%M:%S.%f')

/opt/Enthought/canopy/appdata/canopy-1.0.3.1262.rh5-x86_64/lib/python2.7/_strptime.pyc in _strptime_time(data_string, format)
    452 
    453 def _strptime_time(data_string, format="%a %b %d %H:%M:%S %Y"):
--> 454     return _strptime(data_string, format)[0]

/opt/Enthought/canopy/appdata/canopy-1.0.3.1262.rh5-x86_64/lib/python2.7/_strptime.pyc in _strptime(data_string, format)
    323     if not found:
    324         raise ValueError("time data %r does not match format %r" %
--> 325                          (data_string, format))
    326     if len(data_string) != found.end(): 
    327         raise ValueError("unconverted data remains: %s" %

ValueError: time data "['2013-12-12 00:00:02.251311' '2013-12-12 00:00:03.255296'\n     '2013-12-12 00:00:04.254294' ..., '2013-12-12 16:10:50.579022'\n '2013-12-12    16:10:51.607035' '2013-12-12 16:10:52.604020']" does not match format '%Y-%m-%d %H:%M:%S.%f'

我怀疑我在 time.strptime() 中误用了数据类型分配，但到目前为止我一直未能找到正确的类型。

建议？

Answer 1

我不得不做类似的事情

>>> import numpy as np
>>> from datetime import datetime
>>> wind = np.loadtxt("ws425.log.test", delimiter=",", usecols=(0,4), dtype=object,
...                   converters={0: lambda x: datetime.strptime(x, "%Y-%m-%d %H:%M:%S.%f"),
...                               4: np.float})
>>> 
>>> wind
array([[datetime.datetime(2013, 12, 11, 23, 0, 27, 3293), 5.8],
       [datetime.datetime(2013, 12, 11, 23, 0, 28, 295), 5.5],
       [datetime.datetime(2013, 12, 11, 23, 0, 29, 295), 4.0],
       [datetime.datetime(2013, 12, 11, 23, 0, 30, 3310), 4.9]], dtype=object)

---

不过，对于时间序列数据，我改用pandas，因为它让很多事情变得更容易：

>>> import pandas as pd
>>> df = pd.read_csv("ws425.log.test", parse_dates=[0], header=None, usecols=[0, 4])
>>> df
                           0    4
0 2013-12-11 23:00:27.003293  5.8
1 2013-12-11 23:00:28.000295  5.5
2 2013-12-11 23:00:29.000295  4.0
3 2013-12-11 23:00:30.003310  4.9

[4 rows x 2 columns]
>>> df[0][0]
Timestamp('2013-12-11 23:00:27.003293', tz=None)

Answer 2

我不确定 numpy 有什么问题；但使用 csv 效果很好：

>>> import time
>>> import csv
>>> with open('t.txt') as f:
...    r = csv.reader(f)
...    w = [[time.strptime(i[0], '%Y-%m-%d %H:%M:%S.%f')]+i[1:] for i in r]
...
>>> w
[[time.struct_time(tm_year=2013, tm_mon=12, tm_mday=11, tm_hour=23, tm_min=0, tm_sec=27, tm_wday=2, tm_yday=345, tm_isdst=-1), '$PAMWV', '291', 'R', '005.8', 'M', 'A*36'], [time.struct_time(tm_year=2013, tm_mon=12, tm_mday=11, tm_hour=23, tm_min=0, tm_sec=28, tm_wday=2, tm_yday=345, tm_isdst=-1), '$PAMWV', '284', 'R', '005.5', 'M', 'A*3F'], [time.struct_time(tm_year=2013, tm_mon=12, tm_mday=11, tm_hour=23, tm_min=0, tm_sec=29, tm_wday=2, tm_yday=345, tm_isdst=-1), '$PAMWV', '273', 'R', '004.0', 'M', 'A*33'], [time.struct_time(tm_year=2013, tm_mon=12, tm_mday=11, tm_hour=23, tm_min=0, tm_sec=30, tm_wday=2, tm_yday=345, tm_isdst=-1), '$PAMWV', '007', 'R', '004.9', 'M', 'A*3B']]

Answer 3

time.strptime() 需要一个字符串，例如 '2013-12-11 23:00:30.003310'，但您却给它一个数组的字符串表示形式：

['2013-12-12 00:00:02.251311', '2013-12-12 00:00:03.255296', ...]

最小的解决方法是一次解析一个项目：

ts = [time.strptime(s, '%Y-%m-%d %H:%M:%S.%f') for s in wind[:,0]]

或者您可以将converters 参数用于loadtxt：

from datetime import datetime
import numpy as np

def str2timestamp(timestr, epoch=datetime.fromtimestamp(0)):
    """Convert local time string into seconds since epoch (float)."""
    # np.datetime64 API is experimental so use datetime instead
    #NOTE: local time may be ambiguous, non-monotonous
    dt = datetime.strptime(timestr, '%Y-%m-%d %H:%M:%S.%f')
    return (dt - epoch).total_seconds()

wind = np.loadtxt('input.csv', usecols=(0, 4), delimiter=',',
                  converters={0: str2timestamp})
print(wind)

输出

[[  1.38679203e+09   5.80000000e+00]
 [  1.38679203e+09   5.50000000e+00]
 [  1.38679203e+09   4.00000000e+00]
 [  1.38679203e+09   4.90000000e+00]]

Answer 4

您在定义 dtype 的 NumPy loadtxt 调用中只是有一些错误。应该是dtype=[('date', 'str', 26), ('wind', 'float')]；您必须指定字符串的大小。现在您可以使用其名称引用日期字段，EG：wind['date']。您的 strptime 格式很好，但您需要 Python 的 datetime 包中的 datetime 模块，而不是 time。

import numpy as np
from datetime import datetime
wind = loadtxt("/disk2/Wind/ws425.log.test", dtype=[('date', 'str', 26), ('wind', 'float')], delimiter=',', usecols=(0,4))
ts = [datetime.strptime(d, '%Y-%m-%d %H:%M:%S.%f') for d in wind['date']]

这将返回以下内容：

[datetime.datetime(2013, 12, 11, 23, 0, 27, 3293),
 datetime.datetime(2013, 12, 11, 23, 0, 28, 295),
 datetime.datetime(2013, 12, 11, 23, 0, 29, 295),
 datetime.datetime(2013, 12, 11, 23, 0, 30, 3310)]

也许你想把它反馈到你的 NumPy 数组中？

wind['date'] = np.array(ts, dtype='datetime64[s]')

这会产生

array([('2013-12-11T23:00:27Z', 5.8), ('2013-12-11T23:00:28Z', 5.5),
   ('2013-12-11T23:00:29Z', 4.0), ('2013-12-11T23:00:30Z', 4.9)],
  dtype=[('date', 'S26'), ('wind', '<f8')])

Answer 5

哦，这里真正的问题是time.strptime 不支持微秒的 %f，有关 time.strptime 和 time.strftime 支持的格式字符列表，请参阅here。

您想要的是datetime.strptime，它确实支持微秒的 %f 格式字符。