使用 datetime 查找 csv 文件中事件之间的时间差

Question

我有一长串天气变量。行成对出现，第一行表示事件的开始时间，而第二行表示事件的结束时间。数据文件为csv。下面是一个数据示例，有 6 行 6 行，代表 3 对夫妇（因此是 3 个事件）。有数千行数据，因此可能存在一些例外情况（未与其他日期配对的日期）。

hd,Station Number,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Local time,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Local standard time,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Universal coordinated time,Precipitation since last (AWS) observation in mm,Quality of precipitation since last (AWS) observation value,Air Temperature in degrees Celsius,Quality of air temperature,Air temperature (1-minute maximum) in degrees Celsius,Quality of air temperature (1-minute maximum),Air temperature (1-minute minimum) in degrees Celsius,Quality of air temperature (1-minute minimum),Wet bulb temperature in degrees Celsius,Quality of Wet bulb temperature,Wet bulb temperature (1 minute maximum) in degrees Celsius,Quality of wet bulb temperature (1 minute maximum),Wet bulb temperature (1 minute minimum) in degrees Celsius,Quality of wet bulb temperature (1 minute minimum),Dew point temperature in degrees Celsius,Quality of dew point temperature,Dew point temperature (1-minute maximum) in degrees Celsius,Quality of Dew point Temperature (1-minute maximum),Dew point temperature (1 minute minimum) in degrees Celsius,Quality of Dew point Temperature (1 minute minimum),Relative humidity in percentage %,Quality of relative humidity,Relative humidity (1 minute maximum) in percentage %,Quality of relative humidity (1 minute maximum),Relative humidity (1 minute minimum) in percentage %,Quality of Relative humidity (1 minute minimum),Wind (1 minute) speed in km/h,Wind (1 minute) speed quality,Minimum wind speed (over 1 minute) in km/h,Minimum wind speed (over 1 minute) quality,Wind (1 minute) direction in degrees true,Wind (1 minute) direction quality,Standard deviation of wind (1 minute),Standard deviation of wind (1 minute) direction quality,Maximum wind gust (over 1 minute) in km/h,Maximum wind gust (over 1 minute) quality,Visibility (automatic - one minute data) in km,Quality of visibility (automatic - one minute data),Mean sea level pressure in hPa,Quality of mean sea level pressure,Station level pressure in hPa,Quality of station level pressure,QNH pressure in hPa,Quality of QNH pressure,#
hd,40842,2000,3,21,12,32,2000,3,21,12,32,2000,3,21,12,32,0,N,21,N,21,N,21,N,20.4,N,20.4,N,20.4,N,20.1,N,20.1,N,20.1,N,94,N,,N,,N,0,N,0,N,91,N,22,N,0,N,,N,1016.5,N,1015.4,N,1016.5,N,#
hd,40842,2000,3,21,12,50,2000,3,21,12,50,2000,3,21,12,50,0.2,N,20.9,N,20.9,N,20.9,N,20.4,N,20.5,N,20.4,N,20.2,N,20.3,N,20.2,N,95,N,,N,,N,0,N,0,N,42,N,2,N,0,N,,N,1016.6,N,1015.5,N,1016.6,N,#
hd,40842,2000,3,22,13,39,2000,3,22,13,39,2000,3,22,13,39,0,N,20.4,N,20.4,N,20.4,N,20.2,N,20.2,N,20.1,N,20.1,N,20.1,N,19.9,N,98,N,,N,,N,8,N,5,N,10,N,9,N,11,N,,N,1013.3,N,1012.2,N,1013.3,N,#
hd,40842,2000,3,22,13,47,2000,3,22,13,47,2000,3,22,13,47,0,N,20.5,N,20.5,N,20.5,N,20.2,N,20.2,N,20.2,N,20.1,N,20.1,N,20,N,97,N,,N,,N,4,N,0,N,56,N,75,N,5,N,,N,1013.2,N,1012.1,N,1013.2,N,#
hd,40842,2000,3,23,11,0,2000,3,23,11,0,2000,3,23,11,0,0,N,23.4,N,23.4,N,23.3,N,21.3,N,21.4,N,21.3,N,20.2,N,20.3,N,20.2,N,82,N,,N,,N,8,N,5,N,66,N,2,N,9,N,,N,1013.6,N,1012.5,N,1013.6,N,#
hd,40842,2000,3,23,13,27,2000,3,23,13,27,2000,3,23,13,27,0,N,23.3,N,23.3,N,23.3,N,21.6,N,21.6,N,21.5,N,20.7,N,20.7,N,20.6,N,85,N,,N,,N,5,N,4,N,108,N,5,N,5,N,,N,1013.5,N,1012.4,N,1013.5,N,#

就我的分析而言，我只想考虑持续时间至少为两个小时的事件。例如，在上面的数据中，只有最后几行会写入新的 csv 文件（格式相同）。是否有捷径可寻？到目前为止，我已经制作了可能能够做到这一点的代码的基本框架，但我不确定如何在这个例子中实现 datetime。我不确定的部分遵循主题标签。

import csv
import datetime

with open("X:\weatherresults\sea_breeze_dates_2000_2011.csv", "rb") as input, open("X:\weatherresults\sea_breeze_dates_2000_2011_time_filtered.csv", "wb") as wanted:
    reader = csv.DictReader(input, delimiter=",", skipinitialspace=True)
    fieldnames = reader.fieldnames
    writer_wanted = csv.DictWriter(wanted, fieldnames, delimiter=",")
    prev_row = None
    for line_number, row in enumerate(reader):
        try:
            dt = datetime.date(year=int(row["Year Month Day Hour Minutes in YYYY"]), month=int(row["MM"]), day=int(row["DD"])), hour=int(row["HH24"])
            dta = prev_row['dt']            
            if prev_row is not None and dt > prev_row['dt'] and # (dt-dta) => 2hours:
                writer_wanted.writerow(prev_row['row'])
                writer_wanted.writerow(row)
            prev_row = {'row':row, 'dt':dt}
        except:
            print "Failed to parse line", line_number
            print row

try 命令中的 if 子句是困扰我的部分。如何表示事件的开始和结束之间的差必须至少两个小时才能将该行对写入新的 csv 文件？

Answer 1

使用datetime.timedelta 表示最小两小时时间增量。由于您想比较事件的开始时间和结束时间，您可以一次读取两行并跳过跟踪前一行。

import csv
import datetime

minimum_delta = datetime.timedelta(hours=2)

input_fn = r"X:\weatherresults\sea_breeze_dates_2000_2011.csv"
wanted_fn = r"X:\weatherresults\sea_breeze_dates_2000_2011_time_filtered.csv"

with open(input_fn, "rb") as input, open(wanted_fn, "wb") as wanted:
    reader = csv.DictReader(input, delimiter=",", skipinitialspace=True)
    fieldnames = reader.fieldnames
    writer_wanted = csv.DictWriter(wanted, fieldnames, delimiter=",")
    try:
        while 1:
            try:
                begin = next(reader)
                begin_dt = datetime.datetime(
                    year=int(begin["Year Month Day Hour Minutes in YYYY"]),
                    month=int(begin["MM"]), day=int(begin["DD"]), 
                    hour=int(begin["HH24"]))
                end = next(reader)
                end_dt = datetime.datetime(
                    year=int(end["Year Month Day Hour Minutes in YYYY"]),
                    month=int(end["MM"]), day=int(end["DD"]), 
                    hour=int(end["HH24"]))
                if end_dt - begin_dt > minimum_delta:
                    writer_wanted.writerow(begin)
                    writer_wanted.writerow(end)
            except ValueError as e:
                print "Failed to parse line", line_number
                print row
    except StopIteration:
        pass

Answer 2

datetime.timedelta是用于比较时间跨度的模块：

>>> import datetime
>>> old_time = datetime.datetime.now() #Then wait a little while before running the next line
>>> new_time = datetime.datetime.now()
>>> (new_time - old_time).total_seconds()
4.302
>>> (new_time - old_time) < datetime.timedelta(seconds=2)
False
>>> (new_time - old_time) < datetime.timedelta(hours=2)
True

Answer 3

日期的年份部分的代码中似乎有错字。 正确的 CSV 字段如下所示。你只有“小时”而不是“小时”

年月日小时s YYYY 分钟

下面的代码对我有用

import sys
import csv
import datetime

with open("/tmp/csvinput", "rb") as input, open("/tmp/csv.out","wb") as output:
    reader = csv.DictReader(input,delimiter=",", skipinitialspace=True)
    writer_wanted = csv.DictWriter(output, reader.fieldnames, delimiter=",")
    prev_row = None
    for line_number, row in enumerate(reader):
        try:
            cur_dt = datetime.datetime(year=int(row['Year Month Day Hours Minutes in YYYY']), month=int(row['MM']), day=int(row['DD']), hour=int(row['HH24']))
            print line_number,cur_dt
            if not prev_row:
                prev_row = {'row': row, 'dt': cur_dt}
                continue
            prev_dt = prev_row['dt']
            if (cur_dt - prev_dt) >= datetime.timedelta(hours=2) :
                writer_wanted.writerow(prev_row['row'])
                writer_wanted.writerow(row)
            else:
                prev_row = None
        except:
            e = sys.exc_info()[0]
            print "Failed with exception",e