【问题标题】：start end comparison开始结束比较
【发布时间】：2021-12-23 09:30:23
【问题描述】：

我无法计算每行的开始和结束是连续的记录 ID 总数。连续意味着当一行在前一行结束之前开始并且 Name == Name 时。记录 ID 1-3 是连续的，因为它们重叠并且具有连续的开始/结束日期时间。

我只想在连续冲突总数 > = 3 的情况下显示 TRUE，否则为 FALSE。

import pandas as pd
import io

#SAMPLE DATA 1 DF
df = pd.read_csv(io.StringIO("""
Record ID;Record Name;Record Start;Record End
1;SMITH, JOHN;10/20/20 8:00 AM;10/20/20 9:30 AM
2;SMITH, JOHN;10/20/20 9:20 AM;10/20/20 10:30 AM
3;SMITH, JOHN;10/20/20 10:20 AM;10/20/20 11:00 AM
4;SMITH, JOHN ;10/20/20 1:00 AM;10/20/20 2:15 PM
5;SMITH, JOHN;10/20/20 2:00 PM;10/20/20 4:00 PM
"""),sep=';')

# SAMPLE DATA 2 DF
df = pd.read_csv(io.StringIO("""
Record ID;Record Name;Record Start;Record End
1;SMITH, JOHN;10/4/20 8:00 AM;10/20/20 9:30 AM
2;SMITH, JOHN;10/4/20 9:20 AM;10/20/20 10:30 AM
3;SMITH, JOHN;10/4/20 11:20 AM;10/20/20 12:00 PM
4;SMITH, JOHN ;10/4/20 1:00 PM;10/20/20 2:15 PM
5;SMITH, JOHN;10/4/20 3:15 PM;10/20/20 4:00 PM
"""),sep=';')

df['Start'] = df['Record Start']
df['End'] = df['Record End']

df['Start'] = pd.to_datetime(df['Start'], errors='coerce')
df['End'] = pd.to_datetime(df['End'], errors='coerce')

df['overlap?'] =  False


print(df)

Expected Output for Sample Data 1:
   Record ID     Record Name  ... overlap? total records consec >=3?
0          1     SMITH, JOHN  ...     True                    True
1          2     SMITH, JOHN  ...     True                    True
2          3     SMITH, JOHN  ...     True                    True
3          4    SMITH, JOHN   ...     True                    False
4          5     SMITH, JOHN  ...     True                    False



Expected Output for Sample Data 2:

   Record ID   Record Name  ... overlap? total records consec >=3?
0          1   SMITH, JOHN  ...     True                    False
1          2   SMITH, JOHN  ...     True                    False
2          3   SMITH, JOHN  ...     True                    False
3          4  SMITH, JOHN   ...    False                    False
4          5   SMITH, JOHN  ...     True                    False

这会产生误报。它只是按名称和日期分组并计算重叠次数。但不考虑这些重叠是否连续。

更新：从建议的答案中，如果计数是连续的，我将以下代码添加到它的末尾以获取真值或假值。（对于任何有兴趣的人）。不幸的是，解决方案（对应部分）不起作用的示例数据 3。

prev= -1
consecutive = []
for i, v in enumerate(df['Count'].values):
    if v <= prev:
        if prev >= 3:
            consecutive += prev * [True]
        else:
            consecutive += prev * [False]
    elif len(df) == i + 1:
        if prev >= 3:
            consecutive += v * [True]
        else:
            consecutive += v * [False]
    prev = v

df[['Consecutive']] = consecutive


# SAMPLE DATA 3 DF
df = pd.read_csv(io.StringIO("""
Record ID;Record Name;Record Start;Record End
1;SMITH, JOHN;10/5/20 7:47 AM;10/5/20 8:05 AM
2;SMITH, JOHN;10/5/20 11:43 AM;10/5/20 1:26 AM
3;SMITH, JOHN;10/5/20 12:48 AM;10/5/20 1:31 PM
4;SMITH, JOHN ;10/5/20 2:50 PM;10/5/20 5:00 PM
"""),sep=';')

Current Output: 

Event ID         Name Event Date  ...                End2 overlap Count
0         1  SMITH, JOHN 2021-10-05  ... 2021-10-05 08:05:00   False     1
1         2  SMITH, JOHN 2021-10-05  ... 2021-10-05 13:26:00    True     2
2         3  SMITH, JOHN 2021-10-05  ... 2021-10-05 13:31:00    True     3
3         4  SMITH, JOHN 2021-10-05  ... 2021-10-05 17:53:00   False     4

Expected Output:

Event ID         Name Event Date  ...                End2 overlap Count
0         1  SMITH, JOHN 2021-10-05  ... 2021-10-05 08:05:00   False     1
1         2  SMITH, JOHN 2021-10-05  ... 2021-10-05 13:26:00    True     1
2         3  SMITH, JOHN 2021-10-05  ... 2021-10-05 13:31:00    True     2
3         4  SMITH, JOHN 2021-10-05  ... 2021-10-05 17:53:00   False     1

预期输出的推理：

事件 1 不与任何其他事件发生冲突。计数 =1（从 1 开始）并且重叠 = False
事件 2 和 3 相互重叠。事件 ID 2 的计数设置回 1，事件 ID 3 的计数设置回 2。重叠 = 两者都为真。
事件 4 不与任何其他事件重叠。计数设置回 1。重叠 = 假

样本数据 4

df = pd.read_csv(io.StringIO("""
Record ID;Record Name;Record Start;Record End
1;SMITH, JOHN;10/4/20 12:00 AM;10/4/20 7:00 PM
2;SMITH, JOHN;10/4/20 8:00 AM;10/4/20 9:00 AM
3;SMITH, JOHN;10/4/20 10:00 AM AM;10/4/20 11:00 AM
4;SMITH, JOHN ;10/4/20 4:30 PM;10/4/20 5:00 PM
"""),sep=';')

Current Output:

         Record Start          Record End  overlap  Count
0 2021-10-04 02:00:00 2021-10-04 19:53:00     True      1
1 2021-10-04 08:05:00 2021-10-04 08:47:00     True      2
2 2021-10-04 09:55:00 2021-10-04 10:36:00     True      1
3 2021-10-04 13:19:00 2021-10-04 14:15:00     True      1
4 2021-10-04 16:39:00 2021-10-04 17:07:00     True      1

Expected Output:

         Record Start          Record End  overlap  Count
0 2021-10-04 02:00:00 2021-10-04 19:53:00     True      1
1 2021-10-04 08:05:00 2021-10-04 08:47:00     True      2
2 2021-10-04 09:55:00 2021-10-04 10:36:00     True      3
3 2021-10-04 13:19:00 2021-10-04 14:15:00     True      4
4 2021-10-04 16:39:00 2021-10-04 17:07:00     True      5

【问题讨论】：

标签： python pandas loops python-datetime

【解决方案1】：

使用 dataframe.loc 获取当前行和上一行，如果日期相等，则在上一行计数列中加一，否则如果不相等，则将计数设置为 1。过滤数据框中计数大于 3 的所有行. 您也可以根据姓名和日期建立一个累计。

我在解决方案中经常使用 timedelta。我在开始日期时间和结束日期时间之间使用 total_seconds，然后除以 60 得到分钟，将其添加到开始时间以创建从开始的日期时间偏移量，间隔为一分钟。

apply 创建开始和结束日期时间之间的分钟间隔。

df = pd.read_csv(io.StringIO("""
Record ID;Record Name;Record Start;Record End
1;SMITH, JOHN;10/20/20 8:00 AM;10/20/20 9:30 AM
2;SMITH, JOHN;10/20/20 9:20 AM;10/20/20 10:30 AM
3;SMITH, JOHN;10/20/20 10:20 AM;10/20/20 11:00 AM
4;COOPER, ALLEN;10/20/20 1:00 PM;10/20/20 2:15 PM
5;PEREZ, HILL;10/20/20 3:15 PM;10/20/20 4:00 PM
6;SMITH, JOHN;10/4/21 8:00 AM;10/20/21 9:30 AM
7;SMITH, JOHN;10/4/21 9:20 AM;10/20/21 10:30 AM
8;SMITH, JOHN;10/4/21 11:20 AM;10/20/21 12:00 PM
9;SMITH, JOHN ;10/4/21 1:00 PM;10/20/21 2:15 PM
10;SMITH, JOHN;10/4/21 3:15 PM;10/20/21 4:00 PM
"""),sep=';')

df['Record Start']=pd.to_datetime(df['Record Start'])
df['Record End']=pd.to_datetime(df['Record End'])
def create_datetime(date,hour,minute,second):
    month=date.month
    day=date.day
    year=date.year
    return datetime(year=year,month=month,day=day,hour=hour,minute=minute,second=second,microsecond=0)
def get_minutes(row):
    start=row['Record Start']
    end = row['Record End']

    results=[start + timedelta(minutes=x) for x in range(0, round((end-start).total_seconds()//60)+1)]
    
    #for item in results:
    #    print(item)
    #sys.exit()
    return results

df['minutes'] = df.apply(get_minutes, axis=1)

def intersection(lst1, lst2):
    return list(set(lst1) & set(lst2))

prev_row=None
for index,row in df.iterrows():
    if index==0:
        df.loc[index,'Count']=1
    else:
        prev_row=df.iloc[index-1]
        
    if not prev_row is None:
        if prev_row['Record Name']==row['Record Name']:
            count=prev_row['Count']
            lst1=row['minutes']
            lst2=prev_row['minutes']
            if len(intersection(lst1,lst2))>0:
                df.loc[index,'Count']=count+1
            else:
                df.loc[index,'Count']=1
        else:
            df.loc[index,'Count']=1
        
    #print(df[df['Count']>=3])   
    print(df)

输出：

 Record ID    Record Name       Record Start         Record End  Count
 0          1    SMITH, JOHN   10/20/20 8:00 AM   10/20/20 9:30 AM         1.0
 1          2    SMITH, JOHN   10/20/20 9:20 AM  10/20/20 10:30 AM    2.0
 2          3    SMITH, JOHN  10/20/20 10:20 AM  10/20/20 11:00 AM    3.0
 3          4  COOPER, ALLEN   10/20/20 1:00 PM   10/20/20 2:15 PM    1.0
 4          5    PEREZ, HILL   10/20/20 3:15 PM   10/20/20 4:00 PM    1.0
 5          6    SMITH, JOHN    10/4/21 8:00 AM   10/20/21 9:30 AM    1.0
 6          7    SMITH, JOHN    10/4/21 9:20 AM  10/20/21 10:30 AM    2.0
 7          8    SMITH, JOHN   10/4/21 11:20 AM  10/20/21 12:00 PM    3.0
 8          9   SMITH, JOHN     10/4/21 1:00 PM   10/20/21 2:15 PM    1.0
 9         10    SMITH, JOHN    10/4/21 3:15 PM   10/20/21 4:00 PM    1.0

【讨论】：

当然，请查看 SMITH, JOHN 的 ID 为 1-3 的示例数据。 ID 1 在 ID 1 结束之前开始。 ID 3 在 ID 2 结束之前开始。连续冲突总数 = 3。另一个例子是在我的代码（示例 DF2）中，虽然存在时间冲突（重叠），但总连续冲突 = 0，因为没有一个 >= 计数 3。希望这是有道理的。
如果在同一小时内的同一日期，重叠时间以分钟为单位。这是当前迭代（重叠）出现的地方。我在考虑你提到的分区。问题是有时，记录名称不会改变，我必须依赖是否有连续的开始/结束日期。重叠部分工作正常，计数器部分让我失望，特别是如果记录名称没有改变。
不幸的是，此逻辑不适用于#SAMPLE DATA 2 DF。另外，我需要用 TRUE 或 FALSE 显示所有列。在这种情况下，它只是过滤 3。如果记录导致 3，我需要查看集合中的所有记录。
你能解释一下这是做什么的吗：def intersection(lst1, lst2): return list(set(lst1) & set(lst2))？我收到错误消息：'float' 对象不可迭代
我转换为整数： df['minutes'] = df['minutes'].apply(np.int64) 因为它以前是“[datetime.datetime(2020, 10, 4, 7, 56), datetime.datetime(2020, 10, 4, 7, 57)”。不确定这是否是故意的，但对我来说没有意义。但是，int 不可迭代，因此该函数无法按预期工作

【解决方案2】：

我终于找到了一个可行的解决方案。以下供将来的任何人使用：

df = pd.read_excel(r'PATH\FILE')   

# Create new columns for Start/End values
df['Start'] = df['Record Start']
df['End'] = df['Record End']

# Convert to pandas datetime
df['Start'] = pd.to_datetime(df['Start'], errors='coerce')
df['End'] = pd.to_datetime(df['End'], errors='coerce')

# set static values
nest = []
flat = []

# Find overlapping events
df['overlap'] = False

for i, row in df.iterrows():
    if i in flat:
        continue
    start, end = row["Start"], row["End"]
    flag = True
    counter = 0

    while flag:
        counter += 1
        res = df.loc[(df['Name'] == row['Name']) &
                     (((df["Start"] >= start) & (df["Start"] <= end)) |
                      ((df["End"] >= start) & (df["End"] <= end)) |
                      ((end >= df["Start"]) & (end <= df["End"])) |
                      ((start >= df["Start"]) & (start <= df["End"])))].index.tolist()
        resbkup = res
        res += [i]
        temp_df = df.loc[res]
        temp_start = temp_df['Start'].min()
        temp_end = temp_df['End'].max()
        if counter > 50:
            print("True -- ",start,end, resbkup)
            print("temp", temp_start,  temp_end)
            print(flag, res)
        if ((temp_start == start) and (temp_end == end)):
            flag = False
        else:
            start, end = temp_start, temp_end

    res = list(set(res))
    res.sort()
    nest.append(res)
    flat += [j for j in res]

for i, n in enumerate(nest):
    if len(n) >1:
        df.loc[n, 'overlapIndex'] = "OverLap" +str(int(i+1))
        df.loc[n, 'overlap'] = True
    else:
        df.loc[n, 'overlap'] = False
    if len(n) >= 3:
        print(n)
        df.loc[n, 'Consecutive'] = True
    else:
        df.loc[n, 'Consecutive'] = False
        
print(df)

【讨论】：