有没有办法获得某一列的日期差异

Question

我想计算从订单到到达状态所需的每个唯一名称的时差/日期差。

输入数据框是这样的

+------------------------------+
| Date      | id | name |staus
+------------------------------+
| 1986/10/15| A  |john |order
| 1986/10/16| A  |john |dispatched
| 1986/10/18| A  |john |arrived
| 1986/10/15| B  |peter|order
| 1986/10/16| B  |peter|dispatched
| 1986/10/17| B  |peter|arrived
| 1986/10/16| C  |raul |order
| 1986/10/17| C  |raul |dispatched
| 1986/10/18| C  |raul |arrived
+-----------------------------+

预期的输出数据集应该与此类似

+---------------------------------------------------+
| id | name |time_difference_from_order_to_delivered|
+---------------------------------------------------+
 A   | john |                3days
 B   |peter |                2days
 C   | Raul |                2days
+---------------------------------------------------+
I am stuck on what logic to implement

Answer 1

您可以使用条件聚合进行分组并计算日期差异：

import pyspark.sql.functions as F

df2 = df.groupBy('id', 'name').agg(
    F.datediff(
        F.to_date(F.max(F.when(F.col('staus') == 'arrived', F.col('Date'))), 'yyyy/MM/dd'),
        F.to_date(F.min(F.when(F.col('staus') == 'order', F.col('Date'))), 'yyyy/MM/dd')
    ).alias('time_diff')
)

df2.show()
+---+-----+---------+
| id| name|time_diff|
+---+-----+---------+
|  A| john|        3|
|  C| raul|        2|
|  B|peter|        2|
+---+-----+---------+

也可以直接减去日期，会返回一个区间类型的列：

import pyspark.sql.functions as F

df2 = df.groupBy('id', 'name').agg(
    (
        F.to_date(F.max(F.when(F.col('staus') == 'arrived', F.col('Date'))), 'yyyy/MM/dd') -
        F.to_date(F.min(F.when(F.col('staus') == 'order', F.col('Date'))), 'yyyy/MM/dd')
    ).alias('time_diff')
)

df2.show()
+---+-----+---------+
| id| name|time_diff|
+---+-----+---------+
|  A| john|   3 days|
|  C| raul|   2 days|
|  B|peter|   2 days|
+---+-----+---------+

Answer 2

假设ordered是最早的日期，delivery是最后一个，只需使用aggregation和datediff()：

select id, name, datediff(max(date), min(date)) as num_days
from t
group by id, name;

为了更精确，可以使用条件聚合：

select id, name,
       datediff(max(case when status = 'arrived' then date end)
                min(case when status = 'order' then date end)
               ) as num_days
from t
group by id, name;