在特定时间窗口内计算来自 IP 的会话

Question

我正在使用Amazon Redshift。

我有一个 IP 地址列表。可能有来自同一个 IP 地址的多个条目，但在某个时间窗口内（例如 15 分钟）具有不同的 session_id。我想在这个时间窗口内计算任何给定 IP 地址的这些会话。

换句话说，我想知道在任何 15 分钟时间窗口内从某个 IP 地址登录的会话数。

所以我想出了以下查询：

SELECT t1.client_ip,
             COUNT(DISTINCT t2.session_id) AS sessions
      FROM t AS t1
        JOIN t AS t2
          ON t1.client_ip = t2.client_ip
         AND t2.created_at BETWEEN t1.created_at
         AND dateadd (MINUTE,15,t1.created_at)
      GROUP BY t1.client_ip
      HAVING COUNT(DISTINCT t2.session_id) >= 5
      ORDER BY t1.client_ip

很遗憾，查询耗时过长，而且返回的结果也不正确。必须有更好的方法来实现这一点。表中有大约 1800 万个不同的 IP 地址，而表本身有大约 4 亿多条记录。

以下是一些示例数据：

Client_ip    Session_id    created_at
1.0.0.0       abc         <timestamp>
1.0.0.0       def         <timestamp> + 5 minutes
1.0.0.0       ghi         <timestamp> + 25 minutes
2.0.0.0       jkl         <timestamp1>
2.0.0.0       mno         <timestamp1> + 10 minutes
2.0.0.0       pqr         <timestamp1> + 20 minutes

要求的结果：

Client_ip    #Sessions
1.0.0.0       2          (sessions abc and def)
2.0.0.0       2          (sessions mno and pqr)

任何帮助将不胜感激。

编辑：

也许问题不是很清楚。对此我深表歉意。

我不希望有一个固定的时间窗口，我可以在其中创建相隔 15 分钟的时间间隔。我想从某个 IP 地址计算任何 15 分钟窗口中的会话数。

例如：在我发布的示例数据中，会话mno 和pqr 应该被计算在内（对于它们的 IP 地址），因为它们彼此相距不到 15 分钟。同样，会话abc 和def 也应计入它们各自的 IP 地址，因为它们彼此相距不到 15 分钟。我没有为它创建外部开始时间。理想情况下，查询应将具有相同 IP 地址的每条记录与所有其他记录进行比较。不需要创建外部开始时间。

这是explain <query>的输出

    XN Subquery Scan derived_table1  (cost=6516525010733.39..6516525010733.41 rows=2 width=524)
  ->  XN Merge  (cost=6516525010733.39..6516525010733.39 rows=2 width=1032)
        Merge Key: t1.client_ip
        ->  XN Network  (cost=6516525010733.39..6516525010733.39 rows=2 width=1032)
              Send to leader
              ->  XN Sort  (cost=6516525010733.39..6516525010733.39 rows=2 width=1032)
                    Sort Key: t1.client_ip
                    ->  XN HashAggregate  (cost=5516525010733.36..5516525010733.38 rows=2 width=1032)
                          Filter: (count(DISTINCT session_id) >= 10)
                          ->  XN Hash Join DS_DIST_BOTH  (cost=6284418.61..5516506756947.79 rows=2433838076 width=1032)
                                Outer Dist Key: t2.client_ip
                                Inner Dist Key: t1.client_ip
                                Hash Cond: (("outer".client_ip)::text = ("inner".client_ip)::text)
                                Join Filter: (("inner".created_at <= "outer".created_at) AND ("outer".created_at <= date_add('minute'::text, 15::bigint, "inner".created_at)))
                                ->  XN Seq Scan on fbs_page_view_staging t2  (cost=0.00..6279185.96 rows=2093062 width=1040)
                                      Filter: ((created_at <= '2017-09-30 00:00:00'::timestamp without time zone) AND (created_at >= '2017-09-01 00:00:00'::timestamp without time zone))
                                ->  XN Hash  (cost=6279185.96..6279185.96 rows=2093062 width=524)
                                      ->  XN Seq Scan on fbs_page_view_staging t1  (cost=0.00..6279185.96 rows=2093062 width=524)
                                            Filter: ((created_at <= '2017-09-30 00:00:00'::timestamp without time zone) AND (created_at >= '2017-09-01 00:00:00'::timestamp without time zone))
----- Tables missing statistics: fbs_page_view_staging -----
----- Update statistics by running the ANALYZE command on these tables -----

Answer 1

这就是我的想法..

SELECT t1.client_ip, t1.session_id, COUNT(DISTINCT t2.session_id)
FROM  ( SELECT client_ip, session_id, MIN(created_at) created_at
                     FROM   fbs_page
                     GROUP BY client_ip, session_id) AS t1 
       INNER JOIN (SELECT client_ip, session_id, MIN(created_at) created_at
                     FROM   fbs_page
                     GROUP BY client_ip, session_id) AS t2
         ON t1.client_ip = t2.client_ip
            AND t1.session_id != t2.session_id 
            AND t1.created_at 
            BETWEEN DATEADD(MINUTE,-15,t2.created_at) AND t2.created_at
GROUP BY t1.client_ip, t1.session_id
ORDER  BY 1, 2;

经过一些讨论，我认为这可能接近您的需要。您可以添加 where 子句来根据需要更短的日期跨度或其他事情来过滤结果，以使其运行得更快。

Answer 2

SELECT t1.client_ip, t1.WindowStart, COUNT(DISTINCT t1.session_id) AS sessions
FROM (
        SELECT DISTINCT client_ip, 
                        created_at as WindowStart, 
                        DATEADD(MINUTE,15,created_at) as WindowEnd
        FROM t
        -- Add a where clause in here if you want to reduce the number of rows that you're working with
        -- e.g. WHERE created_at BETWEEN 'some_arbitrary_date' AND 'another_arbitrary_date'
     ) t1
  INNER JOIN t as t2 ON t1.client_ip = t2.client_ip 
                    AND t2.created_at BETWEEN t1.WindowStart AND t1.WindowEnd
GROUP BY t1.client_ip, t1.WindowStart