将 Vertica 中的记录随机拆分为测试和控制

Question

我正在研究一个用例，按日期对记录进行分区，并在 Vertica 中将 80% 的记录随机分配到测试中，将 20% 分配到控制中。如何使用 SQL 来完成。

**For example, Data may look like this:**
UserA 01/01/2000
UserB 01/01/2000 
UserC 01/01/2000 
UserD 01/01/2000 
UserE 01/01/2000
...  
UserF 02/02/2001
UserG 02/02/2001 
UserH 02/02/2001 
UserI 02/02/2001 
UserJ 02/02/2001

**Output should look like:**
UserA 01/01/2000 Test
UserB 01/01/2000 Control
UserC 01/01/2000 Test
UserD 01/01/2000 Test
UserE 01/01/2000 Control
...
UserF 02/02/2001 Control
UserG 02/02/2001 Test
UserH 02/02/2001 Test
UserI 02/02/2001 Test
UserJ 02/02/2001 Control

Answer 1

这是使用PERCENT_RANK Analytic Function 并按RANDOM function 对每个分区中的记录进行排序的可能解决方案：

with t as (
             select 'UserA' as u, date '2000-01-01' as d
   union all select 'UserB' as u, date '2000-01-01' as d
   union all select 'UserC' as u, date '2000-01-01' as d
   union all select 'UserD' as u, date '2000-01-01' as d
   union all select 'UserE' as u, date '2000-01-01' as d
   --
   union all select 'UserF' as u, date '2001-02-02' as d
   union all select 'UserG' as u, date '2001-02-02' as d
   union all select 'UserH' as u, date '2001-02-02' as d
   union all select 'UserI' as u, date '2001-02-02' as d
   union all select 'UserJ' as u, date '2001-02-02' as d
)
select u,
       d, 
       percent_rank() over (partition by d order by random()) as pctrank,
       case when percent_rank() over (partition by d order by random()) <= 0.8 then 'Test' else 'Control' end bucket
  from t
 order by d, u;

输出：

u      d           pctrank  bucket   
-----  ----------  -------  -------  
UserA  2000-01-01  0,25     Test     
UserB  2000-01-01  0,75     Test     
UserC  2000-01-01  0,5      Test     
UserD  2000-01-01  0        Test     
UserE  2000-01-01  1        Control  
UserF  2001-02-02  0,25     Test     
UserG  2001-02-02  0        Test     
UserH  2001-02-02  0,75     Test     
UserI  2001-02-02  1        Control  
UserJ  2001-02-02  0,5      Test     

Answer 2

比这容易得多。 Vertica 具有机器学习功能和子句： 我们通常将数据直接拆分成新表：

CREATE TABLE indata_control AS
SELECT * FROM indata
TABLESAMPLE 20;

-- assuming that `username` is the primary key ...
CREATE TABLE indata_test AS
SELECT * FROM indata
WHERE username NOT IN (
  SELECT username FROM indata_control;
);

但是，如果您更喜欢有一个分类列，那么请执行以下操作。 但是请注意，更新表中的所有行会删除表中的所有行并再次添加所有行 - 因为 Vertica 从未真正删除行 - 这使得此方法的效率大大降低.....

ALTER TABLE indata ADD bucket VARCHAR(7);
UPDATE indata SET bucket='Control' WHERE RANDOM() <= 0.2;
UPDATE indata SET bucket='Test' WHERE bucket IS NULL;
COMMIT;

或者，想想看 - 比更新表中的所有行更好 - 无论 DBMS 是什么，这都是一个坏主意 - 去：

ALTER TABLE indata RENAME TO indata_old;
CREATE TABLE indata LIKE indata_old INCLUDING PROJECTIONS;
ALTER TABLE indata ADD bucket VARCHAR(7);
INSERT INTO indata 
  SELECT 
    *
  , CASE WHEN RANDOM() < 0.2 
      THEN 'Test'
      ELSE 'Control'
    END AS bucket
FROM indata_old;
COMMIT;
DROP TABLE indata_old;

Answer 3

如果您有大量数据，那么一个简单的case 可能就足够了：

(case when random() < 0.8 then 'test' else 'control' end)