SQL 查找具有多个字段的重复项（没有唯一 ID）

Question

我正在尝试使用 vendor 表和 vendor_address 表中的多个字段从数据库中查找重复的供应商。问题是我做的内部连接越多，查询丢失的潜在结果就越少。虽然我在供应商 ID 中没有重复，但我希望找到类似的潜在供应商。

这是我目前的查询：

SELECT 
     o.vendor_id
    ,o.vndr_name_shrt_user
    ,O.COUNTRY 
    ,O.VENDOR_NAME_SHORT 
    ,B.POSTAL
    ,B.ADDRESS1
    ,SAME_ADDRESS_NB
    ,SAME_POSTAL_NB
    ,OC.SAME_SHORT_NAME
    ,oc.SAME_USER_NUM
FROM VENDOR o

JOIN vendor_addr B ON o.VENDOR_ID = B.VENDOR_ID

INNER JOIN (
    SELECT vndr_name_shrt_user, COUNT(*) AS SAME_USER_NUM
    FROM VENDOR 
    WHERE COUNTRY = 'CANADA'
    AND VENDOR_STATUS = 'A'
    GROUP BY vndr_name_shrt_user
    HAVING COUNT(*) > 1
) oc on o.vndr_name_shrt_user = oc.vndr_name_shrt_user

INNER JOIN ( SELECT VENDOR_NAME_SHORT, COUNT(*) AS SAME_SHORT_NAME
    FROM VENDOR 
    WHERE COUNTRY = 'CANADA'
    AND VENDOR_STATUS = 'A'
    GROUP BY VENDOR_NAME_SHORT
    HAVING COUNT(*) > 1
) oc on o.VENDOR_NAME_SHORT = oc.VENDOR_NAME_SHORT

INNER JOIN (SELECT POSTAL, COUNT(*) AS SAME_POSTAL_NB
    FROM vendor_addr 
    WHERE COUNTRY = 'CANADA'
    AND COUNTRY ='CANADA'
    AND POSTAL != ' '
    GROUP BY POSTAL
    HAVING COUNT(*) > 1
) oc on b.POSTAL = oc.POSTAL

INNER JOIN (SELECT ADDRESS1, COUNT(*) AS SAME_ADDRESS_NB
    FROM ps_vendor_addr 
    WHERE COUNTRY = 'CANADA'
    AND COUNTRY ='CANADA'
    AND ADDRESS1 != ' '
    GROUP BY ADDRESS1
    HAVING COUNT(*) > 1
) oc on b.ADDRESS1 = oc.ADDRESS1   
WHERE O.COUNTRY ='CANADA' 
    AND B.COUNTY = 'CANADA';

Answer 1

您的联接似乎有点有趣，原因不止一个。首先，你有内部连接，这将消除除了那些有 all 重复迹象的所有 - 这是你不想要的东西。此外，您似乎在所有派生表上都有相同的别名 oc - 这不会真的飞到这里，而且您不会走得太远。

我建议您对每个重复符号重复基本查询，而不是这样做，如下所示（我删除了 same_address_nb 和 same_postal_nb 字段，您会明白为什么）：

select 
    o.vendor_id
    ,o.vndr_name_shrt_user
    ,O.COUNTRY 
    ,O.VENDOR_NAME_SHORT 
    ,B.POSTAL
    ,B.ADDRESS1
    ,OC.SAME_SHORT_NAME
    ,oc.SAME_USER_NUM
from VENDOR o
JOIN vendor_addr B ON o.VENDOR_ID = B.VENDOR_ID
WHERE O.COUNTRY ='CANADA'
AND B.COUNTY = 'CANADA'
AND ...

对于这些重复符号中的每一个，您将向上面显示的省略号添加一个嵌套查询，如下所示 - 使用 vndr_name_shrt_user 中的重复项显示的示例：

select 
    o.vendor_id
    ,o.vndr_name_shrt_user
    ,O.COUNTRY 
    ,O.VENDOR_NAME_SHORT 
    ,B.POSTAL
    ,B.ADDRESS1
    ,OC.SAME_SHORT_NAME
    ,oc.SAME_USER_NUM
    ,'SAME_USER_NUM' as duplicateFlag
from VENDOR o
JOIN vendor_addr B ON o.VENDOR_ID = B.VENDOR_ID
WHERE O.COUNTRY ='CANADA'
AND B.COUNTY = 'CANADA'
AND o.vndr_name_shrt_user in 
(
    SELECT 
        vndr_name_shrt_user
    FROM VENDOR 
    WHERE COUNTRY = o.country
    AND VENDOR_STATUS = 'A'
    GROUP BY vndr_name_shrt_user
    HAVING COUNT(*) > 1
) 

您可以将这些查询 UNION ALL 放在一起，然后查看所有重复项。

附带说明，您在最后三个派生表中检查了两次country = 'canada'。

更新：显示多个重复标志

select 
    o.vendor_id
    ,o.vndr_name_shrt_user
    ,O.COUNTRY 
    ,O.VENDOR_NAME_SHORT 
    ,B.POSTAL
    ,B.ADDRESS1
    ,OC.SAME_SHORT_NAME
    ,oc.SAME_USER_NUM
    ,'SAME_USER_NUM' as duplicateFlag
from VENDOR o
JOIN vendor_addr B ON o.VENDOR_ID = B.VENDOR_ID
WHERE O.COUNTRY ='CANADA'
AND B.COUNTY = 'CANADA'
AND o.vndr_name_shrt_user in 
(
    SELECT 
        vndr_name_shrt_user
    FROM VENDOR 
    WHERE COUNTRY = o.country
    AND VENDOR_STATUS = 'A'
    GROUP BY vndr_name_shrt_user
    HAVING COUNT(*) > 1
) 

UNION ALL

select 
    o.vendor_id
    ,o.vndr_name_shrt_user
    ,O.COUNTRY 
    ,O.VENDOR_NAME_SHORT 
    ,B.POSTAL
    ,B.ADDRESS1
    ,OC.SAME_SHORT_NAME
    ,oc.SAME_USER_NUM
    ,'VENDOR_NAME_SHORT' as duplicateFlag
from VENDOR o
JOIN vendor_addr B ON o.VENDOR_ID = B.VENDOR_ID
WHERE O.COUNTRY ='CANADA'
AND B.COUNTY = 'CANADA'
AND o.VENDOR_NAME_SHORT in 
(
    SELECT 
        VENDOR_NAME_SHORT
    FROM VENDOR 
    WHERE COUNTRY = o.country
    AND VENDOR_STATUS = 'A'
    GROUP BY VENDOR_NAME_SHORT
    HAVING COUNT(*) > 1
) 

Answer 2

让我们有一些有趣的数据，其中包含不同属性的链式重复：

CREATE TABLE data ( ID, A, B, C ) AS
  SELECT 1, 1, 1, 1 FROM DUAL UNION ALL -- Related to #2 on column A
  SELECT 2, 1, 2, 2 FROM DUAL UNION ALL -- Related to #1 on column A, #3 on B & C, #5 on C
  SELECT 3, 2, 2, 2 FROM DUAL UNION ALL -- Related to #2 on columns B & C, #5 on C
  SELECT 4, 3, 3, 3 FROM DUAL UNION ALL -- Related to #5 on column A
  SELECT 5, 3, 4, 2 FROM DUAL UNION ALL -- Related to #2 and #3 on column C, #4 on A
  SELECT 6, 5, 5, 5 FROM DUAL;          -- Unrelated

现在，我们可以使用分析函数获得一些关系（无需任何连接）：

SELECT d.*,
       LEAST(
         FIRST_VALUE( id ) OVER ( PARTITION BY a ORDER BY id ),
         FIRST_VALUE( id ) OVER ( PARTITION BY b ORDER BY id ),
         FIRST_VALUE( id ) OVER ( PARTITION BY c ORDER BY id )
       ) AS duplicate_of
FROM   data d;

这给出了：

ID A B C DUPLICATE_OF
-- - - - ------------
 1 1 1 1            1
 2 1 2 2            1
 3 2 2 2            2
 4 3 3 3            4
 5 3 4 2            2
 6 5 5 5            6

但这并没有表明 #4 与 #5 相关，而 #5 与 #2 相关，然后与 #1 相关......

这可以通过分层查询找到：

SELECT id, a, b, c,
       CONNECT_BY_ROOT( id ) AS duplicate_of
FROM   data
CONNECT BY NOCYCLE ( PRIOR a = a OR PRIOR b = b OR PRIOR c = c );

但这会产生很多很多重复的行（因为它不知道从哪里开始层次结构，所以会依次选择每一行作为根） - 相反，您可以使用第一个查询为分层查询提供一个起始当ID 和DUPLICATE_OF 值相同时：

SELECT id, a, b, c,
       CONNECT_BY_ROOT( id ) AS duplicate_of
FROM   (
  SELECT d.*,
         LEAST(
           FIRST_VALUE( id ) OVER ( PARTITION BY a ORDER BY id ),
           FIRST_VALUE( id ) OVER ( PARTITION BY b ORDER BY id ),
           FIRST_VALUE( id ) OVER ( PARTITION BY c ORDER BY id )
         ) AS duplicate_of
  FROM   data d
)
START WITH id = duplicate_of
CONNECT BY NOCYCLE ( PRIOR a = a OR PRIOR b = b OR PRIOR c = c );

这给出了：

ID A B C DUPLICATE_OF
-- - - - ------------
 1 1 1 1            1
 2 1 2 2            1
 3 2 2 2            1
 4 3 3 3            1
 5 3 4 2            1
 1 1 1 1            4
 2 1 2 2            4
 3 2 2 2            4
 4 3 3 3            4
 5 3 4 2            4
 6 5 5 5            6

仍然有一些行重复，因为搜索中的局部最小值出现 #4 ... 可以使用简单的GROUP BY 删除：

SELECT id, a, b, c,
       MIN( duplicate_of ) AS duplicate_of
FROM   (
  SELECT id, a, b, c,
         CONNECT_BY_ROOT( id ) AS duplicate_of
  FROM   (
    SELECT d.*,
           LEAST(
             FIRST_VALUE( id ) OVER ( PARTITION BY a ORDER BY id ),
             FIRST_VALUE( id ) OVER ( PARTITION BY b ORDER BY id ),
             FIRST_VALUE( id ) OVER ( PARTITION BY c ORDER BY id )
           ) AS duplicate_of
    FROM   data d
  )
  START WITH id = duplicate_of
  CONNECT BY NOCYCLE ( PRIOR a = a OR PRIOR b = b OR PRIOR c = c )
)
GROUP BY id, a, b, c;

它给出了输出：

ID A B C DUPLICATE_OF
-- - - - ------------
 1 1 1 1            1
 2 1 2 2            1
 3 2 2 2            1
 4 3 3 3            1
 5 3 4 2            1
 6 5 5 5            6