【问题标题】:Duplicate Records On Join加入时重复记录
【发布时间】:2019-10-04 13:02:54
【问题描述】:

BigQuery 的新用户。

inventory 表中,粒度为depot_idproduct_idinventorytransaction 表具有从inventory 表完成的每个操作(加法或减法)日志。

需要的是在inventorySELECT 中获取当年每个月(1 月到 12 月)的数量总和,就像这样

SELECT inventory.*, janTotalQuantity, febTotalQuantity, marTotalQuantity,...

我尝试的是带有子查询的 LEFT JOIN 库存表,该子查询获取每个仓库和产品的月年总数量(例如 2019 年 1 月、2019 年 2 月、2019 年 3 月……)。下面是执行此操作的 SQL 语句。

SELECT inv.inventory_id, p.product_name, p.product_type, p.product_distributor as distributor, p.product_category as category, d.depot_name as location, inv.quantity, inv.lower_limit, inv.unit_cost, inv.quantity * inv.unit_cost as value, p.product_id, d.depot_id, TIMESTAMP_SECONDS(inv.update_date) as last_update, inv.delete_status, IF(agg_sd.mon_year = "Jan-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS janQuantityTotal,IF(agg_sd.mon_year = "Feb-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS febQuantityTotal,IF(agg_sd.mon_year = "Mar-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS marQuantityTotal,IF(agg_sd.mon_year = "Apr-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS aprQuantityTotal,IF(agg_sd.mon_year = "May-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS mayQuantityTotal,IF(agg_sd.mon_year = "Jun-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS junQuantityTotal,IF(agg_sd.mon_year = "Jul-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS julQuantityTotal,IF(agg_sd.mon_year = "Aug-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS augQuantityTotal,IF(agg_sd.mon_year = "Sep-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS sepQuantityTotal,IF(agg_sd.mon_year = "Oct-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS octQuantityTotal,IF(agg_sd.mon_year = "Nov-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS novQuantityTotal,IF(agg_sd.mon_year = "Dec-{{ execution_date.year }}", agg_sd.totalQuantity, 0) AS decQuantityTotal      
FROM iprocure_stage.inventory inv
JOIN iprocure_ods.product p ON p.product_id = inv.product_id 
JOIN iprocure_ods.depot d ON d.depot_id = inv.depot_id
LEFT JOIN (
     SELECT FORMAT_TIMESTAMP('%b-%Y', transaction_date) mon_year, product_id, depot_id, SUM(quantity) as totalQuantity
     FROM `iprocure_ods.inventorytransaction`
     WHERE EXTRACT(YEAR FROM transaction_date) = {{ execution_date.year }}
                AND transaction_type = 1 AND (reference_type = 1 OR reference_type = 6) AND delete_status = 0
                GROUP BY mon_year, product_id, depot_id 
 ) AS agg_sd ON agg_sd.product_id = inv.product_id AND agg_sd.depot_id = inv.depot_id

上述查询的问题是,对于给定仓库产品的每个月的总数量,重复库存记录,像这样

----------------------------------------------------------------------------------
  inventory_id    depot_id    product_id    janTotalQuantity    febTotalQuantity
-------------------------------------------------------------------------------------
    123             2             3              56                   0
    123             2             3              0                    65

如何避免重复inventory 表并在 BigQuery 中添加每月总量列

【问题讨论】:

  • 简化您的问题 - stackoverflow.com/help/reprex。向我们展示一些示例表数据和预期结果(格式文本,无图像。)
  • @jarlh,我添加了示例表数据
  • 更新你的问题并添加预期的结果(你的行没有重复)所以你可能是重复的其他意思

标签: sql join google-bigquery airflow


【解决方案1】:

您可以在部分总和之外进行分组,并在这些总和上应用 SUM 聚合函数。这应该会使您的输出数据集变平:

SELECT inv.inventory_id, p.product_name, p.product_type, p.product_distributor as distributor, p.product_category as category, d.depot_name as location, inv.quantity, inv.lower_limit, inv.unit_cost, inv.quantity * inv.unit_cost as value, p.product_id, d.depot_id, TIMESTAMP_SECONDS(inv.update_date) as last_update, inv.delete_status,
SUM(IF(agg_sd.mon_year = "Jan-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS janQuantityTotal,
SUM(IF(agg_sd.mon_year = "Feb-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS febQuantityTotal,
SUM(IF(agg_sd.mon_year = "Mar-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS marQuantityTotal,
SUM(IF(agg_sd.mon_year = "Apr-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS aprQuantityTotal,
SUM(IF(agg_sd.mon_year = "May-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS mayQuantityTotal,
SUM(IF(agg_sd.mon_year = "Jun-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS junQuantityTotal,
SUM(IF(agg_sd.mon_year = "Jul-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS julQuantityTotal,
SUM(IF(agg_sd.mon_year = "Aug-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS augQuantityTotal,
SUM(IF(agg_sd.mon_year = "Sep-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS sepQuantityTotal,
SUM(IF(agg_sd.mon_year = "Oct-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS octQuantityTotal,
SUM(IF(agg_sd.mon_year = "Nov-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS novQuantityTotal,
SUM(IF(agg_sd.mon_year = "Dec-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS decQuantityTotal      
FROM iprocure_stage.inventory inv
JOIN iprocure_ods.product p ON p.product_id = inv.product_id 
JOIN iprocure_ods.depot d ON d.depot_id = inv.depot_id
LEFT JOIN (
     SELECT FORMAT_TIMESTAMP('%b-%Y', transaction_date) mon_year, product_id, depot_id, SUM(quantity) as totalQuantity
     FROM `iprocure_ods.inventorytransaction`
     WHERE EXTRACT(YEAR FROM transaction_date) = {{ execution_date.year }}
                AND transaction_type = 1 AND (reference_type = 1 OR reference_type = 6) AND delete_status = 0
                GROUP BY mon_year, product_id, depot_id 
 ) AS agg_sd ON agg_sd.product_id = inv.product_id AND agg_sd.depot_id = inv.depot_id
GROUP BY inv.inventory_id, p.product_name, p.product_type, p.product_distributor as distributor, p.product_category as category, d.depot_name as location, inv.quantity, inv.lower_limit, inv.unit_cost, inv.quantity * inv.unit_cost as value, p.product_id, d.depot_id, TIMESTAMP_SECONDS(inv.update_date), inv.delete_status

【讨论】:

    【解决方案2】:

    您正在尝试模仿数据透视表,因为它应该使用(假)聚合函数

    SELECT inv.inventory_id
      , p.product_name
      , p.product_type
      , p.product_distributor as distributor
      , p.product_category as category
      , d.depot_name as location
      , inv.quantity
      , inv.lower_limit
      , inv.unit_cost
      , inv.quantity * inv.unit_cost as value
      , p.product_id, d.depot_id
      , TIMESTAMP_SECONDS(inv.update_date) as last_update
      , inv.delete_status
      , max(IF(agg_sd.mon_year = "Jan-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS janQuantityTotal
      , max(IF(agg_sd.mon_year = "Feb-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS febQuantityTotal
      , max(IF(agg_sd.mon_year = "Mar-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS marQuantityTotal
      , max(IF(agg_sd.mon_year = "Apr-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS aprQuantityTotal
      , max(IF(agg_sd.mon_year = "May-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS mayQuantityTotal
      , max(IF(agg_sd.mon_year = "Jun-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS junQuantityTotal
      , max(IF(agg_sd.mon_year = "Jul-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS julQuantityTotal
      , max(IF(agg_sd.mon_year = "Aug-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS augQuantityTotal
      , max(IF(agg_sd.mon_year = "Sep-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS sepQuantityTotal
      , max(IF(agg_sd.mon_year = "Oct-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS octQuantityTotal
      , max(IF(agg_sd.mon_year = "Nov-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS novQuantityTotal
      , max(IF(agg_sd.mon_year = "Dec-{{ execution_date.year }}", agg_sd.totalQuantity, 0)) AS decQuantityTotal      
    FROM iprocure_stage.inventory inv
    JOIN iprocure_ods.product p ON p.product_id = inv.product_id 
    JOIN iprocure_ods.depot d ON d.depot_id = inv.depot_id
    LEFT JOIN (
         SELECT FORMAT_TIMESTAMP('%b-%Y', transaction_date) mon_year, product_id, depot_id, SUM(quantity) as totalQuantity
         FROM `iprocure_ods.inventorytransaction`
         WHERE EXTRACT(YEAR FROM transaction_date) = {{ execution_date.year }}
                    AND transaction_type = 1 AND (reference_type = 1 OR reference_type = 6) AND delete_status = 0
                    GROUP BY mon_year, product_id, depot_id 
     ) AS agg_sd ON agg_sd.product_id = inv.product_id AND agg_sd.depot_id = inv.depot_id
    GROUP BY inv.inventory_id
      , p.product_name
      , p.product_type
      , p.product_distributor as distributor
      , p.product_category as category
      , d.depot_name as location
      , inv.quantity
      , inv.lower_limit
      , inv.unit_cost
      , inv.quantity * inv.unit_cost as value
      , p.product_id, d.depot_id
      , TIMESTAMP_SECONDS(inv.update_date) as last_update
      , inv.delete_status 
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2012-04-13
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2017-11-03
      • 1970-01-01
      • 1970-01-01
      • 2023-03-19
      相关资源
      最近更新 更多