如何解释 Google Big Query 中的 STREAMING_TIMELINE_BY_* 视图以进行流式插入成本分析

Question

GBQ (Google Big Query) 为流式插入元数据提供视图，请参阅STREAMING_TIMELINE_BY_*。我想使用这些数据来了解“流式插入”的计费。但是，数字不加 ab，如果我在某个地方弄错了，我想了解一下。

---

流式插入元数据视图中的一个数据点是total_input_bytes：

total_input_bytes   

INTEGER 

Total number of bytes from all rows within the 1 minute interval.

另外，Pricing for data ingestion 说：

Streaming inserts (tabledata.insertAll) 

$0.010 per 200 MB

You are charged for rows that are successfully inserted. Individual rows are calculated using a 1 KB minimum size.

因此，应该可以通过

获得每天流式插入的成本

                 0.01/200 * (SUM(total_input_bytes)/1024/1024)

cost per 200 mb -----^

total bytes in mb ---------------------^

这应该是下界，因为我们忽略了所有小于 1KB 且四舍五入为 1KB 的行。

完整查询：

SELECT
 project_id,
 dataset_id,
 table_id,
 SUM(total_rows) AS num_rows,
 round(SUM(total_input_bytes)/1024/1024,2) AS num_bytes_in_mb,
 # 0.01$ per 200MB
 # @see https://cloud.google.com/bigquery/pricing#data_ingestion_pricing
 round(0.01*(SUM(total_input_bytes)/1024/1024)/200, 2) AS cost_in_dollar,
 SUM(total_requests) AS num_requests
FROM
 `region-us`.INFORMATION_SCHEMA.STREAMING_TIMELINE_BY_PROJECT
where 
  start_timestamp BETWEEN "2021-04-10" and "2021-04-14"
  AND error_code IS NULL
GROUP BY 1, 2, 3
ORDER BY table_id asc

但是，结果并未反映在我们的实际结算报告中。帐单显示的费用不到我预期的一半：

现在我想知道是否可以这样计算成本。

Answer 1

您的查询将小于 0.49kb 的每一行四舍五入为 0kb。这应该可以解释为什么您计算的成本更低。

尝试插入将处理这些值的 CASE 语句：

SELECT
 project_id,
 dataset_id,
 table_id,
 SUM(total_rows) AS num_rows,
 CASE SUM(total_input_bytes)/1024/1024 < 0.001 THEN 0.001 ELSE
 round(SUM(total_input_bytes)/1024/1024,2) END AS num_bytes_in_mb,
 # 0.01$ per 200MB
 # @see https://cloud.google.com/bigquery/pricing#data_ingestion_pricing
CASE SUM(total_input_bytes)/1024/1024 < 0.001 THEN 0.001 ELSE
 round(0.01*(SUM(total_input_bytes)/1024/1024)/200, 2) END AS cost_in_dollar,
 SUM(total_requests) AS num_requests
FROM
 `region-us`.INFORMATION_SCHEMA.STREAMING_TIMELINE_BY_PROJECT
where 
  start_timestamp BETWEEN "2021-04-10" and "2021-04-14"
  AND error IS NULL
GROUP BY 1, 2, 3
ORDER BY table_id asc