【发布时间】：2015-09-24 19:59:42
【问题描述】：
我想计算 Google Big Query Storage 的表成本，但我不知道如何单独查看每个表的存储大小。
【问题讨论】：
标签： google-bigquery
【发布时间】：2015-09-24 19:59:42
【问题描述】：
有几种方法可以做到这一点，但请注意，对于主动接收流式插入的表，表的大小（以字节为单位）属性不可用。
A.使用BQ 命令行工具和JQ linux 库解析JSON。
bq --format=json show publicdata:samples.gsod | jq '.numBytes | tonumber'
这个输出：
17290009238
B.使用 REST api 进行Tables:get 调用
GET https://www.googleapis.com/bigquery/v2/projects/projectId/datasets/datasetId/tables/tableId
这会返回一个完整的 JSON，您可以解析并获取 numBytes。
{
"kind": "bigquery#table",
"description": "This dataset contains weather information collected by NOAA, such a…",
"creationTime": "1335916040125",
"tableReference": {
"projectId": "publicdata",
"tableId": "gsod",
"datasetId": "samples"
},
"numRows": "114420316",
"numBytes": "17290009238",
"etag": "\"Gn3Hpo5WaKnpFuT457VBDNMgZBw/MTQxMzkzNzk4Nzg0Ng\"",
"location": "US",
"lastModifiedTime": "1413937987846",
"type": "TABLE",
"id": "publicdata:samples.gsod",
"selfLink": "https://www.googleapis.com/bigquery/v2/projects/publicdata/datasets…",
"schema": {
"fields": [
{
"description": "The World Meteorological Organization (WMO) / DATSAV3 station numbe…",
"type": "INTEGER",
"name": "station_number",
"mode": "REQUIRED"
},
{
"description": "The Weather-Bureau-Army-Navy (WBAN) station number where the data w…",
"type": "INTEGER",
"name": "wban_number",
"mode": "NULLABLE"
},
{
"description": "The year the data was collected in",
"type": "INTEGER",
"name": "year",
"mode": "REQUIRED"
},
{
"description": "The month the data was collected in",
"type": "INTEGER",
"name": "month",
"mode": "REQUIRED"
},
{
"description": "The day the data was collected in.",
"type": "INTEGER",
"name": "day",
"mode": "REQUIRED"
},
{
"description": "The mean temperature of the day in degrees Fahrenheit, accurate to …",
"type": "FLOAT",
"name": "mean_temp",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_temp.",
"type": "INTEGER",
"name": "num_mean_temp_samples",
"mode": "NULLABLE"
},
{
"description": "The mean dew point of the day in degrees Fahrenheit, accurate to on…",
"type": "FLOAT",
"name": "mean_dew_point",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_dew_point.",
"type": "INTEGER",
"name": "num_mean_dew_point_samples",
"mode": "NULLABLE"
},
{
"description": "The mean sea level pressure of the day in millibars, accurate to on…",
"type": "FLOAT",
"name": "mean_sealevel_pressure",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_sealevel_pressure…",
"type": "INTEGER",
"name": "num_mean_sealevel_pressure_samples",
"mode": "NULLABLE"
},
{
"description": "The mean station pressure of the day in millibars, accurate to one …",
"type": "FLOAT",
"name": "mean_station_pressure",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_station_pressure.…",
"type": "INTEGER",
"name": "num_mean_station_pressure_samples",
"mode": "NULLABLE"
},
{
"description": "The mean visibility of the day in miles, accurate to one tenth of a…",
"type": "FLOAT",
"name": "mean_visibility",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_visibility.",
"type": "INTEGER",
"name": "num_mean_visibility_samples",
"mode": "NULLABLE"
},
{
"description": "The mean wind speed of the day in knots, accurate to one tenth of a…",
"type": "FLOAT",
"name": "mean_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_wind_speed.",
"type": "INTEGER",
"name": "num_mean_wind_speed_samples",
"mode": "NULLABLE"
},
{
"description": "The maximum sustained wind speed reported on the day in knots, accu…",
"type": "FLOAT",
"name": "max_sustained_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The maximum wind gust speed reported on the day in knots, accurate …",
"type": "FLOAT",
"name": "max_gust_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The maximum temperature of the day in degrees Fahrenheit, accurate …",
"type": "FLOAT",
"name": "max_temperature",
"mode": "NULLABLE"
},
{
"description": "Indicates the source of max_temperature.",
"type": "BOOLEAN",
"name": "max_temperature_explicit",
"mode": "NULLABLE"
},
{
"description": "The minimum temperature of the day in degrees Fahrenheit, accurate …",
"type": "FLOAT",
"name": "min_temperature",
"mode": "NULLABLE"
},
{
"description": "Indicates the source of min_temperature.",
"type": "BOOLEAN",
"name": "min_temperature_explicit",
"mode": "NULLABLE"
},
{
"description": "The total precipitation of the day in inches, accurate to one hundr…",
"type": "FLOAT",
"name": "total_precipitation",
"mode": "NULLABLE"
},
{
"description": "The snow depth of the day in inches, accurate to one tenth of an in…",
"type": "FLOAT",
"name": "snow_depth",
"mode": "NULLABLE"
},
{
"description": "Indicates if fog was reported on this day.",
"type": "BOOLEAN",
"name": "fog",
"mode": "NULLABLE"
},
{
"description": "Indicates if rain was reported on this day.",
"type": "BOOLEAN",
"name": "rain",
"mode": "NULLABLE"
},
{
"description": "Indicates if snow was reported on this day.",
"type": "BOOLEAN",
"name": "snow",
"mode": "NULLABLE"
},
{
"description": "Indicates if hail was reported on this day.",
"type": "BOOLEAN",
"name": "hail",
"mode": "NULLABLE"
},
{
"description": "Indicates if thunder was reported on this day.",
"type": "BOOLEAN",
"name": "thunder",
"mode": "NULLABLE"
},
{
"description": "Indicates if a tornado was reported on this day.",
"type": "BOOLEAN",
"name": "tornado",
"mode": "NULLABLE"
}
]
}
}
C.有称为__TABLES__ 和__TABLES_SUMMARY__ 的元表
您可以运行如下查询：
SELECT size_bytes FROM <dataset>.__TABLES__ WHERE table_id='mytablename'
该查询的__TABLES__ 部分可能看起来不熟悉。 __TABLES_SUMMARY__ 是一个元表，其中包含有关数据集中表的信息。您可以自己使用此元表。例如，查询SELECT * FROM publicdata:samples.__TABLES_SUMMARY__ 将返回有关publicdata:samples 数据集中表的元数据。你也可以SELECT * FROM publicdata:samples.__TABLES__
可用字段：
__TABLES_SUMMARY__ 元表的字段（在TABLE_QUERY 查询中都可用）包括：
-
table_id：表名。 -
creation_time：自 1970 年 1 月 1 日 UTC 以来的时间，以毫秒为单位，创建表。这与表格中的creation_time字段相同。 -
type：无论是视图（2）还是常规表（1）。
以下字段在TABLE_QUERY() 中不可用，因为它们是__TABLES__ 但不是__TABLES_SUMMARY__ 的成员。出于历史兴趣和部分记录__TABLES__ 元表，将它们保留在这里：
-
last_modified_time：自 1970 年 1 月 1 日 UTC 以来的时间，以毫秒为单位，表已更新（元数据或表内容）。请注意，如果您使用tabledata.insertAll()将记录流式传输到您的表，这可能会过时几分钟。 -
row_count：表中的行数。 -
size_bytes：表的总大小（以字节为单位）。
【讨论】：
-
我可以使用 Tables.Get 方法获取总字节数。目前我有 4 张桌子（1 张大桌子，3 张小桌子）。当我获得较大表的总字节数时，它返回 18200091100 (16.95 GB) 我现在还没有计算较小的表，但 Google 大查询计费显示 BigQuery 存储为“4.035 GB”。为什么我会得到这些差异？
-
存储定价按每 MB、每秒按比例分配。当您检查当月报告的估计成本时，该数字按每秒按比例分配（例如，本月到目前为止已过去的天数）。您应该检查上个月的数字，那里有全月数据。无论如何，在月底你会看到数字接近你的 gibibyte。
-
我在 2015 年 6 月 24 日创建了我的表（总字节数：18200091100 [16.95 GB]），但我收到了 6 月份的 Google 发票，即“4.035 GB”的 Big Query 存储，但我没有不知道为什么会有这么大的差异（~12GB）？即使是 7 月 1 日的一周发票也显示“4.035 GB”的 Big Query 存储空间。
-
@selvakumar 再次，您无需为 GB 付费，而是按 GB/月付费。如果您有 16.95GB，但在一个月的四分之一时间内，那么整个月您的数据集中平均有 4.035GB
-
@selvakumar 它将收取 2 天的费用。
你可以用命令行工具做到这一点
bq show ds_name.table_name
它将显示有关表格的一些信息，包括“总字节数”。参考这里https://cloud.google.com/bigquery/bq-command-line-tool
【讨论】：
或者从 GUI 中，您可以使用元数据内部表 __TABLES__ ，例如这将为您提供以 GB 为单位的大小：
select
sum(size_bytes)/pow(10,9) as size
from
<your_dataset>.__TABLES__
where
table_id = '<your_table>'
【讨论】：
-
只是一个注释，Filter WHERE 可以省略，你会得到整个数据集的大小。
-
太好了，谢谢！我真的认为如果没有管理员身份，您将无法访问此元数据。您是否获得了 TABLES 中所有可用属性的列表？
-
抱歉，我可以用 SELECT * 回答我自己的问题。这些查询似乎检索 0 个可计费字节，因此它们是免费的。您可以获得：project_id、dataset_id、table_id、creation_time、last_modified_time、row_count、size_bytes、type。时间戳以 Unix 毫秒为单位。
-
令人惊讶的是我不得不依赖这个答案，并且搜索 Google 以计算 BigQuery 表大小会返回此页面，并且 GCP 没有关于此主题的官方文档
只需使用命令行工具。您将看到包含 Last-Modified、Schema、Total Rows、Total Bytes 等列的表格。
- 单表
bq show dataset_name.table_name
- 适用于所有表格
bq show dataset_name.*
【讨论】：