Bigquery 无法从 Google Cloud Storage 加载数据

Question

我有一个 json 记录，看起来像

{"customer_id":"2349uslvn2q3","order_id":"9sufd23rdl40",
 "line_item": [{"line":"1","sku":"10","amount":10},
               {"line":"2","sku":"20","amount":20}]}

我正在尝试将上述记录加载到具有架构定义的表中，

"fields": [
  {
    "mode": "NULLABLE",
    "name": "customer_id",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "order_id",
    "type": "STRING"
  },
  {
    "mode": "REPEATED",
    "name": "line_item",
    "type": "STRING"
  }
]

我收到以下错误“消息”：

文件中从位置 0 开始的行中的 JSON 解析错误： gs://gcs_bucket/file0.为非记录字段指定的 JSON 对象： line_item

我想要line_item json 字符串，它可以有超过 1 行作为表中行项目列中的 json 字符串数组。

有什么建议吗？

Answer 1

第一件事是你的输入 JSON 不应该有“\n”字符，所以你应该把它保存为：

{"customer_id":"2349uslvn2q3","order_id":"9sufd23rdl40", "line_item": [{"line":"1","sku":"10","amount":10}, {"line":"2","sku":"20","amount":20}]}

您的 JSON 文件应如下所示的一个示例：

{"customer_id":"2349uslvn2q3","order_id":"9sufd23rdl40", "line_item": [{"line":"1","sku":"10","amount":10}, {"line":"2","sku":"20","amount":20}]}
{"customer_id":"2","order_id":"2", "line_item": [{"line":"2","sku":"20","amount":20}, {"line":"2","sku":"20","amount":20}]}
{"customer_id":"3","order_id":"3", "line_item": [{"line":"3","sku":"30","amount":30}, {"line":"3","sku":"30","amount":30}]}

而且您的架构也不正确。应该是：

[
  {
    "mode": "NULLABLE",
    "name": "customer_id",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "order_id",
    "type": "STRING"
  },
  {
    "mode": "REPEATED",
    "name": "line_item",
    "type": "RECORD",
    "fields": [{"name": "line", "type": "STRING"}, {"name": "sku", "type": "STRING"}, {"name": "amount", "type": "INTEGER"}]
  }
]

为了更好地理解模式的工作原理，我尝试在 this answer 中编写指南。希望它可以有一些价值。

如果您的数据内容保存在一个名为 gs://gcs_bucket/file0 的文件中，而您的架构在 schema.json 中，那么此命令应该适合您：

bq load --source_format=NEWLINE_DELIMITED_JSON dataset.table gs://gcs_bucket/file0 schema.json

（假设您正在使用 CLI 工具，因为您的问题似乎就是这种情况）。