Airflow BigQueryOperator 标签问题：标签值包含无效字符答案

【问题标题】：Airflow BigQueryOperator Label issue: Label value has invalid charactersAirflow BigQueryOperator 标签问题：标签值包含无效字符
【发布时间】：2020-09-19 13:00:36
【问题描述】：

我一直在尝试利用 Airflow macros reference default variables 和 BigQuery 标签来记录通过 Airflow BigQueryOperator 提交的查询的元数据。这是运算符定义：

BigQuery_Labels_Test_Task = BigQueryOperator(
    task_id="BigQuery_Labels_Test_Task",
    bql="SELECT 1",
    use_legacy_sql=False,
    bigquery_conn_id="gcp_bq_connection",
    destination_dataset_table=f"test_dataset.test_table",
    create_disposition="CREATE_IF_NEEDED",
    write_disposition="WRITE_TRUNCATE",
    labels={
        "dag_id": "{{ dag.dag_id }}",
        "task_id": "{{ task.task_id }}",
        "run_id": "{{ run_id }}",
    },
    dag=dag,
)

但是当它执行时会抛出以下错误：

[2020-06-01 08:15:13,495] {{taskinstance.py:887}} INFO - Executing <Task(BigQueryOperator): BigQuery_Labels_Test_Task> on 2020-06-01T07:51:40.752935+00:00
[2020-06-01 08:15:13,499] {{standard_task_runner.py:53}} INFO - Started process 16317 to run task
[2020-06-01 08:15:13,567] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: BigQuery_Labels_Test_DAG.BigQuery_Labels_Test_Task 2020-06-01T07:51:40.752935+00:00 [running]> b438a71d2d52
[2020-06-01 08:15:13,592] {{bigquery_operator.py:255}} INFO - Executing: SELECT 1
[2020-06-01 08:15:14,161] {{taskinstance.py:1128}} ERROR - <HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/test-project/jobs?alt=json returned "Label value "manual__2020-06-01T07:51:40.752935+00:00" has invalid characters.">
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 966, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/bigquery_operator.py", line 282, in execute
    encryption_configuration=self.encryption_configuration
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 910, in run_query
    return self.run_with_configuration(configuration)
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 1318, in run_with_configuration
    .execute(num_retries=self.num_retries)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/googleapiclient/http.py", line 907, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/test-project/jobs?alt=json returned "Label value "manual__2020-06-01T07:51:40.752935+00:00" has invalid characters.">
[2020-06-01 08:15:14,165] {{taskinstance.py:1151}} INFO - Marking task as UP_FOR_RETRY

有没有人遇到过这样的事情？ bq中的label字段有字符限制吗？

PS：当我对标签值进行如下硬编码时，它可以工作。：

labels={
    "dag_id": "dag_id",
    "task_id": "task_id",
},

只有小写值也适用于硬编码。

【问题讨论】：

这个答案对你有用吗？如果是，请接受它，以便它对面临类似问题的人更明显和有帮助。

标签： python google-bigquery jinja2 airflow

【解决方案1】：

此问题与气流无关，但与 BigQuery 有关。对于定义标签，run_id 不满足以下要求（run_id 有 +,:)：

键和值只能包含小写字母、数字字符、下划线和破折号。所有字符必须使用 UTF-8 编码，允许使用国际字符。

有关 BigQuery 标签的更多信息，请参阅this。

【讨论】：