云作曲家任务无法创建 dataproc 集群答案

【问题标题】：cloud composer task unable to create dataproc cluster云作曲家任务无法创建 dataproc 集群
【发布时间】：2021-06-16 21:35:30
【问题描述】：

我正在尝试使用 Cloud Composer 运算符创建 dataproc 集群。这是我的 DAG 的样子：

default_dag_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': days_ago(1),
    'email': ['****************'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
  }
CLUSTER_CONFIG = {
    "master_config": {
        "num_instances": 1,
        "machine_type_uri": "n1-standard-4",
        "disk_config": {"boot_disk_type": "pd-standard", "boot_disk_size_gb": 10},
    },
    "worker_config": {
        "num_instances": 2,
        "machine_type_uri": "n1-standard-4",
        "disk_config": {"boot_disk_type": "pd-standard", "boot_disk_size_gb": 10},
    },
}

with models.DAG(
        'PanelSettings_dag',
        schedule_interval="@daily",
        default_args=default_dag_args) as dag:

    t1 = BashOperator(
        task_id='print_date',
        bash_command='date',
    )
    
    create_cluster = DataprocCreateClusterOperator(
        task_id="create_cluster",
        gcp_conn_id='google-dataproc',
        project_id=GCP_PROJECT_ID,
        cluster_config=CLUSTER_CONFIG,
        region=REGION,
        cluster_name=CLUSTER_NAME,
    )

我已经在气流上创建了 dataproc 连接，并为服务帐户赋予了 dataproc 管理员和存储管理员角色。如果没有这个连接，我会收到一个错误：

Getting connection using `google.auth.default()` since no key file is defined for hook.

现在我收到错误：

[2021-06-16 21:30:48,109] {taskinstance.py:1152} ERROR - 501 Received http2 header with status: 404
Traceback (most recent call last)
  File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 73, in error_remapped_callabl
    return callable_(*args, **kwargs
  File "/opt/python3.6/lib/python3.6/site-packages/grpc/_channel.py", line 923, in __call_
    return _end_unary_response_blocking(state, call, False, None
  File "/opt/python3.6/lib/python3.6/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blockin
    raise _InactiveRpcError(state
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with
    status = StatusCode.UNIMPLEMENTE
    details = "Received http2 header with status: 404
    debug_error_string = "{"created":"@1623879048.108981125","description":"Received http2 :status header with non-200 OK status","file":"src/core/ext/filters/http/client/http_client_filter.cc","file_line":129,"grpc_message":"Received http2 header with status: 404","grpc_status":12,"value":"404"}

我是气流新手。有人可以帮助调试这个。无法理解我做错了什么。

【问题讨论】：

您的 DAG 与我一起使用 us-central1 作曲家和 dataproc 的效果很好。我还将引导磁盘更改为 35GB，因为这是 dataproc 集群的最低要求。你是如何设置作曲家的？
看来我没有给出正确的区域。我正在进入区域字段中的区域。虽然对错误感到惊讶。

标签： google-cloud-dataproc google-cloud-composer airflow

【解决方案1】：

错误是在区域字段中输入区域名称。我纠正了它，它起作用了。如果错误提到“区域未找到/不存在”会有所帮助。

【讨论】：