如何设置适当的权限以在 Cloud Composer 中运行 KubernetesPodOperator？

Question

我正在尝试按照此处的文档在我的 Composer 环境中运行一个简单的 KubernetesPodOperator。

由于缺少用户“默认”的权限，气流运行时失败。

也就是说，如何正确创建环境或设置默认用户权限以使此代码正常工作？

DAG：

    price_analysis = KubernetesPodOperator(
        task_id='price-analysis',
        name='price-analysis',
        namespace='default',
        image='bash',
        image_pull_policy='Always',
        cmds=['echo'],
        arguments=['something'],
        env_vars={
            'EXPOSURE_THRESHOLD': '5',
            'ESTIMATE_WINDOW': '3,7',
        },
        in_cluster=True,
    )

日志：

-------------------------------------------------------------------------------
Starting attempt 1 of 
-------------------------------------------------------------------------------

[2019-04-03 14:54:15,611] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): price-analysis> on 2019-04-03T14:53:59.658367+00:00
[2019-04-03 14:54:15,612] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run vat-analysis price-analysis 2019-04-03T14:53:59.658367+00:00 --job_id 54 --raw -sd DAGS_FOLDER/vat_analysis_dag.py --cfg_path /tmp/tmp3RdZOV']
[2019-04-03 14:54:18,375] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:18,374] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-04-03 14:54:19,652] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,651] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2019-04-03 14:54:19,659] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,659] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-04-03 14:54:19,826] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,825] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2019-04-03 14:54:19,842] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,842] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:19,868] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:20,380] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:20,378] {models.py:271} INFO - Filling up the DagBag from /home/airflow/gcs/dags/vat_analysis_dag.py
[2019-04-03 14:54:21,490] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,490] {cli.py:484} INFO - Running <TaskInstance: vat-analysis.price-analysis 2019-04-03T14:53:59.658367+00:00 [running]> on host airflow-worker-5b6d7c75c9-w6995
[2019-04-03 14:54:22,093] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,822] {pod_launcher.py:58} ERROR - Exception when attempting to create Namespaced Pod.
[2019-04-03 14:54:22,103] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis Traceback (most recent call last):
[2019-04-03 14:54:22,107] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
[2019-04-03 14:54:22,113] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
[2019-04-03 14:54:22,116] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
[2019-04-03 14:54:22,122] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
[2019-04-03 14:54:22,126] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
[2019-04-03 14:54:22,129] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     collection_formats=collection_formats)
[2019-04-03 14:54:22,134] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
[2019-04-03 14:54:22,150] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     _return_http_data_only, collection_formats, _preload_content, _request_timeout)
[2019-04-03 14:54:22,155] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis   File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
[2019-04-03 14:54:22,159] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis     _request_timeout=_request_timeout)
[2019-04-03 14:54:22,138] {models.py:1760} ERROR - (403
Reason: Forbidde
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Apr 2019 14:54:21 GMT', 'Audit-Id': 'c027d4cb-5186-498a-a9b5-0e6c4420b816', 'Content-Length': '284', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:composer-1-6-0-airflow-1-10-1-ea0745b4:default\" cannot create pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403

Answer 1

我收到了来自 Google 网上论坛中 Google Composer 讨论组的回复。 一个细节：服务帐户 default:default 必须与您在错误消息中看到的服务帐户相同（在我的情况下是composer-1-6-0-airflow-1-10-1-ea0745b4:default）。

CLUSTER_NAME=.....
NAMESPACE=k8s-tasks
kubectl create ns ${NAMESPACE}

kubectl create clusterrolebinding default-admin \
    --clusterrole cluster-admin \
    --serviceaccount=default:default \
    --namespace ${NAMESPACE}