使用 KubernetesPodOperator 在 Kubernetes pod 内运行 Docker 映像的云作曲家答案

【问题标题】：Cloud composer running Docker image inside Kubernetes pod using KubernetesPodOperator使用 KubernetesPodOperator 在 Kubernetes pod 内运行 Docker 映像的云作曲家
【发布时间】：2022-09-29 16:42:08
【问题描述】：

我想通过 Cloud Composer 在 Kubernetes pod 中运行一个 docker，为此我遵循了所有提到的步骤here 并验证了绑定，我使用了默认服务帐户以及 Kubernetes pod 的默认命名空间。

我正在尝试访问 Cloud Composer 中的 Form API--> KubernetesPodOperator --> Docker 映像（访问 Google Form API 的应用程序）这可以提取图像甚至运行它，但是，当它尝试访问表单 API 时，收到以下错误：

[2022-09-23 16:17:49,839] {pod_manager.py:197} INFO - java.io.IOException: Unexpected Error code 403 trying to get security access token from Compute Engine metadata for the default service account: Missing required header \"Metadata-Flavor\": \"Google\"
[2022-09-23 16:17:49,840] {pod_manager.py:197} INFO - 
[2022-09-23 16:17:49,840] {pod_manager.py:197} INFO -   at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:75) ~[cli.jar:?]
[2022-09-23 16:17:49,841] {pod_manager.py:197} INFO -   at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:76) ~[cli.jar:?]
[2022-09-23 16:17:49,841] {pod_manager.py:197} INFO -   at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:53) ~[cli.jar:?]
[2022-09-23 16:17:49,841] {pod_manager.py:197} INFO -   at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:40) ~[cli.jar:?]
[2022-09-23 16:17:49,842] {pod_manager.py:197} INFO -   at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91) ~[cli.jar:?]
[2022-09-23 16:17:49,842] {pod_manager.py:197} INFO -   at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:415) ~[cli.jar:?]
[2022-09-23 16:17:49,843] {pod_manager.py:197} INFO -   at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:525) ~[cli.jar:?]
[2022-09-23 16:17:49,844] {pod_manager.py:197} INFO -   at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:466) ~[cli.jar:?]
[2022-09-23 16:17:49,845] {pod_manager.py:197} INFO -   at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:576) ~[cli.jar:?]

DAG sn-p:

kubernetes_min_pod = KubernetesPodOperator(
        task_id=\'google-form-importer\',
        name=\'google-form-importer\',
        namespace=\'default\',
        service_account_name=\'default\',
        image=\'pulling to be pulled\',
        arguments=[\"An_argument\"],
        get_logs=True,
        image_pull_policy=\'Always\',
        log_events_on_failure=True,
        env_vars= {
\'ALL_ENVS\':\'ItsValue\'
        }
)

谢谢您的帮助！

标签： airflow service-accounts docker-container google-cloud-composer kubernetespodoperator

【解决方案1】：

更新到最新的 API 依赖项解决了错误：

    <dependency>
      <groupId>com.google.api-client</groupId>
      <artifactId>google-api-client</artifactId>
      <version>2.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.google.apis</groupId>
      <artifactId>google-api-services-forms</artifactId>
      <version>v1-rev20220329-2.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.google.auth</groupId>
      <artifactId>google-auth-library-oauth2-http</artifactId>
      <version>1.11.0</version>
    </dependency>

之前的版本分别是 1.35.2、v1-rev20220329-1.32.1、0.1.0。

【讨论】：