【问题标题】:Unable to create SparkApplications on Kubernetes cluster using SparkKubernetesOperator from Airflow DAG (Airflow version 2.0.2 MWAA)无法使用 Airflow DAG 中的 SparkKubernetesOperator 在 Kubernetes 集群上创建 SparkApplications(Airflow 版本 2.0.2 MWAA)
【发布时间】:2023-03-15 05:57:02
【问题描述】:

我尝试使用 SparkKubernetesOperator 使用与以下问题相同的 DAG 和 yaml 文件在 Kubernetes 中运行 spark 作业:

Unable to create SparkApplications on Kubernetes cluster using SparkKubernetesOperator from Airflow DAG

但气流显示以下错误:

HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e2e1833d-a1a6-40d4-9d05-104a32897deb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 10 Sep 2021 08:38:33 GMT', 'Content-Length': '462'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the object provided is unrecognized (must be of type SparkApplication): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" } (222f7573722f6c6f63616c2f616972666c6f772f646167732f636f6e6669 ...)","reason":"BadRequest","code":400}

有解决这个问题的建议吗???

【问题讨论】:

    标签: apache-spark kubernetes airflow


    【解决方案1】:

    觉得你和我有同样的问题

    SparkKubernetesOperator(
      task_id='spark_pi_submit',
      namespace="default",
      application_file=open("/opt/airflow/dags/repo/script/spark-test.yaml").read(), #officially know bug
      kubernetes_conn_id="kubeConnTest", #ns default in airflow connection UI
      do_xcom_push=True,
      dag=dag
    )
    

    我是这样包裹的。 它就像魅力一样工作

    https://github.com/apache/airflow/issues/17371

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2019-05-02
      • 1970-01-01
      • 2022-07-12
      • 1970-01-01
      • 2020-10-10
      • 1970-01-01
      • 2021-10-15
      • 2018-08-05
      相关资源
      最近更新 更多