Airflow DockerOperator 无法正确挂载 tmp 目录

Question

我正在尝试在使用 Airflow 安排的 docker run 命令中运行一个简单的 python 脚本。

我已按照Airflow init 此处的说明进行操作。

我的.env 文件：

AIRFLOW_UID=1000
AIRFLOW_GID=0

而docker-compose.yaml 是基于默认的docker-compose.yaml。我必须添加 - /var/run/docker.sock:/var/run/docker.sock 作为附加卷才能在 docker 内运行 docker。

我的dag配置如下：

""" this is an example dag """
from datetime import timedelta

from airflow import DAG

from airflow.operators.docker_operator import DockerOperator
from airflow.utils.dates import days_ago
from docker.types import Mount

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['info@foo.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 10,
    'retry_delay': timedelta(minutes=5),
}
with DAG(
    'msg_europe_etl',
    default_args=default_args,
    description='Process MSG_EUROPE ETL',
    schedule_interval=timedelta(minutes=15),
    start_date=days_ago(0),
    tags=['satellite_data'],
) as dag:

    download_and_store = DockerOperator(
        task_id='download_and_store',
        image='satellite_image:latest',
        auto_remove=True,
        api_version='1.41',
        mounts=[Mount(source='/home/archive_1/archive/satellite_data',
                      target='/app/data'),
                Mount(source='/home/dlassahn/projects/forecast-system/meteoIntelligence-satellite',
                      target='/app')],
        command="python3 src/scripts.py download_satellite_images "
                     "{{ (execution_date - macros.timedelta(hours=4)).strftime('%Y-%m-%d %H:%M') }} "
                     "'msg_europe' ",
    )

    download_and_store

气流日志：

[2021-08-03 17:23:58,691] {docker.py:231} INFO - Starting docker container from image satellite_image:latest
[2021-08-03 17:23:58,702] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/home/airflow/.local/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 319, in execute
    return self._run_image()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 258, in _run_image
    tty=self.tty,
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py", line 430, in create_container
    return self.create_container_from_config(config, name)
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/container.py", line 441, in create_container_from_config
    return self._result(res, True)
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/airflowtmp037k87u6")

由于未知的关键字参数 mount_tmp_dir，尝试将 mount_tmp_dir=False 设置为 Dag ImportError。 （这可能是文档的问题）

但是我不知道如何正确配置tmp目录。

我的气流版本：2.1.2

Answer 1

Docker Provider 2.0.0 中存在一个错误，导致 Docker Operator 无法使用 Docker-In-Docker 解决方案运行。

您需要升级到最新的 Docker Provider 2.1.0 https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/index.html#id1

您可以按照https://airflow.apache.org/docs/docker-stack/build.html#extending-the-image 中的描述扩展图像来做到这一点 - 例如 - 这个 docker 文件：

FROM apache/airflow
RUN pip install --no-cache-dir apache-airflow-providers-docker==2.1.0

在这种情况下，操作员将使用“回退”模式（和警告消息）开箱即用地工作，但您也可以禁用导致问题的挂载。更多解释来自https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html

默认情况下，在主机上创建一个临时目录并挂载 到一个容器中，以允许存储一起超过 容器中的默认磁盘大小为 10GB。在这种情况下，路径 可以通过环境变量访问挂载的目录 AIRFLOW_TMP_DIR。

如果无法安装卷，则会打印警告并尝试 在没有临时文件夹的情况下执行 docker 命令 安装。这是为了使其默认与远程 docker 引擎一起工作 或者当您运行 docker-in-docker 解决方案并且临时目录是 不与 docker 引擎共享。在此日志中打印警告 案例。

如果您知道您使用远程引擎或通过 docker-in-docker 您应该将 mount_tmp_dir 参数设置为 False。在 在这种情况下，您仍然可以使用 mounts 参数来挂载 Docker 引擎中现有的命名卷以实现类似 您可以存储超过默认磁盘大小的文件的功能 容器，