全新安装 Apache Airflow 2.2.3 .. OOps 发生了一些不好的事情

Question

我已经通过 pip 成功地在本地安装了 apache 气流 .. 有几个需要的引脚

pip3 install zipp==3.1.0
pip3 install sqlalchemy==1.3.24
python3 -m pip install virtualenv
pip3 install apache-airflow[cncf.kubernetes]

pip3 install apache-airflow

因为我是n00b，所以我从基础开始。我第一次尝试airflow standalone，但在文档中没有说明默认用户名和密码是什么...所以我进去了，只是用一个基本用户启动了一些服务......

airflow db init
airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin

现在这只是需要启动..我意识到我需要启动调度程序和 webapp...由于某种原因我的自动脚本没有这样做..我必须手动执行但是..

airflow scheduler &
airflow webapp

现在一切都好了..我可以看到我有一个启动 GUI 并正在运行..一切似乎都很好..我想启动我发现的第一个 DAG

example_bash_operator

问题是...当我单击名称时...

Python version: 3.8.10
Airflow version: 2.2.3
Node: juju-2dd159-310.lxd
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 51, in decorated
    return func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/decorators.py", line 72, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/views.py", line 1732, in trigger
    if unpause and dag.is_paused:
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 1081, in is_paused
    warnings.warn(
  File "/usr/lib/python3.8/warnings.py", line 109, in _showwarnmsg
    sw(msg.message, msg.category, msg.filename, msg.lineno,
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/settings.py", line 117, in custom_show_warning
    write_console.print(msg, soft_wrap=True)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1642, in print
    self._buffer.extend(new_segments)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 842, in __exit__
    self._exit_buffer()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 800, in _exit_buffer
    self._check_buffer()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1935, in _check_buffer
    self.file.flush()
BrokenPipeError: [Errno 32] Broken pipe

如果我忽略了这一点，也许稍等片刻，或者只是再试一次.. 突然它起作用了......有什么线索可以让这种体验变得顺畅吗？

编辑：如果这有助于回答问题

ubuntu@juju-2dd159-311:~$ pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
ubuntu@juju-2dd159-311:~$ python3 --version
Python 3.8.10

编辑#2

我按照这些说明安装了他们想要的约束https://airflow.apache.org/docs/apache-airflow/stable/start/local.html

这极大地修复了 GUI 和 UI 的稳定性.. .但是...然后我开始连接 postgresql 数据库...现在它甚至不会在没有brokenpipe 错误的情况下登录

Python version: 3.8.10
Airflow version: 2.2.3
Node: juju-2dd159-318.lxd
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 51, in decorated
    return func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/views.py", line 718, in index
    paging=wwwutils.generate_pages(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/www/utils.py", line 113, in generate_pages
    previous_node = Markup(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/jinja2/utils.py", line 838, in __new__
    warnings.warn(
  File "/usr/lib/python3.8/warnings.py", line 109, in _showwarnmsg
    sw(msg.message, msg.category, msg.filename, msg.lineno,
  File "/home/ubuntu/.local/lib/python3.8/site-packages/airflow/settings.py", line 117, in custom_show_warning
    write_console.print(msg, soft_wrap=True)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1642, in print
    self._buffer.extend(new_segments)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 842, in __exit__
    self._exit_buffer()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 800, in _exit_buffer
    self._check_buffer()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/rich/console.py", line 1935, in _check_buffer
    self.file.flush()

Answer 1

您是否尝试按照“快速入门”说明进行操作？

https://airflow.apache.org/docs/apache-airflow/stable/start/index.html

Airflow 提供了关于如何开始的很好且全面的说明，如果您按部就班地进行操作，您将启动并运行 Airlfow。这可以通过 docker compose 或本地 virtualenv 来完成。

您的问题可能是缺乏资源 - 内存（很可能）。 Airflow 需要相当多的内存（4GB）才能启动，因为它是一个复杂的系统。它是作为先决条件编写的，尤其是在 Docker Compose 快速入门中。如果您没有足够的资源，Docker Compose 甚至会警告您，所以如果您想要真正可靠和强大的快速入门，我推荐这个。

您需要查看您的日志，以了解您出现管道错误的原因。但最可能的原因是缺乏资源。

关于“独立”模式和用户密码 - 您可能错过了气流写给您的内容。它在启动时动态生成密码，并实际告诉您应该使用什么密码：

standalone | 
standalone | Airflow is ready
standalone | Login with username: admin  password: 4hfH8mATcvMFmne9
standalone | Airflow Standalone is for development purposes only. Do not use this in production!
standalone | 

Answer 2

事实证明.. 编写一个简单的脚本在后台运行 apache 气流

#!/bin/bash
airflow webserver -D

在 Ubuntu LTS 20.04 中无法正常工作或无法正常运行。我现在发现我应该让 SystemD 处理启动停止。现在它工作得非常好。这就是我注入所需的方法Ubuntu 20.04 LTS 中的脚本

这会注册网络服务器、调度程序和触发器服务

if [ ! -e /etc/systemd/system/airflow-scheduler.service ]; then
  cat <<EOT >> /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon

[Service]
Environment="PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
User=ubuntu
Type=simple
ExecStart=/home/ubuntu/.local/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOT
fi

if [ ! -e /etc/systemd/system/airflow-webserver.service ]; then
  cat <<EOT >> /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon

[Service]
Environment="PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
User=ubuntu
Type=simple
ExecStart=/home/ubuntu/.local/bin/airflow webserver
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOT
fi

if [ ! -e /etc/systemd/system/airflow-triggerer.service ]; then
  cat <<EOT >> /etc/systemd/system/airflow-triggerer.service
[Unit]
Description=Airflow triggerer daemon

[Service]
Environment="PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
User=ubuntu
Type=simple
ExecStart=/home/ubuntu/.local/bin/airflow triggerer
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOT
fi

systemctl daemon-reload
systemctl enable airflow-scheduler
systemctl enable airflow-webserver
systemctl enable airflow-triggerer

那我就开始吧

systemctl start airflow-webserver
systemctl start airflow-scheduler
systemctl start airflow-triggerer