【问题标题】:uWSGI and joblib Semaphore: Joblib will operate in serial modeuWSGI 和 joblib 信号量:Joblib 将以串行模式运行
【发布时间】:2019-02-18 16:12:51
【问题描述】:

我在 Docker 容器内的 Flask 应用程序中运行 joblib,以及由 supervisord 启动的 uWSGI(从启用线程开始)。

网络服务器启动显示如下错误:

unable to load configuration from from multiprocessing.semaphore_tracker import main;main(15)
/usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/_multiprocessing_helpers.py:38: UserWarning:

[Errno 32] Broken pipe.  joblib will operate in serial mode

知道如何解决这个问题并让 joblib 并行运行吗?谢谢!


以下软件包安装在 docker 容器中:

pytest==4.0.1
pytest-cov==2.6.0
flake8==3.6.0
Cython==0.29.3
numpy==1.16.1
pandas==0.24.0
scikit-learn==0.20.2
fancyimpute==0.4.2
scikit-garden==0.1.3
category_encoders==1.3.0
boto3==1.9.86
joblib==0.13.1
dash==0.37.0
dash-renderer==0.18.0
dash-core-components==0.43.1
dash-table==3.4.0
dash-html-components==0.13.5
dash-auth==1.3.2
Flask-Caching==1.4.0
plotly==3.6.1
APScheduler==3.5.3

编辑

问题是由于 uWSGI、nginx 或 supervisord 造成的。 dev/shm 缺少权限不是问题,因为如果我直接运行烧瓶服务器,可以创建信号量。在下面找到三个服务的配置文件。免责声明,我是网络服务器菜鸟,配置是通过从不同的博客复制和粘贴而诞生的,只是为了让它工作:-D

这是我的 uwsgi 配置:

[uwsgi]
module = prism_dash_frontend.__main__
callable = server

uid = nginx
gid = nginx

plugins = python3

socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664

# set cheaper algorithm to use, if not set default will be used
cheaper-algo = spare

# minimum number of workers to keep at all times
cheaper = 3

# number of workers to spawn at startup
cheaper-initial = 5

# maximum number of workers that can be spawned
workers = 5

# how many workers should be spawned at a time
cheaper-step = 1
processes = 5

die-on-term = true
enable-threads = true

nginx 配置:

# based on default config of nginx 1.12.1
# Define the user that will own and run the Nginx server
user nginx;
# Define the number of worker processes; recommended value is the number of
# cores that are being used by your server
# auto will default to number of vcpus/cores
worker_processes auto;

# altering default pid file location
pid /tmp/nginx.pid;

# turn off daemon mode to be watched by supervisord
daemon off;

# Enables the use of JIT for regular expressions to speed-up their processing.
pcre_jit on;

# Define the location on the file system of the error log, plus the minimum
# severity to log messages for
error_log /var/log/nginx/error.log warn;

# events block defines the parameters that affect connection processing.
events {
    # Define the maximum number of simultaneous connections that can be opened by a worker process
    worker_connections  1024;
}


# http block defines the parameters for how NGINX should handle HTTP web traffic
http {
    # Include the file defining the list of file types that are supported by NGINX
    include /etc/nginx/mime.types;
    # Define the default file type that is returned to the user
    default_type text/html;

    # Don't tell nginx version to clients.
    server_tokens off;

    # Specifies the maximum accepted body size of a client request, as
    # indicated by the request header Content-Length. If the stated content
    # length is greater than this size, then the client receives the HTTP
    # error code 413. Set to 0 to disable.
    client_max_body_size 0;

    # Define the format of log messages.
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';

    # Define the location of the log of access attempts to NGINX
    access_log /var/log/nginx/access.log  main;

    # Define the parameters to optimize the delivery of static content
    sendfile       on;
    tcp_nopush     on;
    tcp_nodelay    on;

    # Define the timeout value for keep-alive connections with the client
    keepalive_timeout  65;

    # Define the usage of the gzip compression algorithm to reduce the amount of data to transmit
    #gzip  on;

    # Include additional parameters for virtual host(s)/server(s)
    include /etc/nginx/conf.d/*.conf;
}

supervisord 配置:

[supervisord]
nodaemon=true

[program:uwsgi]
command=/usr/bin/uwsgi --ini /etc/uwsgi/uwsgi.ini
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:nginx]
command=/usr/sbin/nginx
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

第二次编辑

从 Python 3.5 迁移到 3.7.2 后,错误的性质略有变化:

unable to load configuration from from multiprocessing.semaphore_tracker import main;main(15)
/usr/local/lib/python3.7/multiprocessing/semaphore_tracker.py:55: UserWarning:

semaphore_tracker: process died unexpectedly, relaunching.  Some semaphores might leak.

unable to load configuration from from multiprocessing.semaphore_tracker import main;main(15)

非常感谢您的帮助,目前这对我来说是一个很大的障碍:-/


第三次编辑:

HERE on my github account 是一个最小、完整且可验证的示例。

您可以通过以下方式轻松运行它 make build 后跟 make run。

它将显示以下日志消息:

unable to load configuration from from multiprocessing.semaphore_tracker import main;main(14)

一旦你访问http://127.0.0.1:8080/就会崩溃,并出现以下错误:

exception calling callback for <Future at 0x7fbc520c7eb8 state=finished raised TerminatedWorkerError>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/joblib/externals/loky/_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 309, in __call__
    self.parallel.dispatch_next()
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 731, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/usr/local/lib/python3.7/site-packages/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/usr/local/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 510, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "/usr/local/lib/python3.7/site-packages/joblib/externals/loky/reusable_executor.py", line 151, in submit
    fn, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 1022, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {EXIT(1), EXIT(1), EXIT(1), EXIT(1)}

【问题讨论】:

  • 你能提供一张minimal reproducible example 的图片吗?
  • 这是个好主意,今天晚些时候或明天会提供一个。
  • 我添加了一个托管在 github 上的示例。有点太大了,无法以文本形式显示在这里。
  • 这是一个错误,因为 uwsgi 二进制文件嵌入了 Python 解释器,sys.executable 指向uwsgi 二进制文件而不是通常的 python 二进制文件。可以修复 Loky 以检测这种情况并尝试查找 python 命令行而不是天真地使用 sys.executable:github.com/tomMoral/loky/issues/207 随意提交 PR :) 可以为标准中的 concurrent.futures 模块提供类似的修复Python 库。

标签: python docker flask uwsgi joblib


【解决方案1】:

这真是个兔子洞。

Github 上的 joblib 问题页面有类似的关于 joblib 因 Uwsgi 失败的帖子。但大多数是针对较旧的multiprocessing 后端。新的loky 后端应该可以解决这些问题。

PR 用于 multiprocessing 后端解决了 uwsgi 的这个问题:

joblib.Parallel(n_jobs=4,backend="multiprocessing")(joblib.delayed(sqrt)(i ** 2) for i in range(10))

但它有时会随机失败并退回到上面的 PR 试图解决的相同问题。

进一步挖掘显示,当前后端loky 默认情况下并行处理进程 (docs)。但是这些进程没有共享内存访问,因此需要序列化和排队的通道。这可能是 uWSGI 失败而 gunicorn 工作的原因。

所以我尝试切换到线程而不是进程:

joblib.Parallel(n_jobs=4,prefer="threads")(joblib.delayed(sqrt)(i ** 2) for i in range(10))

它有效:)

【讨论】:

  • 非常感谢 manish,帮助我找到了一个非常相似的issue 的解决方法。有什么方法可以说服依赖项中使用的 joblib 使用prefer="threads"?
  • 这比你的回答过了一年多,但joblib中仍然存在问题。我也有同样的随机故障问题。现在,我尝试使用多处理后端,它似乎正在工作。使用线程如何帮助/工作?特别是因为 Python 本身不是多线程的,但更适合多处理?
【解决方案2】:

嗯,我确实找到了问题的答案。它解决了能够在 docker 中使用 supervisor 和 nginx 运行 joblib 依赖库的问题。但是,它并不是很令人满意。因此,我不会接受我自己的答案,但我将其发布在这里,以防其他人遇到同样的问题并需要找到 okayish 修复。

解决方案是用 gunicorn 替换 uWSGI。好吧,至少我现在知道这是谁的错。我仍然很感激使用 uWSGI 代替 gunicorn 来解决问题的答案。

【讨论】:

  • 你能验证joblib是否需要同步线程来工作吗? uWSGI 在异步线程上创建工作线程,这会给需要同步特性的应用程序带来问题。我在部署 tensorflow 模型时遇到了同样的问题。 Gunicorn 默认支持同步工作程序,自然解决方案也是如此。
  • 如何验证或检查 joblib 是否需要同步线程?
  • 正确的解决方案是改变 loky 来实现:github.com/tomMoral/loky/issues/207#issuecomment-485137208
【解决方案3】:

您的图像似乎未启用信号:Joblib 检查multiprocessing.Semaphore(),并且它只有root 对/dev/shm 中的共享内存具有读/写权限。 看看this question 和this answer。

这是在我的一个容器中运行的。

$ ls -ld /dev/shm
drwxrwxrwt 2 root root 40 Feb 19 15:23 /dev/shm

如果您以非 root 身份运行,则应更改 /dev/shm 的权限。要设置正确的权限,您需要修改 Docker 镜像中的/etc/fstab:

none /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0

【讨论】:

  • 知道怎么做吗?如果做RUN echo "none /dev/shm tmpfs rw,nosuid,nodev,noexec 0 0" &gt; /etc/fstab 和RUN ls -ld /dev/shm 我仍然得到drwxrwxrwt 2 root root 40 Feb 21 09:21 /dev/shm
  • 你能尝试在你的容器中编译和运行这个program吗?使用gcc -lrt -pthread main.c 编译。我想了解这是系统问题还是特定于某些库。
  • 这很奇怪。如果您在/dev/shm/ 上获得了drwxrwxrwt 权限,您应该能够以非root 身份访问它。
  • 在 docker 容器中编译和运行你的程序会得到Sample data written Value of mySem=1
  • 肯定是因为uWSGI、nginx或者supverisord。如果我在容器中以调试模式运行烧瓶服务器,并且 joblib 可以完美运行。我会将配置添加到我的主帖中。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2021-09-19
  • 1970-01-01
相关资源
最近更新 更多