使用 Nginx、uWSGI 和 web.py 的 MemoryError答案

【问题标题】：MemoryError using Nginx, uWSGI, and web.py使用 Nginx、uWSGI 和 web.py 的 MemoryError
【发布时间】：2016-10-15 12:44:39
【问题描述】：

我有一个 python Apple MDM 服务器，我正在尝试将它从命令行进程运行转换为使用 Nginx 和 uWSGI 托管。我正在使用 web.py 进行路由。一切都很顺利。我已经有了我认为对 Nginx 和 uWSGI 来说是一个很好的基本配置，但是今天早上在运行它不到 24 小时后，我在服务器收到的每个其他请求上都会收到这个错误。有大量可用内存，我使用的是 64 位版本的 python。

EDIT1：SSL 正在使用端口 8080，因为它前面有一个负载平衡器，将 SSL 流量路由到该端口。

EDIT2：我已将 uWSGI 中的进程设置从 3 更改为 1，看看是否是问题所在。

EDIT3：这是来自 AWS 的新盒子。安装了我的服务器及其依赖项、nginx 和 uWSGI。我在将 uWSGI 作为服务运行时遇到了麻烦，我安装和卸载了它的 aptitude 和 pip 版本，然后我最终得到了运行 systemd 的 pip 版本。这就是它现在的运行方式，它正在优雅地停止、启动和重新启动。

[ Tue Jun 14 2016 10:29:22 ] Traceback (most recent call last):
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 565, in processor
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 77, in reload_mapping
MemoryError
[ Tue Jun 14 2016 10:29:22 ] 
[ Tue Jun 14 2016 10:29:22 ] Traceback (most recent call last):
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 566, in processor
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in <lambda>
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 246, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 477, in internalerror
MemoryError
[ Tue Jun 14 2016 10:29:22 ] 
[ Tue Jun 14 2016 10:29:22 ] Traceback (most recent call last):
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 581, in processor
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in <lambda>
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 246, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 477, in internalerror
MemoryError
[ Tue Jun 14 2016 10:29:22 ] 
[ Tue Jun 14 2016 10:29:22 ] Traceback (most recent call last):
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 566, in processor
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 237, in <lambda>
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 246, in process
  File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 477, in internalerror
MemoryError
[ Tue Jun 14 2016 10:29:22 ] 
[ Tue Jun 14 2016 10:29:22 ] Traceback (most recent call last):
[ Tue Jun 14 2016 10:29:22 ]   File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 279, in wsgi
[ Tue Jun 14 2016 10:29:22 ]     [ Tue Jun 14 2016 10:29:22 ] result = self.handle_with_processors()
[ Tue Jun 14 2016 10:29:22 ]   File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 249, in handle_with_processors
[ Tue Jun 14 2016 10:29:22 ]     [ Tue Jun 14 2016 10:29:22 ] return process(self.processors)
[ Tue Jun 14 2016 10:29:22 ]   File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 246, in process
[ Tue Jun 14 2016 10:29:22 ]     [ Tue Jun 14 2016 10:29:22 ] raise self.internalerror()
[ Tue Jun 14 2016 10:29:22 ]   File "/path/to/app/virtualenv/local/lib/python2.7/site-packages/web/application.py", line 477, in internalerror
[ Tue Jun 14 2016 10:29:22 ]     [ Tue Jun 14 2016 10:29:22 ] import debugerror
[ Tue Jun 14 2016 10:29:22 ] MemoryError[ Tue Jun 14 2016 10:29:22 ]

nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}
#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
# 
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
# 
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

启用站点

server {
    listen 80;

    location ^~ /static/ {
        root /path/to/app/;
    }

    location / {
        include   uwsgi_params;
        uwsgi_pass  127.0.0.1:9000;
    }
}
server {
    listen   8080;

    ssl    on;
    ssl_certificate    /path/to/app/certs/ssl.crt;
    ssl_certificate_key    /path/to/app/certs/ssl.key;

    location ^~ /static/ {
        root /path/to/app/static/;
    }

    location / {
        include   uwsgi_params;
        uwsgi_pass  127.0.0.1:9000;
        uwsgi_param UWSGI_CHDIR /path/to/app/;
        uwsgi_param UWSGI_PYHOME /path/to/app/virtualenv/;
        uwsgi_param UWSGI_SCRIPT server;
    }
}

uWSGI 初始化

home = /path/to/app/virtualenv/
chdir = /path/to/app/
module = server:application
logto = /var/log/uwsgi/%n.log
rlogdate =  [ %%a %%b %%d %%Y %%H:%%M:%%S ]
logformat = %(method) %(status) %(addr)  %(uri) wid: %(wid) 
log-encoder = format ${strftime:[ %%a %%b %%d %%Y %%H:%%M:%%S ]} ${msgnl}
gid = ubuntu
uid = ubuntu
vhost = false 
socket = 127.0.0.1:9000
master = true
processes = 3
harakiri = 20
limit-as = 128
memory-report
no-orphans

有人告诉我，我有一个相当严重的意外 SysAdmin 综合症病例，因此感谢您提供任何和所有帮助/建议。只是为了更好的衡量，这是proc/meminfo的输出

MemTotal:        7657772 kB
MemFree:         5971508 kB
MemAvailable:    7145084 kB
Buffers:          225528 kB
Cached:           900756 kB
SwapCached:            0 kB
Active:          1142912 kB
Inactive:         358288 kB
Active(anon):     380568 kB
Inactive(anon):    11956 kB
Active(file):     762344 kB
Inactive(file):   346332 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                48 kB
Writeback:             0 kB
AnonPages:        375008 kB
Mapped:            55840 kB
Shmem:             17616 kB
Slab:             129300 kB
SReclaimable:     106084 kB
SUnreclaim:        23216 kB
KernelStack:        3120 kB
PageTables:         6672 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3828884 kB
Committed_AS:     602572 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    143360 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       71680 kB
DirectMap2M:     7923712 kB

【问题讨论】：

真的很神秘。 64 位操作系统和 python，启用了 harakiri。超限？
我对 ulimit 不熟悉，但我阅读了一些关于它的内容并运行了命令。它返回unlimited。所以，是的，我没有限制可用的资源或任何东西。添加另一个编辑，其中包含更多可能相关的信息。
啊，我想我明白了。您没有在 uWSGI .ini 中设置 max-requests。因此，您的 python 解释器永远存在并且可能会积累垃圾。尝试将其设置为某些东西，例如5000，希望泄漏会停止。
这是有道理的。我会测试几天，看看我是否再次收到错误。谢谢你，先生。
是的，看起来就是这样，@Bob。我让它运行了一段时间，工人 3 再次收到错误，1 和 2 没有。我现在打开了指标，工人 3 高于 4400 个请求，而其他两个低于 1000 个。未设置最大请求似乎是问题，但看起来我必须将其设置为 3500 左右~ 4000 而不是建议的 5000。至少在我的开发环境中。 prod 服务器有更多的资源。如果您发布答案，我会接受。

标签： python nginx uwsgi web.py mdm

【解决方案1】：

你没有在你的 uWSGI .ini 中设置max-requests。因此，您的 python 解释器永远存在并且可能会积累垃圾。尝试将其设置为某些东西，例如5000，希望泄漏会停止。

如果您将来获得更多统计结果并且他们反驳max-requests 版本，请分享。祝你好运，詹姆斯·菲特。

【讨论】：