Elastic Beanstalk Redis 失败，Webapp 无响应答案

【问题标题】：Elastic Beanstalk Redis Fail, Webapp UnresponsiveElastic Beanstalk Redis 失败，Webapp 无响应
【发布时间】：2017-10-04 08:34:06
【问题描述】：

无法通过 sidekiq 错误。

尝试从 Heroku 迁移到 AWS EB。我有一个在 linux 机器上运行 rails 4.2.0、ruby 2.3 的 rails 应用程序，但一直遇到问题。 webapp 不会加载 - 它只是一遍又一遍地超时。

INFO: Running in ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
INFO: See LICENSE and the LGPL-3.0 for licensing details.
INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
INFO: Booting Sidekiq 3.5.4 with redis options {:url=>nil}
ERROR: heartbeat: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Redis 一直告诉我它的url is nil，尽管它似乎是一个可靠的设置。（它适用于我设法以相同配置运行的另一个应用程序。我还发现 ERROR MISCONF 通知也很麻烦，但 Redis URL 甚至没有设置。此外，两者都在同一个安全组中

这是我的config/sidekiq.rb：

rails_root = Rails.root || File.dirname(__FILE__) + '/../..'
rails_env = Rails.env || 'development'
redis_config = YAML.load_file(rails_root.to_s + '/config/redis.yml')
redis_config.merge! redis_config.fetch(Rails.env, {})
redis_config.symbolize_keys!

Sidekiq.configure_server do |config|
  config.redis = { url: "redis://#{ENV['REDIS_HOST']}:#{redis_config[:port]}/12" }
end

Sidekiq.configure_client do |config|
  config.redis = { url: "redis://#{ENV['REDIS_HOST']}:#{redis_config[:port]}/12" }
end

还有我的 config/redis.yml：

development:
  host: localhost
  port: 6379
test:
  host: localhost
  port: 6379
production:
  host: ENV['REDIS_HOST']
  port: 6379

我的application.yml：

REDIS_HOST: project-name-001.random-token.0001.use1.cache.amazonaws.com

Here'ssetup_swap.config、sidekiq.config 和 nginx.config。

我也看到过这个问题，但我认为它无关紧要。也许我错了？如果不相关，将在另一篇文章中解决。

Starting nginx: nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored

可能是什么？有什么重要的我错过了吗？

编辑：添加 nginx.conf

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    index   index.html index.htm;

    server {
        listen       80 ;
        listen       [::]:80 ;
        server_name  localhost;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        # redirect server error pages to the static page /40x.html
        #
        error_page 404 /404.html;
            location = /40x.html {
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 ;
#        listen       [::]:443 ssl http2 ;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
#        ssl_ciphers <redacted>;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

更新回复：我将nginx.conf 更新为：include /etc/nginx/conf.d/webapp_healthd.conf;，但仍然得到以下信息：

[root] service nginx restart
nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
Stopping nginx:                                            [  OK  ]
Starting nginx: nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
                                                       [  OK  ]

而且，以下情况仍然存在：

ERROR: heartbeat: MISCONF Redis is configured to save RDB snapshots, 
but is currently not able to persist on disk. Commands that may modify 
the data set are disabled. Please check Redis logs for details about 
the error.

更新 2 删除了对 localhost 端口 80 的重复引用，并且 nginx 停止抱怨，但我仍然收到 Heartbeat MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. 错误。

来自Sidekiq.redis(&:info)的输出：

{ 
  "redis_version"=>"3.2.8", 
  "redis_git_sha1"=>"00000000", 
  "redis_git_dirty"=>"0", 
  "redis_build_id"=>"12e5c8be08dc4d3", 
  "redis_mode"=>"standalone", 
  "os"=>"Linux 4.4.51-40.60.amzn1.x86_64 x86_64", 
  "arch_bits"=>"64", 
  "multiplexing_api"=>"epoll", 
  "gcc_version"=>"4.8.3", 
  "process_id"=>"24835", 
  "run_id"=>"83a8de8b50f482a4e271228435b2f0c8e3fa5b5c", 
  "tcp_port"=>"6379", 
  "uptime_in_seconds"=>"341217", 
  "uptime_in_days"=>"3", 
  "hz"=>"10", 
  "lru_clock"=>"1108155", 
  "executable"=>"/usr/local/bin/redis-server", 
  "config_file"=>"/etc/redis/redis.conf", 
  "connected_clients"=>"2", 
  "client_longest_output_list"=>"0", 
  "client_biggest_input_buf"=>"0", 
  "blocked_clients"=>"0", 
  "used_memory"=>"842664", 
  "used_memory_human"=>"822.91K", 
  "used_memory_rss"=>"3801088", 
  "used_memory_rss_human"=>"3.62M", 
  "used_memory_peak"=>"924360", 
  "used_memory_peak_human"=>"902.70K", 
  "total_system_memory"=>"1043574784", 
  "total_system_memory_human"=>"995.23M", 
  "used_memory_lua"=>"37888", 
  "used_memory_lua_human"=>"37.00K", 
  "maxmemory"=>"0", 
  "maxmemory_human"=>"0B", 
  "maxmemory_policy"=>"noeviction", 
  "mem_fragmentation_ratio"=>"4.51", 
  "mem_allocator"=>"jemalloc-4.0.3", 
  "loading"=>"0", 
  "rdb_changes_since_last_save"=>"177", 
  "rdb_bgsave_in_progress"=>"0", 
  "rdb_last_save_time"=>"1493941570", 
  "rdb_last_bgsave_status"=>"err", 
  "rdb_last_bgsave_time_sec"=>"0", 
  "rdb_current_bgsave_time_sec"=>"-1", 
  "aof_enabled"=>"0", 
  "aof_rewrite_in_progress"=>"0", 
  "aof_rewrite_scheduled"=>"0", 
  "aof_last_rewrite_time_sec"=>"-1", 
  "aof_current_rewrite_time_sec"=>"-1", 
  "aof_last_bgrewrite_status"=>"ok", 
  "aof_last_write_status"=>"ok", 
  "total_connections_received"=>"17", 
  "total_commands_processed"=>"141824", 
  "instantaneous_ops_per_sec"=>"0", 
  "total_net_input_bytes"=>"39981126", 
  "total_net_output_bytes"=>"72119284", 
  "instantaneous_input_kbps"=>"0.00", 
  "instantaneous_output_kbps"=>"0.00", 
  "rejected_connections"=>"0", 
  "sync_full"=>"0", 
  "sync_partial_ok"=>"0", 
  "sync_partial_err"=>"0", 
  "expired_keys"=>"3", 
  "evicted_keys"=>"0", 
  "keyspace_hits"=>"14", 
  "keyspace_misses"=>"533", 
  "pubsub_channels"=>"0", 
  "pubsub_patterns"=>"0", 
  "latest_fork_usec"=>"160", 
  "migrate_cached_sockets"=>"0", 
  "role"=>"master", 
  "connected_slaves"=>"0", 
  "master_repl_offset"=>"0", 
  "repl_backlog_active"=>"0", 
  "repl_backlog_size"=>"1048576", 
  "repl_backlog_first_byte_offset"=>"0", 
  "repl_backlog_histlen"=>"0", 
  "used_cpu_sys"=>"167.52", 
  "used_cpu_user"=>"46.03", 
  "used_cpu_sys_children"=>"0.00", 
  "used_cpu_user_children"=>"0.00", 
  "cluster_enabled"=>"0", 
  "db0"=>"keys=1,expires=0,avg_ttl=0"
}

有趣的是，我找不到我的 redis 日志来进一步调查。在我的redis.conf 中，我看到的只有这个。

# Specify the log file name. Also the empty string can be used to force
# Redis to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile ""

我什至运行了find / -path /sys -prune -o -path /proc -prune -o -name *redis* 并没有看到任何日志文件。 (╯°□°)╯︵┻━┻

奇怪的是 production.log 根本没有被写入，检查权限： rw-r--r-- 1 webapp webapp 0 May 8 20:01 production.log

【问题讨论】：

标签： ruby-on-rails redis amazon-elastic-beanstalk sidekiq

【解决方案1】：

请分享您的/etc/nginx/nginx.conf，我猜nginx.conf 在conf.d 文件夹中包含其他服务器conf 文件，检查nginx.conf 中的include /etc/nginx/conf.d/*.conf; 行，如果是这样，它可能会加载文件两次或其他默认值具有相同server name 的文件，您可以将其更改为include /etc/nginx/conf.d/webapp_healthd.conf 或您想要的任何名称，但在检查机器上的文件之前。

如果有任何临时文件，例如~default 或.save，请检查/etc/nginx/sites-enabled/ 目录。用ls -lah检查它，删除它们，重新启动nginx并检查错误，或者通过ebextensions重新部署。

更新

尝试从nginx.confall 中删除server { ... } 的所有部分，确保在http 中包含您的文件/etc/nginx/conf.d/webapp_healthd.conf，那里已经有server listen 80; 和localhost..

nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/webapp_healthd.conf;

    index   index.html index.htm;
}

003_nginx.config

files:
    "/etc/nginx/conf.d/webapp_healthd.conf" :
        mode: "000755"
        owner: root
        group: root
        content: |
            upstream my_app {
              server unix:///var/run/puma/my_app.sock;
            }

            log_format healthd '$msec"$uri"'
                            '$status"$request_time"$upstream_response_time"'
                            '$http_x_forwarded_for';

            server {
              listen 80;
              server_name _ localhost; # need to listen to localhost for worker tier
              root /var/app/current/public;

              if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
                set $year $1;
                set $month $2;
                set $day $3;
                set $hour $4;
              }

              access_log  /var/log/nginx/access.log  main;
              access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;

              try_files $uri/index.html $uri @my_app;

              location @my_app {
                proxy_pass http://my_app; # match the name of upstream directive which is defined above
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              }

              location /assets {
                alias /var/app/current/public/assets;
                gzip_static on;
                gzip on;
                expires max;
                add_header Cache-Control public;
              }
                      # redirect server error pages to the static page /40x.html
              #
              error_page 404 /404.html;
                  location = /40x.html {
              }

              # redirect server error pages to the static page /50x.html
              #
              error_page 500 502 503 504 /50x.html;
                 location = /50x.html {
              }
            }
    "/opt/elasticbeanstalk/hooks/appdeploy/post/03_restart_nginx.sh":
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/usr/bin/env bash
            rm /etc/nginx/conf.d/webapp_healthd.conf.bak
            rm /etc/nginx/conf.d/custom.conf
            service nginx restart

【讨论】：

感谢您的回复！我添加了nginx.conf 文件，我确实注意到include /etc/nginx/conf.d/*.conf; 但是当我去检查/etc/nginx/sites-enabled/ 时，我什么也没找到。没有这样的目录……这是什么意思？
另外，更改为包括webapp_healthd.conf（它在机器上）。查看更新的描述@Berlin
我认为这是因为在 nginx conf 和 webapp_healthd.conf 你有监听端口 80 的服务器
@Berlin- 非常感谢 - 有一个骗局！仍然收到MISCONF Redis is currently not able to persist on disk. 错误。试图调查，但 /var/logs 中根本不存在 redis logs！？有任何想法吗？ redis.conf 就像阅读拉丁文一样。奇怪的是 production.log 根本没有被写入，检查权限：rw-r--r-- 1 webapp webapp 0 May 8 20:01 production.log
你是否在这台机器上为这个环境安装了本地 redis？如果是这样，请告诉我，我们更改 redis conf 文件，redis 版本是什么？ redis.conf 上的唯一一行是 logfilg 吗？更改 nginx.conf 后现在会发生什么，您可以通过浏览器访问您的应用程序吗？