【问题标题】:Elastic Beanstalk Database Pool IssueElastic Beanstalk 数据库池问题
【发布时间】:2017-07-31 07:08:18
【问题描述】:

我最近决定从 Heroku 切换到通过他们的 Elastic Beanstalk 服务直接部署到 AWS - 希望我的术语是正确的,仍然可以理解 AWS。我的应用程序在 EB 和 Heroku 上运行(这是一个免费的 Heroku 应用程序,因此可能需要 10 秒才能启动)。该应用程序在 Heroku 上完美运行,但在 EB 上仅有时会加载。可能有 4 次中有 1 次得到Internal Server Error。我的数据库池设置为 5,如果我将其扩展到 25,它可以工作,但我是唯一使用该网站的人,据我所知,5 应该足够了。

这是我的 database.yml:

development:
  adapter: sqlite3
  database: db/website.db
  pool: 5
  timeout: 5000
test:
  adapter: sqlite3
  database: db/cucumber-tests.db
  pool: 5
  timeout: 5000
production:
  adapter: postgresql
  encoding: utf8
  database: <%= ENV['RDS_DB_NAME'] %>
  username: <%= ENV['RDS_USERNAME'] %>
  password: <%= ENV['RDS_PASSWORD'] %>
  pool: 5
  host: <%= ENV['RDS_HOSTNAME'] %>
  port: <%= ENV['RDS_PORT'] %>

旁注,在我弄清楚为什么池 5 不够之后,如何使池大小可扩展?据我所知,没有RDS_POOL 变量。有没有办法可以根据用户数量进行扩展?

这是我的 puma 错误日志:

-------------------------------------
/var/log/puma/puma.log
-------------------------------------
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/nulllogger.rb:9:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/deflater.rb:35:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/configuration.rb:232:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/server.rb:578:in `handle_request'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/server.rb:415:in `process_client'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/server.rb:275:in `block in run'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
2017-03-10 03:43:14 - ActiveRecord::ConnectionTimeoutError - could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were i
n use:
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:202:in `block in wait_poll'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `loop'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `wait_poll'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:154:in `internal_poll'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:278:in `internal_poll'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:148:in `block in poll'
        /opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:158:in `synchronize'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:148:in `poll'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:717:in `acquire_connection'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_handling.rb:128:in `retrieve_connection'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_handling.rb:91:in `connection'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/associations/join_dependency.rb:96:in `initialize'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/relation/finder_methods.rb:399:in `new'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/relation/finder_methods.rb:399:in `construct_join_dependency'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/relation/finder_methods.rb:321:in `exists?'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/querying.rb:3:in `exists?'
        app/views/index.erb:66:in `block in singleton class'
app/views/index.erb:-6:in `instance_eval'
        app/views/index.erb:-6:in `singleton class'
        app/views/index.erb:-8:in `__tilt_47116404404940'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/tilt-2.0.6/lib/tilt/template.rb:161:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/tilt-2.0.6/lib/tilt/template.rb:161:in `evaluate'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/tilt-2.0.6/lib/tilt/template.rb:100:in `render'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:823:in `render'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:667:in `erb'
        /var/app/current/app/controllers/application_controller.rb:60:in `block in <class:ApplicationController>'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1611:in `block in compile!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (3 levels) in route!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:994:in `route_eval'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:975:in `block (2 levels) in route!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1015:in `block in process_route'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `catch'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1013:in `process_route'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:973:in `block in route!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `each'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:972:in `route!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1085:in `block in dispatch!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1082:in `dispatch!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `block in call!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `block in invoke'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `catch'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1067:in `invoke'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:907:in `call!'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:895:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/nulllogger.rb:9:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:182:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:2013:in `call'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `block in call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1787:in `synchronize'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/sinatra-1.4.8/lib/sinatra/base.rb:1487:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/deflater.rb:35:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/configuration.rb:232:in `call'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/server.rb:578:in `handle_request'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/server.rb:415:in `process_client'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/server.rb:275:in `block in run'
        /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/puma-3.7.1/lib/puma/thread_pool.rb:120:in `block in spawn_thread'

- 所有代码都是开源的,可以在here找到-

谢谢。非常感谢任何帮助。

更新:我不能再让这个应用程序宕机了。我现在将池设置为 25。我还更改了 DSN,所以 old app can be found herethe new app here

更新 2:我添加了 ruby-on-rails 标签以获得更多关注,因为这个问题是相关的,但该项目使用的是 Sinatra 而不是 Rails。

【问题讨论】:

  • 请阅读“minimal reproducible example”。您的代码需要减少到最低限度以证明问题并且在问题本身而不是在链接中。您的标签/关键字不是主要关键字。如果你想让很多人看到这个问题,我建议用 Ruby 替换一个,用 Rails 替换另一个。此外,不要使用称呼(“hi”)、告别词(“thank you”)或签名。 SO 不是讨论列表,而是在线参考书。最后,“编辑”和“更新”标签是不可取的。将更改后的文本合并到问题中,就好像它最初就在那里一样。我们可以看到什么时候发生了变化。

标签: ruby-on-rails ruby activerecord sinatra puma


【解决方案1】:

看起来您正在使用 Puma — 一个线程化的 Web 服务器。 Puma 在一个进程中启动多个 ruby​​ 线程来同时处理请求。 puma 创建的线程数可能高于池中的数据库连接数(在database.yml 中定义)。

ActiveRecord 自然地保持数据库连接池打开并懒惰地创建新连接。您的database.yml 看起来您将最大池大小设置为 5。这意味着 ActiveRecord 不会在池中创建超过 5 个连接,并且只会在池为空且请求连接时创建新连接。

我在您的存储库中找不到 puma 配置,所以我假设 puma 正在使用默认设置。 The default max threads for Puma is 16。这可能表明您的网络服务器有 16 个线程正在运行(最坏的情况),并且第 5 个线程之后的每个线程都在等待将连接返回到池中以供使用。这是您的 16 个线程之间共享的 5 个连接。作为最佳实践,我始终确保数据库池大小为n+1,其中n 是我的应用程序中将运行的最大线程数。

我猜不出为什么这个问题在 heroku 中也不存在。

【讨论】:

    猜你喜欢
    • 2018-12-31
    • 1970-01-01
    • 2017-12-30
    • 2015-01-19
    • 2019-08-11
    • 2014-10-19
    • 2019-08-06
    • 1970-01-01
    • 2018-02-27
    相关资源
    最近更新 更多