ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool with Sidekiq

TL:DR This error was caused by having more threads running in Sidekiq then connections available in the connection pool.

There’s a fantastic explainer video of this by Rails Autoscale here https://www.youtube.com/watch?v=XOfELjqm188&t=330s&ab_channel=RailsAutoscale

Threads and database connections

Each thread needs at least one DB connection.

The problem

In our sidekiq worker we had the possibility of 10 threads but a maximum of 5 connections available through the database connection pool. This is because in our Rails app we set our max_threads to 5 but in sidekiq we had our “sidekiq_concurrency” (threads) set to 10.

The solution

The solution was to increase the connection pool size in sidekiq worker to 10. So now we have different sized database connection pool size between our web app and our sidekiq worker.

We did this in our Procfile which starts all our dynos in heroku

BEFORE

# Procfile
release: bin/heroku_release
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml

AFTER

# Procfile
release: bin/heroku_release
web: DB_POOL=$RAILS_MAX_THREADS bundle exec puma -C config/puma.rb
worker: DB_POOL=$SIDEKIQ_CONCURRENCY bundle exec sidekiq -C config/sidekiq.yml

We use this value to set our database connection pool within database.yml

# config/database.yml
production:
  adapter: postgresql
  encoding: unicode
  url: <%= ENV['DATABASE_URL'] %>
  pool: <%= ENV["DB_POOL"]