How to have background jobs in Rails
In the world of backend development, sometimes you have some tasks or processes that take a long time to complete, especially when you have to make external API calls that you don’t control the time the request is going to take or just a procedure that takes a long time like importing information or processing images. It is unrealistic to think that the user will wait in the app for the task to end, and the request will time out at 5 minutes anyway if you are using a regular HTTP request, so to solve this problem some gems can handle those tasks in the background so the user can continue doing other thing in your application. In this article, we will see how to implement two of those gems in your Rails application and some considerations when using them.
Let’s start with some definitions. A background job is a process that runs outside the main server request-response cycle, waiting in the background for when the system has time to execute it. They are used to handle time-consuming or resource-intensive processes asynchronously, allowing the application to handle incoming requests more efficiently, speeding up the application for your users.
To achieve this premise there are some gems that can help you with that. The first one is Sidekiq. Sidekiq is a powerful, efficient, and popular background job processing gem for Rails, it handles, all the queueing and the behind-the-scenes of running the background jobs, like retries, dead jobs, queue management, priorities, and workers, among others. It has some strong points that make Sidekiq a good choice:
- Concurrency: Sidekiq is built on the Celluloid framework, enabling it to handle multiple jobs concurrently in a multi-threaded environment. This makes it significantly faster and more efficient compared to other job-processing libraries that rely on single-threaded execution.
- Reliability: Sidekiq uses Redis as its storage backend, ensuring reliable job queuing and persistence, which makes Sidekiq highly reliable for background processing.
- Simplicity: Integrating Sidekiq into a Rails application is straightforward. Developers can define jobs using simple Ruby classes and methods, making it easy to understand and maintain.
- Scalability: Sidekiq is designed to scale horizontally. You can run multiple Sidekiq processes across different servers, allowing your application to handle a large volume of background jobs efficiently.
- Web Interface: Sidekiq Pro comes with a web-based interface that provides insights into the status of jobs, queues, and workers. This makes monitoring and debugging easier.
For all those reasons and more, Sidekiq is an essential tool for any Rails developer looking to implement background jobs into their application. Its performance, ease of use, and scalability make it a top choice for many production applications.
# Sidekiq Job using the sidekiq modules
# To run the job, in the rails console just run:
# - SidekiqJob1.perform_async
class SidekiqJob1
include Sidekiq::Job
sidekiq_options queue: 'custom_queue', retry: 0
def perform(*args)
(1..5).each do |i|
p 'Doing important work from sidekiq'
sleep 5
end
end
end# Sidekiq Job inheriting from Active::Job
# To run the job, in the rails console just run:
# - SidekiqJob2.perform_later
class SidekiqJob2 < ApplicationJob
queue_as :low
def perform(*args)
(1..3).each do |i|
p 'Doing important work inheriting from active job in sidekiq'
sleep 5
end
end
endTo get Sidekiq running, you have to make some configuration files for the Sidekiq server and the queues and priorities. Here are some examples of the configuration files. All the configuration guides can be found in the Sidekiq GitHub
# ./config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = {
url: ENV.fetch('REDIS_URL','redis://localhost:6379')
}
end
Sidekiq.configure_client do |config|
config.redis = {
url: ENV.fetch('REDIS_URL','redis://localhost:6379')
}
end# ./config/sidekiq.yml
:concurrency: 5
staging:
:concurrency: 10
:queues:
- [critical, 2]
- default
- low
- custom_queue# ./config/application.rb
# Add this line in your application configuration file
# Choose the line for the manager you are using
config.active_job.queue_adapter = :sidekiq
config.active_job.queue_adapter = :delayed_jobThere is another gem that can be used alongside Sidekiq that prevents new jobs from running if the same job has not finished a previous execution, this gem is Unique Job, and it is very useful in preventing the server from being out of resources fast. This gem is not perfect though, it has known issues with jobs that error out, when this happens the gem doesn’t know that the job failed and counts it as active, preventing you from kickoff another one, so if your job is not executing this may be the reason for it.
Now that we saw Sidekiq and have a notion of how background jobs work let’s talk about another gem used for that. Delayed Jobs i a gem that allows you to also run background jobs but in a slightly different way than Sidekiq. It is built to work with Active Record and uses the database to store job data, making it straightforward to set up and use, especially for Rails applications that already rely on it. Some key features of delayed jobs are:
- Database-Backed: Unlike some background job processors that use in-memory stores like Redis, Delayed Job stores all job data in the database. This can simplify setup and maintenance, as there is no need for additional infrastructure beyond the existing database.
- Simplicity: Delayed Job is easy to use and integrates well with Rails. Jobs are defined as methods that can be easily enqueued and managed through Active Record models.
- Error Handling and Retry Logic: Delayed Job automatically handles job retries on failure, allowing for configurable retry intervals and maximum attempts. It also logs errors for easier debugging.
- Serialization: Jobs are serialized as YAML, allowing complex objects and data structures to be passed to background jobs without issues.
- Customizable: Delayed Job provides hooks and customization options to manage job priorities, delays, and other behaviors to suit specific application needs.
- Persistence: Using the database for job storage ensures job persistence, even if the job worker crashes or the server restarts.
But also has some drawbacks that might make you reconsider using it:
- Performance: Storing jobs in the database can become a bottleneck for very high-throughput applications. For such cases, a more performant solution like Sidekiq or Resque, which uses in-memory data stores, might be more appropriate.
- Scalability: While sufficient for many applications, Delayed Job’s reliance on the database can limit scalability compared to other solutions that use distributed in-memory data stores.
- It is older than Sidekiq and it works differently.
- Most of the configuration is done inside the job itself.
# DelayedJob job using a class
# To run the job, in the rails console just run:
# - DelayedJob1.new.do_something1
# - DelayedJob1.new.delay.do_something2
class DelayedJob1
def do_something1
(1..5).each do |i|
p 'Doing stuff in Do Something 1'
end
end
handle_asynchronously :do_something1, queue: 'custom_queue_dj1'
def do_something2
(1..5).each do |i|
p 'Doing stuff in Do Something 2'
end
end
end# DelayedJob job using a Struct
# To run the job, in the rails console just run:
# - Delayed::Job.enqueue DelayedJob2.new
DelayedJob2 = Struct.new(:arg1, :arg2) do
def perform
(1..5).each do |i|
p 'Doing stuff with a struct Delayed Job'
end
end
def queue_name
'custom_queue_dj2'
end
def destroy_failed_jobs?
true
end
def reschedule_at(current_time, attempts)
current_time + 30.seconds
end
endI have created a sample repo with an application that uses both gems, Sidekiq and Delayed Job, so you can see how each one is configured and how to create jobs for them and run them for testing. The application is very easy to set up, it uses docker to spin up all the needed servers and all the instructions are in the readme file. Here is the docker-compose file to get the project running.
# Base image to use for multiple services
x-base-app-image:
&base-app
build:
context: .
dockerfile: ./docker/app.Dockerfile
image: base-app:1.0
tmpfs:
- /tmp
tty: true
stdin_open: true
working_dir: /app
volumes:
- .:/app
environment:
- RAILS_ENV=development
- BUNDLE_PATH=/usr/local/bundle
- DB_USER=postgres
- DB_PASSWORD=password
- DB_HOST=db
- REDIS_URL=redis://redis:6379
networks:
- background-jobs-network
services:
app:
<<: *base-app # Use the base app
entrypoint: ./docker/entrypoint.sh
command: rails server -b 0.0.0.0
ports:
- 3500:3000
depends_on:
- db
- redis
sidekiq:
<<: *base-app
entrypoint: ./docker/entrypoint.sh
command: bundle exec sidekiq
depends_on:
- app
- db
dj:
<<: *base-app
entrypoint: ./docker/entrypoint.sh
command: ruby ./script/delayed_job
depends_on:
- app
- db
redis:
image: redis:alpine
ports:
- 6379:6379
command: redis-server --save 20 1 --loglevel warning
volumes:
- redis:/data
networks:
- background-jobs-network
db:
image: postgres:14
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=password
networks:
- background-jobs-network
volumes:
- postgres:/var/lib/postgresql/data
networks:
background-jobs-network:
driver: bridge
volumes:
postgres:
redis:That is all I wanted to share about background jobs in a Rails application. We went through the essentials of background jobs and some examples of how to implement them using gems. Again check out the sample repo for all the code I used in the making of this article. Thank you for reading and if you liked it give it a clap and check my other articles to see what else can be useful to you.
References
