Background job processor based on Socketry Async
Jiggler is a Sidekiq-inspired background job processor using Socketry Async and Optimized JSON. It uses fibers to processes jobs, making context switching lightweight. Requires Ruby 3+, Redis 6+.
Jiggler is based on Sidekiq implementation, and re-uses most of its concepts and ideas.
NOTE: Jiggler is a small gem made purely for fun and to gain some hand-on experience with async and fibers. It isn't tested with production projects and might have not-yet-discovered issues. Use at your own risk.
However, it's good to play around and/or to try it in the name of science.
Install the gem:
gem install jiggler
Start Jiggler server as a separate process with bin command:
jiggler -r <FILE_PATH>
-r
specifies a file with loading instructions.
For Rails apps the command'll be jiggler -r ./config/environment.rb
Run jiggler --help
to see the list of command line arguments.
Jiggler 0.1.0rc4 performance results (at most once delivery) against sidekiq 7.0.3
Jiggler 0.1.0 performance results (at least once delivery) against (at most once delivery)
Conceptually Jiggler consists of two parts: the client
and the server
.
The client
is responsible for pushing jobs to Redis
and allows to read stats, while the server
reads jobs from Redis
, processes them, and writes stats.
The server
uses async Redis
connections.
The client
on default is sync
. It's possible to configure the client to be async as well via setting client_async
to true
.
Client settings are:
client_concurrency
client_async
redis_url
(this one is shared with theserver
)
The rest of the settings are server
specific.
NOTE: require "jiggler"
loads only client classes. It doesn't include async
lib, this dependency is being required only within the server
part.
require "jiggler"
Jiggler.configure do |config|
config[:client_concurrency] = 12 # Should equal to the number of threads/fibers in the client app. Defaults to 10
config[:concurrency] = 12 # The number of running workers on the server. Defaults to 10
config[:timeout] = 12 # Seconds Jiggler wait for jobs to finish before shutdown. Defaults to 25
config[:environment] = "myenv" # On default fetches the value ENV["APP_ENV"] and fallbacks to "development"
config[:require] = "./jobs.rb" # Path to file with jobs/app initializer
config[:redis_url] = ENV["REDIS_URL"] # On default fetches the value from ENV["REDIS_URL"]
config[:queues] = ["shippers"] # An array of queue names the server is going to listen to. On default uses ["default"]
config[:config_file] = "./jiggler.yml" # .yml file with Jiggler settings
config[:mode] = :at_most_once # at_most_once and at_least_once modes supported. Defaults to :at_least_once
end
at_least_once
mode grants reliability for the regular enqueued jobs which are going to be executed by workers. The scheduled jobs (the ones planned to be executed at a specific time, or which failed and going to be retried) still support only at_most_once
strategy. The support for them is going to be added in the upcoming versions.
On default all queues have the same priority (equals to 0). Higher number means higher prio.
It's possible to specify custom priorities as follows:
Jiggler.configure do |config|
config[:queues] = [["shippers", 0], ["shipments", 1], ["delivery", 2]]
end
IO_EVENT_SELECTOR
is an env variable which allows to specify the event selector used by the Ruby scheduler.
On default it uses Epoll
(IO_EVENT_SELECTOR=EPoll
).
Another available option is URing
(IO_EVENT_SELECTOR=URing
). Underneath it uses io_uring
library. It is a Linux kernel library that provides a high-performance interface for asynchronous I/O operations. It was introduced in Linux kernel version 5.1 and aims to address some of the limitations and scalability issues of the existing AIO (Asynchronous I/O) interface.
In the future it might bring a lot of performance boost into Ruby fibers world (once async
project fully adopts it), but at the moment in the most cases its performance is similar to EPoll
, yet it could give some boost with File IO.
The gem allows to use libs/calls from socketry
stack (https://github.com/socketry) within workers.
Sample:
def perform(ids)
resources = Resource.where(id: ids)
Async do
resources.each do |resource|
Async do
result = api_client.get(resource)
resource.update(data: result) if result
rescue => err
logger.error(err)
end
end
end
end
Internally Jiggler server
among others includes the next entities: Manager
, Poller
, Monitor
.
Manager
is responsible for workers.
Poller
fetches data for retries and scheduled jobs.
Monitor
periodically loads stats data into redis.
Manager
and Monitor
are mandatory, while Poller
can be disabled in case there's no need for retries/scheduled jobs.
Jiggler.configure do |config|
config[:stats_interval] = 12 # Defaults to 10
config[:poller_enabled] = true # Defaults to true
config[:poll_interval] = 12 # Defaults to 5
end
Jiggler::Web.new
is a rack application. It can be run on its own or be mounted in app routes, f.e. with Rails:
require "jiggler/web"
Rails.application.routes.draw do
mount Jiggler::Web.new => "/jiggler"
# ...
end
To get the available stats run:
irb(main)> Jiggler.summary
=>
{"retry_jobs_count"=>0,
"dead_jobs_count"=>0,
"scheduled_jobs_count"=>0,
"failures_count"=>6,
"processed_count"=>0,
"processes"=>
{"jiggler:svr:3513d56f7ed2:10:25:default:1:1673875240:83568:JulijaA-MBP.local"=>
{"heartbeat"=>1673875270.551845,
"rss"=>32928,
"current_jobs"=>{},
"name"=>"jiggler:svr:3513d56f7ed2",
"concurrency"=>"10",
"timeout"=>"25",
"queues"=>"default",
"poller_enabled"=>true,
"started_at"=>"1673875240",
"pid"=>"83568",
"hostname"=>"JulijaA-MBP.local"}},
"queues"=>{"mine"=>1, "unknown"=>1, "test"=>1}}
Note: Jiggler summary shows only queues which have enqueued jobs.
Job classes should include Jiggler::Job
and implement perform
method.
class MyJob
include Jiggler::Job
def perform
puts "Performing..."
end
end
The job can be enqued with:
MyJob.enqueue
Specify custom job options:
class AnotherJob
include Jiggler::Job
job_options queue: "custom", retries: 10, retry_queue: "custom_retries"
def perform(num1, num2)
puts num1 + num2
end
end
To override the options for a specific job:
AnotherJob.with_options(queue: "default").enqueue(num1, num2)
It's possible to enqueue multiple jobs at once with:
arr = [[num1, num2], [num3, num4], [num5, num6]]
AnotherJob.enqueue_bulk(arr)
For the cases when you want to enqueue jobs with a delay or at a specific time run:
seconds = 100
AnotherJob.enqueue_in(seconds, num1, num2)
To cleanup the data from Redis you can run one of these:
# prune data for a specific queue
Jiggler.config.cleaner.prune_queue(queue_name)
# prune all queues data
Jiggler.config.cleaner.prune_all_queues
# prune all Jiggler data from Redis including all enqued jobs, stats, etc.
Jiggler.config.cleaner.prune_all
On default client
uses synchronous Redis
connections.
In case the client is being used in async app (f.e. with Falcon web server, etc.), then it's possible to set a custom redis pool capable of sending async requests into redis.
The pool should be compatible with Async::Pool
- support acquire
method.
Jiggler.configure_client do |config|
config[:client_redis_pool] = my_async_redis_pool
end
# or use built-in async pool with
require "async/pool"
Jiggler.configure_client do |config|
config[:client_async] = true
end
Then, the client methods could be called with something like:
Sync { Jiggler.config.cleaner.prune_all }
Async { MyJob.enqueue }
Docker! You can spin up a local development environment without the need to install dependencies directly on your local machine.
To get started, make sure you have Docker installed on your system. Then, simply run the following command to build the Docker image and start a development server:
docker-compose up --build
Debug:
docker-compose up -d && docker attach jiggler_app
Start irb:
docker-compose exec app bundle exec irb
Run tests:
docker-compose run --rm web -- bundle exec rspec
To run the load tests modify the docker-compose.yml
to point to bin/jigglerload
Fork & Pull Request.