• Stars
    star
    668
  • Rank 67,542 (Top 2 %)
  • Language
    Elixir
  • License
    MIT License
  • Created almost 8 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Simple Job Processing in Elixir with Mnesia ⚡

Que

Build Status Coverage Status Version Downloads License

Simple Background Job Processing in Elixir ⚡

Que is a job processing library backed by Mnesia, a distributed real-time database that comes with Erlang / Elixir. That means it doesn't depend on any external services like Redis for persisting job state. This makes it really easy to use since you don't need to install anything other than Que itself.

See the Documentation.


Installation

Add que to your project dependencies in mix.exs:

def deps do
  [{:que, "~> 0.10.1"}]
end

and then add it to your list of applications:

def application do
  [applications: [:que]]
end

Mnesia Setup

Que runs out of the box, but by default all jobs are stored in-memory. To persist jobs across application restarts, specify the DB path in your config.exs:

config :mnesia, dir: 'mnesia/#{Mix.env}/#{node()}'        # Notice the single quotes

And run the following mix task:

$ mix que.setup

This will create the Mnesia schema and job database for you. For a detailed guide, see the Mix Task Documentation. For compiled releases where Mix is not available see this.


Usage

Que is very similar to other job processing libraries such as Ku and Toniq. Start by defining a Worker with a perform/1 callback to process your jobs:

defmodule App.Workers.ImageConverter do
  use Que.Worker

  def perform(image) do
    ImageTool.save_resized_copy!(image, :thumbnail)
    ImageTool.save_resized_copy!(image, :medium)
  end
end

You can now add jobs to be processed by the worker:

Que.add(App.Workers.ImageConverter, some_image)
#=> {:ok, %Que.Job{...}}

Pattern Matching

The argument here can be any term from a Tuple to a Keyword List or a Struct. You can also pattern match and use guard clauses like any other method:

defmodule App.Workers.NotificationSender do
  use Que.Worker

  def perform(type: :like, to: user, count: count) do
    User.notify(user, "You have #{count} new likes on your posts")
  end

  def perform(type: :message, to: user, from: sender) do
    User.notify(user, "You received a new message from #{sender.name}")
  end

  def perform(to: user) do
    User.notify(user, "New activity on your profile")
  end
end

Concurrency

By default, all workers process one Job at a time, but you can customize that by passing the concurrency option:

defmodule App.Workers.SignupMailer do
  use Que.Worker, concurrency: 4

  def perform(email) do
    Mailer.send_email(to: email, message: "Thank you for signing up!")
  end
end

Job Success / Failure Callbacks

The worker can also export optional on_success/1 and on_failure/2 callbacks that handle appropriate cases.

defmodule App.Workers.ReportBuilder do
  use Que.Worker

  def perform({user, report}) do
    report.data
    |> PDFGenerator.generate!
    |> File.write!("reports/#{user.id}/report-#{report.id}.pdf")
  end

  def on_success({user, _}) do
    Mailer.send_email(to: user.email, subject: "Your Report is ready!")
  end

  def on_failure({user, report}, error) do
    Mailer.send_email(to: user.email, subject: "There was a problem generating your report")
    Logger.error("Could not generate report #{report.id}. Reason: #{inspect(error)}")
  end
end

Setup and Teardown

You can similarly export optional on_setup/1 and on_teardown/1 callbacks that are respectively run before and after the job is performed (successfully or not). But instead of the job arguments, they pass the job struct as an argument which holds a lot more internal details that can be useful for custom features such as logging, metrics, requeuing and more.

defmodule MyApp.Workers.VideoProcessor do
  use Que.Worker

  def on_setup(%Que.Job{} = job) do
    VideoMetrics.record(job.id, :start, process: job.pid, status: :starting)
  end

  def perform({user, video, options}) do
    User.notify(user, "Your video is processing, check back later.")
    FFMPEG.process(video.path, options)
  end

  def on_teardown(%Que.Job{} = job) do
    {user, video, _options} = job.arguments
    link = MyApp.Router.video_path(user.id, video.id)

    VideoMetrics.record(job.id, :end, status: job.status)
    User.notify(user, "We've finished processing your video. See the results.", link)
  end
end

Head over to Hexdocs for detailed Worker documentation.


Roadmap

  • Write Documentation
  • Write Tests
  • Persist Job State to Disk
    • Provide an API to interact with Jobs
  • Add Concurrency Support
    • Make jobs work in Parallel
    • Allow customizing the number of concurrent jobs
  • Success/Failure Callbacks
  • Find a more reliable replacement for Amnesia
  • Delayed Jobs
  • Allow job cancellation
  • Job Priority
  • Support running in a multi-node enviroment
    • Recover from node failures
  • Support for more Persistence Adapters
    • Redis
    • Postgres
  • Mix Task for creating Mnesia Database
  • Better Job Failures
    • Option to set timeout on workers
    • Add strategies to automatically retry failed jobs
  • Web UI

Contributing

  • Fork, Enhance, Send PR
  • Lock issues with any bugs or feature requests
  • Implement something from Roadmap
  • Spread the word ❤️

License

This package is available as open source under the terms of the MIT License.


More Repositories

1

memento

Simple + Powerful interface to the Mnesia Distributed Database 💾
Elixir
737
star
2

mongo-sync

Sync Remote and Local MongoDB Databases 🔥
Shell
322
star
3

dotfiles

📦 My Configs and Dotfiles
Shell
121
star
4

ecto_rut

Ecto Model shortcuts to make your life easier! 🎉
Elixir
112
star
5

cloudup.dev

Tools to jump-start development on the Cloud ☁️
JavaScript
106
star
6

better_params

Cleaner request parameters in Elixir web applications 🙌
Elixir
97
star
7

dbfs

Distributed Blockchain-based File Storage 📡
Elixir
63
star
8

capistrano-rake

Execute rake tasks on remote servers (Only Capistrano 3+)
Ruby
41
star
9

github-trending

A gem that fetches trending github repos
Ruby
40
star
10

ztd

Distributed real-time Todo App over RabbitMQ in Elixir 🐰⚡️🌐💧💻
Elixir
24
star
11

mongo-sync-ruby

A Ruby Gem for syncing local and remote Mongo Databases
Ruby
16
star
12

explay

Google Play API in Elixir 💻
Protocol Buffer
16
star
13

cipherjs

Javascript Implementation of simple Ciphers
JavaScript
12
star
14

collab

Details and ElixirConf talk: https://shyr.io/t/real-time-collab
Elixir
12
star
15

sheharyarn.github.io

My Blog
HTML
11
star
16

simple-phoenix-chat

Super simple implementation of Real-time chat in Elixir & Phoenix using WebSockets / Channels
Elixir
11
star
17

freelancer-leaderboard

Displays Freelancer Scavenger 2014 Leaderboard
Ruby
10
star
18

vncviewer

Simple utility to invoke VNC from shell in OSX
Shell
6
star
19

ex_utils

⚡ Collection of Awesome Elixir shortcuts and utilities ⚡
Elixir
6
star
20

werewolf.nvim

🔆🌑 Apply custom Neovim configs based on system theme
Lua
6
star
21

httpbin-node

Node.js implementation of httpbin
JavaScript
5
star
22

httpcodes.co

Quick Informational pages on HTTP Status Codes and HTTP Verbs
HTML
4
star
23

s3_direct_upload_example

Example RoR Application on using the s3_direct_upload gem
Ruby
4
star
24

etv

Ethereum Transaction Verifier
Elixir
2
star
25

dbfs-web

DBFS Web Client
JavaScript
2
star
26

geo-bounds

Axis-aligned geospatial bounded-boxes in Elixir
Elixir
2
star
27

zippy

DEPRECATED! - Zippy is a Ruby Program that downloads files from Zippyshare in your Terminal
Ruby
2
star
28

vertex

2
star
29

cm-test-solution

CareMerge Test Solution
JavaScript
1
star
30

daniyal.me

HTML
1
star
31

nustalumni-android

Android app for the NUST Alumni Association
Java
1
star
32

syklean-html

HTML Layout for the Syklean Octopress Theme
JavaScript
1
star
33

sparkrelay-android

Java
1
star
34

harisandhafsa

HTML
1
star
35

rmd-template

Rails 4 + Mongoid 4 + Devise Template
Ruby
1
star
36

ecto_atomized_map

Use Atom keys for Ecto Schema Maps ⚛️
Elixir
1
star
37

elixir-yaml

YAML Parser and Encoder for Elixir – WIP!
Elixir
1
star
38

ex_gun

Mailgun Integration in Elixir
Elixir
1
star