• Stars
    star
    569
  • Rank 78,369 (Top 2 %)
  • Language
    Python
  • License
    BSD 3-Clause "New...
  • Created about 2 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Background task queue for Python backed by Redis, a super minimal Celery

logo WakaQ

Distributed background task queue for Python backed by Redis, a super minimal Celery. Read about the motivation behind this project on this blog post and the accompanying Hacker News discussion. WakaQ is currently used in production at WakaTime.com.

Features

  • Queue priority
  • Delayed tasks (run tasks after a timedelta eta)
  • Scheduled periodic tasks
  • Broadcast a task to all workers
  • Task soft and hard timeout limits
  • Optionally retry tasks on soft timeout
  • Combat memory leaks with max_mem_percent or max_tasks_per_worker
  • Super minimal

Want more features like rate limiting, task deduplication, etc? Too bad, feature PRs are not accepted. Maximal features belong in your app’s worker tasks.

Installing

pip install wakaq

Using

import logging
from datetime import timedelta
from wakaq import WakaQ, Queue, CronTask


# use constants to prevent misspelling queue names
Q_HIGH = 'a-high-priority-queue'
Q_MED = 'a-medium-priority-queue'
Q_LOW = 'a-low-priority-queue'
Q_OTHER = 'another-queue'
Q_DEFAULT = 'default-lowest-priority-queue'


wakaq = WakaQ(

    # List your queues and their priorities.
    # Queues can be defined as Queue instances, tuples, or just a str.
    queues=[
        (0, Q_HIGH),
        (1, Q_MED),
        (2, Q_LOW),
        Queue(Q_OTHER, priority=3, max_retries=5, soft_timeout=300, hard_timeout=360),
        Q_DEFAULT,
    ],

    # Number of worker processes. Must be an int or str which evaluates to an
    # int. The variable "cores" is replaced with the number of processors on
    # the current machine.
    concurrency="cores*4",

    # Raise SoftTimeout in a task if it runs longer than 30 seconds. Can also be set per
    # task or queue. If no soft timeout set, tasks can run forever.
    soft_timeout=30,  # seconds

    # SIGKILL a task if it runs longer than 1 minute. Can be set per task or queue.
    hard_timeout=timedelta(minutes=1),

    # If the task soft timeouts, retry up to 3 times. Max retries comes first
    # from the task decorator if set, next from the Queue's max_retries,
    # lastly from the option below. If No max_retries is found, the task
    # is not retried on a soft timeout.
    max_retries=3,

    # Combat memory leaks by reloading a worker (the one using the most RAM),
    # when the total machine RAM usage is at or greater than 98%.
    max_mem_percent=98,

    # Combat memory leaks by reloading a worker after it's processed 5000 tasks.
    max_tasks_per_worker=5000,

    # Schedule two tasks, the first runs every minute, the second once every ten minutes.
    # Scheduled tasks can be passed as CronTask instances or tuples.
    schedules=[

        # Runs mytask on the queue with priority 1.
        CronTask('* * * * *', 'mytask', queue=Q_MED, args=[2, 2], kwargs={}),

        # Runs mytask once every 5 minutes.
        ('*/5 * * * *', 'mytask', [1, 1], {}),

        # Runs anothertask on the default lowest priority queue.
        ('*/10 * * * *', 'anothertask'),
    ],
)


# timeouts can be customized per task with a timedelta or integer seconds
@wakaq.task(queue=Q_MED, max_retries=7, soft_timeout=420, hard_timeout=480)
def mytask(x, y):
    print(x + y)


@wakaq.task
def anothertask():
    print("hello world")


@wakaq.wrap_tasks_with
def custom_task_decorator(fn):
    def inner(*args, **kwargs):
        # do something before each task runs
        fn(*args, **kwargs)
        # do something after each task runs
    return inner


if __name__ == '__main__':

    # add 1 plus 1 on a worker somewhere
    mytask.delay(1, 1)

    # add 1 plus 1 on a worker somewhere, overwriting the task's queue from medium to high
    mytask.delay(1, 1, queue=Q_HIGH)

    # print hello world on a worker somewhere, running on the default lowest priority queue
    anothertask.delay()

    # print hello world on a worker somewhere, after 10 seconds from now
    anothertask.delay(eta=timedelta(minutes=10))

Deploying

Optimizing

See the WakaQ init params for a full list of options, like Redis host and Redis socket timeout values.

When using in production, make sure to increase the max open ports allowed for your Redis server process.

When using eta tasks a Redis sorted set is used, so eta tasks are automatically deduped based on task name, args, and kwargs. If you want multiple pending eta tasks with the same arguments, just add a throwaway random string to the task’s kwargs for ex: str(uuid.uuid1()).

Running as a Daemon

Here’s an example systemd config to run wakaq-worker as a daemon:

[Unit]
Description=WakaQ Worker Service

[Service]
WorkingDirectory=/opt/yourapp
ExecStart=/opt/yourapp/venv/bin/python /opt/yourapp/venv/bin/wakaq-worker --app=yourapp.wakaq
RemainAfterExit=no
Restart=always
RestartSec=30s
KillSignal=SIGINT
LimitNOFILE=99999

[Install]
WantedBy=multi-user.target

Create a file at /etc/systemd/system/wakaqworker.service with the above contents, then run:

systemctl daemon-reload && systemctl enable wakaqworker

More Repositories

1

vscode-wakatime

Visual Studio Code plugin for automatic time tracking and metrics generated from your programming activity.
TypeScript
1,214
star
2

jetbrains-wakatime

IntelliJ IDEA, PyCharm, RubyMine, PhpStorm, AppCode, AndroidStudio, Goland, Rider, & WebStorm plugin for quantifying your coding.
Java
1,124
star
3

legacy-python-cli

Command line interface used by all WakaTime text editor plugins.
Python
1,021
star
4

vim-wakatime

Vim plugin for automatic time tracking and metrics generated from your programming activity.
Vim Script
1,015
star
5

sublime-wakatime

Sublime Text 2 & 3 plugin for automatic time tracking and metrics generated from your programming activity.
Python
525
star
6

browser-wakatime

Chrome extension for automatic time tracking and metrics generated from your browsing activity.
TypeScript
397
star
7

xcode-wakatime

Xcode plugin for automatic time tracking and metrics generated from your programming activity.
Objective-C
393
star
8

visualstudio-wakatime

Visual Studio plugin for automatic time tracking and metrics generated from your programming activity.
C#
331
star
9

atom-wakatime

Atom plugin for automatic time tracking and metrics generated from your programming activity.
JavaScript
317
star
10

wakatime-cli

Command line interface used by all WakaTime text editor plugins
Go
281
star
11

wakatime-mode

Emacs plugin for automatic time tracking and metrics generated from your programming activity.
Emacs Lisp
268
star
12

eclipse-wakatime

Eclipse plugin for automatic time tracking and metrics generated from your programming activity.
Java
174
star
13

netbeans-wakatime

NetBeans plugin for automatic time tracking and metrics generated from your programming activity.
Java
162
star
14

brackets-wakatime

Brackets plugin for automatic time tracking and metrics generated from your programming activity.
JavaScript
145
star
15

macos-wakatime

Mac system tray app for automatic time tracking and metrics generated from your Xcode, Figma, Postman, etc. usage.
Swift
131
star
16

komodo-wakatime

Komodo plugin for automatic time tracking and metrics generated from your programming activity.
Python
118
star
17

wakadump

Command line tool for converting WakaTime data dump files into various formats.
Python
106
star
18

notepadpp-wakatime

Notepad++ plugin for automatic time tracking and metrics generated from your programming activity.
C#
79
star
19

desktop-old

Plugin install helper app, unfinished.
TypeScript
75
star
20

office-wakatime

Office Solution add-ins for automatic time tracking and metrics generated from your programming activity.
C#
67
star
21

sketch-wakatime

Sketch plugin for automatic time tracking and metrics generated from your Sketch usage.
JavaScript
62
star
22

statuspage

Uptime and latency status page for website and api.
41
star
23

ssms-wakatime

SQL Server Management Studio plugin to quantify your coding with automatic time tracking and metrics about your programming. https://wakatime.com
C#
40
star
24

wakatime-mobile

mobile WakaTime app for displaying dashboard metrics
JavaScript
31
star
25

delphi-wakatime

Embarcadero Delphi plugin for automatic time tracking and metrics generated from your programming activity.
Pascal
29
star
26

wakatime-blog

Markdown post content for the WakaTime blog
23
star
27

micro-wakatime

Micro editor plugin for automatic time tracking and metrics generated from your programming activity.
Lua
22
star
28

wakaq-ts

Background task queue for TypeScript backed by Redis, a super minimal Celery
TypeScript
19
star
29

gedit-wakatime

Gedit 3.8+ plugin for automatic time tracking and metrics generated from your programming activity.
Python
17
star
30

slack-demo-chat

Chat relay server using socket.io for WakaTime + Slack demo
CSS
16
star
31

c9-wakatime

Cloud9 plugin for automatic time tracking and metrics generated from your programming activity.
JavaScript
14
star
32

discord-wakatime

BetterDiscord plugin for automatic time tracking and stats about your Discord usage
JavaScript
12
star
33

adobe-xd-wakatime

Adobe XD plugin for automatic time tracking and metrics generated from your XD usage.
JavaScript
12
star
34

semver-action

Auto-generate the next semantic version.
Go
12
star
35

figma-wakatime

Figma plugin for automatic time tracking and metrics generated from your Figma usage.
TypeScript
12
star
36

desktop-wakatime

Windows & Linux system tray app for automatic time tracking and metrics generated from your Figma, Postman, etc. usage.
TypeScript
12
star
37

coda-wakatime

Coda plugin for automatic time tracking and metrics generated from your programming activity.
Objective-C
11
star
38

texstudio-wakatime

TeXstudio macro for automatic time tracking and metrics generated from your TeXstudio usage.
Python
10
star
39

textmate-wakatime

TextMate plugin for automatic time tracking and metrics generated from your programming activity.
Objective-C++
10
star
40

eric6-wakatime

Eric6 & Pymakr plugin for automatic time tracking and metrics generated from your programming activity.
Python
7
star
41

repl-python-wakatime

Python REPL plugin for automatic time tracking and metrics generated from your programming activity.
Python
7
star
42

vencord-wakatime

Vencord plugin for automatic time tracking and stats about your Discord usage
TypeScript
6
star
43

legal

legal documents for WakaTime
6
star
44

nuget-wakatime-shared-extension-utils

Shared C# utilities used in wakatime extensions
C#
5
star
45

WakaTime.novaextension

Nova plugin for automatic time tracking and metrics generated from your programming activity.
JavaScript
5
star
46

wing-wakatime

A plugin to quantify your programming inside Wing IDE.
Python
4
star
47

monodevelop-wakatime

WakaTime add-in for VSMac/MonoDevelop/Xamarin Studio
C#
3
star
48

homebrew-tap

Homebrew Tap for WakaTime
Ruby
3
star
49

prompt-style.lua

Lua plugin for powerlevel10k style prompt and WakaTime time tracking
Lua
2
star
50

jupyterlab-wakatime

WakaTime for JupyterLab
Python
2
star
51

tcl-prompt

Tcl plugin for powerlevel10k style prompt and WakaTime time tracking
Tcl
2
star
52

Reply-Plugin-Prompt

Perl Reply plugin for powerlevel10k style prompt and WakaTime time tracking
Perl
1
star
53

wakatime.io

Resources and documentation for learning about WakaTime
TypeScript
1
star