ranaroussi/multitasking

MultiTasking: Non-blocking Python methods using decorators

MultiTasking is a tiny Python library lets you convert your Python methods into asynchronous, non-blocking methods simply by using a decorator.

Example

# example.py
import multitasking
import time
import random
import signal

# kill all tasks on ctrl-c
signal.signal(signal.SIGINT, multitasking.killall)

# or, wait for task to finish on ctrl-c:
# signal.signal(signal.SIGINT, multitasking.wait_for_tasks)

@multitasking.task # <== this is all it takes :-)
def hello(count):
    sleep = random.randint(1,10)/2
    print("Hello %s (sleeping for %ss)" % (count, sleep))
    time.sleep(sleep)
    print("Goodbye %s (after for %ss)" % (count, sleep))

if __name__ == "__main__":
    for i in range(0, 10):
        hello(i+1)

The output would look something like this:

$ python example.py

Hello 1 (sleeping for 0.5s)
Hello 2 (sleeping for 1.0s)
Hello 3 (sleeping for 5.0s)
Hello 4 (sleeping for 0.5s)
Hello 5 (sleeping for 2.5s)
Hello 6 (sleeping for 3.0s)
Hello 7 (sleeping for 0.5s)
Hello 8 (sleeping for 4.0s)
Hello 9 (sleeping for 3.0s)
Hello 10 (sleeping for 1.0s)
Goodbye 1 (after for 0.5s)
Goodbye 4 (after for 0.5s)
Goodbye 7 (after for 0.5s)
Goodbye 2 (after for 1.0s)
Goodbye 10 (after for 1.0s)
Goodbye 5 (after for 2.5s)
Goodbye 6 (after for 3.0s)
Goodbye 9 (after for 3.0s)
Goodbye 8 (after for 4.0s)
Goodbye 3 (after for 5.0s)

Settings

The default maximum threads is equal to the # of CPU Cores. This is just a rule of thumb! The Thread module isn't actually using more than one core at a time.

You can change the default maximum number of threads using:

import multitasking
multitasking.set_max_threads(10)

...or, if you want to set the maximum number of threads based on the number of CPU Cores, you can:

import multitasking
multitasking.set_max_threads(multitasking.config["CPU_CORES"] * 5)

For applications that doesn't require access to shared resources, you can set MultiTasking to use multiprocessing.Process() instead of the threading.Thread(), thus avoiding some of the GIL constraints.

import multitasking
multitasking.set_engine("process") # "process" or "thread"

Installation

Install multitasking using pip:

$ pip install multitasking --upgrade --no-cache-dir

Install multitasking using conda:

$ conda install -c ranaroussi multitasking

Legal Stuff

MultiTasking is distributed under the Apache Software License. See the LICENSE.txt file in the release for details.

P.S.

Please drop me an note with any feedback you have.

Ran Aroussi