cscotta/Octobot

Introduction –

Octobot is a task queue worker designed for reliability, ease of use, and 
throughput.

Octobot can listen on any number of queues, with any number of workers 
processing messages from each. Each queue can be set at a custom priority to 
ensure that more system resources are available for more important tasks. AMQP
/ RabbitMQ, Redis, and Beanstalk are supported as backends, with an extensible 
architecture to allow for additional backends to be added as needed.


Architecture –

Octobotʼs internal architecture is a shared-nothing, threadsafe design that 
allows for any number of concurrent queues and tasks to be processed at once,
limited only by system resources. Taking full advantage of parallel execution,
Octobotʼs ability to spawn multiple workers on multiple queues can be used to
more efficiently process tasks at a higher level of parallelism to offset the
cost of IO-bound tasks on remote systems such as database reads, writes, and
queue IO. Octobot also provides for connection pooling to datastores such as
Cassandra and MongoDB, as well as outbound connection management for AMQP.

As an isolated worker with support for multiple queues, Octobot instances can
be geographically distributed across multiple datacenters and availability
regions, assuring that so long as a queue is available, work will get done.

Octobotʼs modular design also makes it easy to add system introspection and
reporting tools, providing metrics such as job throughput per queue, total
messages processed per second, and time per message, which can be passed onto
an external system if additional time calculations are required to determine
end-to-end messaging performance.