Kickass-Redis - a loose framework of Redis based data solutions
This project aims to create a repository of useful python libraries built on top of redis (and using each other), to automate data modeling with Redis.
##For discussion, help and contributing code - join the google group at https://groups.google.com/forum/#!forum/kickass-redis
Redis is relatively low level, and while it is simple to start using, getting a good knowledge of how to model problems with it in an efficient way can be tricky. So I've created this project to wrap common use cases, into a loose framework of redis based solutions for real world problems.
The project has started out as a bunch of code examples for a presentation I recently gave on IL Tech Talks, that can be found here: http://www.slideshare.net/dvirsky/kicking-ass-with-redis
#Installation
Option 1:
-
Clone this repo or download the sources
-
cd kickass-redis
-
python setup.py build
-
sudo python setup.py install
Option 2:
sudo pip install kickass_redis
#Components
To kick things off, the framework includes the following components:
object_store
a fast yet simple ORM (well, OM actually) that automates creation, indexing and searching for complex objects using redis.
Indexes include: simple string index, numeric index that supports sorting and ranges, simplistic full text index, and a unique key.
###Example:
from kickass_redis.patterns.object_store.objects import IndexedObject, KeySpec
from kickass_redis.patterns.object_store.indexing import UnorderedKey, OrderedNumericalKey
from kickass_redis.patterns.object_store.condition import Condition
class User(IndexedObject):
#which fields should be saved to redis
_spec = ('id', 'name', 'email', 'pwhash', 'registrationDate', 'score')
#The keys for this object
_keySpec = KeySpec(
UnorderedKey(prefix='users',fields=('name',)),
OrderedNumericalKey(prefix='users', field='score')
)
def __init__(self, **kwargs):
IndexedObject.__init__(self, **kwargs)
self.registrationDate = int(kwargs.get('registrationDate', time.time()))
#Creating a user
user = User(email = '[email protected]', name = 'John Doe', pwhash = 'eabc626ec26bc6ae6cb2', score = 100)
user.save()
#loading by name key
users = User.get(Condition({'name': 'John Doe'}))
#loading by id:
users = User.loadObjects((1,))
#See example/users_example for a more detailed exmample and some benchmarks
bitmap_counter
efficient unique value counter (to be used mostly as a unique users counter) with time slots, making use of redis bitmaps.
It makes use of new redis-2.6 commands BITCOUNT and BITOP, so it will not function on redis-2.4.
###Example:
from kickass_redis.patterns.bitmap_counter import BitmapCounter
#Daily unique users counter
counter = BitmapCounter('unique_users', timeResolutions=(BitmapCounter.RES_DAY,))
#sampling current user
counter.add(3)
#Getting the unique user count for today
counter.getCount((time.time(),), counter.RES_DAY)
#Getting cohort analysis on your users for the past week
week = tuple((int(time.time() - i*86400) for i in xrange(7, 0, -1)))
print counter.cohortAnalysis(week, counter.RES_DAY)
#Getting funnel analysis on your users for the past week
print counter.funnelAnalysis(week, counter.RES_DAY)
###New: It now also supports mapping of non sequential or non numeric ids to incemental ids, that makes it memory optimized.
LuaCall
A convenience wrapper that allows you to edit, precache and call Lua scripts available in redis-2.6, as if they were native python functions.
Example:
let mult.lua contain the code:
local val = ARGV[1]*ARGV[2]
redis.call('set', KEYS[1], val)
return redis.call('get', KEYS[1])
Running it from python:
import redis
from kickass_redis.patterns.lua import LuaCall, LuaScriptError
conn = redis.Redis()
#Define the call, and make it runn on our connection
mult = LuaCall(open('mult.lua'), conn)
#Call it once:
print "Result: %s" % mult(keys = ('foor',), args = (3,10))
#call it again
print "Result: %s" % mult(keys = ('foor2',), args = (5,20))
idgenerator
Used in the object store, this can also be used standalone, as a centralized unique, incremental id generator using redis. To optimize performance, it reserves in local memory many ids when accessing redis, which can be tuned.
redis_unit
A unit-test like set of assertions about redis data to be used to validate the data inside a redis database.
requirements
- redis-2.6 server(BITCOUNT/BITOP)
- redis-py
- pyhash package
###Example:
from kickass_redis.patterns.redis_unit import RedisDataTest
class MyTest(RedisDataTest):
def testSomeStuff(self):
self.assertPrefixCount('users:*', minAmount=100000)
self.assertKeysExists('users:1')
self.assertKeysType(self.test.T_HASH, 'users:1')
self.assertHashValue('users:1', 'name', self.equals('John'))
test = MyTest('localhost', 6379)
test.run()
To follow soon:
-
geo search
-
full text search
-
hierarchical counters
-
MySQL data sync
-
Generic expiring object cache.
Feel free to contribute more recipes...
#Project TODO:
-
Add unit tests for all objects
-
Add geo-key