• Stars
    star
    154
  • Rank 242,095 (Top 5 %)
  • Language
    Python
  • Created over 12 years ago
  • Updated over 7 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Loose python framework for kickass redis patterns

Kickass-Redis - a loose framework of Redis based data solutions

This project aims to create a repository of useful python libraries built on top of redis (and using each other), to automate data modeling with Redis.

##For discussion, help and contributing code - join the google group at https://groups.google.com/forum/#!forum/kickass-redis

Redis is relatively low level, and while it is simple to start using, getting a good knowledge of how to model problems with it in an efficient way can be tricky. So I've created this project to wrap common use cases, into a loose framework of redis based solutions for real world problems.

The project has started out as a bunch of code examples for a presentation I recently gave on IL Tech Talks, that can be found here: http://www.slideshare.net/dvirsky/kicking-ass-with-redis

#Installation

Option 1:

  1. Clone this repo or download the sources

  2. cd kickass-redis

  3. python setup.py build

  4. sudo python setup.py install

Option 2:

sudo pip install kickass_redis

#Components

To kick things off, the framework includes the following components:

object_store

a fast yet simple ORM (well, OM actually) that automates creation, indexing and searching for complex objects using redis.

Indexes include: simple string index, numeric index that supports sorting and ranges, simplistic full text index, and a unique key.

###Example:

from kickass_redis.patterns.object_store.objects import IndexedObject, KeySpec
from kickass_redis.patterns.object_store.indexing import UnorderedKey, OrderedNumericalKey
from kickass_redis.patterns.object_store.condition import Condition

class User(IndexedObject):


    #which fields should be saved to redis
    _spec = ('id', 'name', 'email', 'pwhash', 'registrationDate', 'score')

    #The keys for this object
    _keySpec = KeySpec(
        UnorderedKey(prefix='users',fields=('name',)),
        OrderedNumericalKey(prefix='users', field='score')
    )

    def __init__(self, **kwargs):
        IndexedObject.__init__(self, **kwargs)
        self.registrationDate = int(kwargs.get('registrationDate', time.time()))

#Creating a user
user = User(email = '[email protected]', name = 'John Doe', pwhash = 'eabc626ec26bc6ae6cb2', score = 100)
user.save()

#loading by name key
users =  User.get(Condition({'name': 'John Doe'}))

#loading by id:
users = User.loadObjects((1,))

#See example/users_example for a more detailed exmample and some benchmarks

bitmap_counter

efficient unique value counter (to be used mostly as a unique users counter) with time slots, making use of redis bitmaps.

It makes use of new redis-2.6 commands BITCOUNT and BITOP, so it will not function on redis-2.4.

###Example:

from kickass_redis.patterns.bitmap_counter import BitmapCounter

#Daily unique users counter
counter = BitmapCounter('unique_users', timeResolutions=(BitmapCounter.RES_DAY,))

#sampling current user
counter.add(3)

#Getting the unique user count for today
counter.getCount((time.time(),), counter.RES_DAY)

#Getting cohort analysis on your users for the past week
week = tuple((int(time.time() - i*86400) for i in  xrange(7, 0, -1)))
print counter.cohortAnalysis(week, counter.RES_DAY)

#Getting funnel analysis on your users for the past week
print counter.funnelAnalysis(week, counter.RES_DAY)

###New: It now also supports mapping of non sequential or non numeric ids to incemental ids, that makes it memory optimized.

LuaCall

A convenience wrapper that allows you to edit, precache and call Lua scripts available in redis-2.6, as if they were native python functions.

Example:

let mult.lua contain the code:

local val = ARGV[1]*ARGV[2]
redis.call('set', KEYS[1], val)
return redis.call('get', KEYS[1])

Running it from python:

import redis
from kickass_redis.patterns.lua import LuaCall, LuaScriptError
conn = redis.Redis()

#Define the call, and make it runn on our connection
mult = LuaCall(open('mult.lua'), conn)

#Call it once:
print "Result: %s" % mult(keys = ('foor',), args = (3,10))

#call it again
print "Result: %s" % mult(keys = ('foor2',), args = (5,20))

idgenerator

Used in the object store, this can also be used standalone, as a centralized unique, incremental id generator using redis. To optimize performance, it reserves in local memory many ids when accessing redis, which can be tuned.

redis_unit

A unit-test like set of assertions about redis data to be used to validate the data inside a redis database.

requirements

###Example:

from kickass_redis.patterns.redis_unit import RedisDataTest

class MyTest(RedisDataTest):

    def testSomeStuff(self):


        self.assertPrefixCount('users:*', minAmount=100000)
        self.assertKeysExists('users:1')
        self.assertKeysType(self.test.T_HASH, 'users:1')
        self.assertHashValue('users:1', 'name', self.equals('John'))


test = MyTest('localhost', 6379)
test.run()

To follow soon:

  • geo search

  • full text search

  • hierarchical counters

  • MySQL data sync

  • Generic expiring object cache.

Feel free to contribute more recipes...

#Project TODO:

  1. Add unit tests for all objects

  2. Add geo-key

More Repositories

1

overscroll-decor

Android: iOS-like over-scrolling effect applicable over almost all scrollable Android views.
Java
2,833
star
2

easy-content-providers

Easy integration with Android's built-in and custom content providers data
Java
351
star
3

geodis

A redis based geo-resolving library
Python
322
star
4

inbloom

Cross language bloom filter implementation
Java
296
star
5

webp-android

webp support for Android API Level 4 and up, includes the native library and a WebPImageView to render webp in your app
C
266
star
6

OverScrollView

A ScrollView with over scroll capabilities, a complete replacement for android's ScrollView.
Java
264
star
7

plaxien

An Android library to create Explain Views for algorithms
Java
142
star
8

go-disque

Go client for Disque
Go
136
star
9

ncdu-s3

Run ncdu on S3 buckets
Python
110
star
10

redshift_console

Redshift Ops Console
JavaScript
92
star
11

openspace

An open source app to showcase your open source work (yo dawg)
JavaScript
74
star
12

fabric-aws

fabric AWS integration
Python
72
star
13

rainbow

Cloudformation on steroids
Python
55
star
14

vertex

Go API management framework
JavaScript
54
star
15

probe

Android performance instrumentation tool
Python
42
star
16

webp-test

Testing WebP compression for app icons
Ruby
33
star
17

click-config

Config parsing for click cli applications
Python
31
star
18

recat

logcat alternative with on the fly log deobfuscation!
Python
31
star
19

meduza

A fast data store on top of redis
Go
29
star
20

teleport

Execute Python code in country-specific networking context
Python
21
star
21

pyretrace

A python reimplementation on Proguard's Retrace
Python
20
star
22

lobo

Our Dev Flow Tool
Python
20
star
23

TouchyJS

Mobile web application framework
JavaScript
18
star
24

disposable-redis

Create disposable redis servers on the fly for testing
Go
16
star
25

pystream

Stream backups directly to/from S3/HDFS without wasting disk space during the process
Python
14
star
26

gofigure

A simple config file reading library for Go
Go
11
star
27

pyfrank

python binding for iOS automation using frank.
Python
9
star
28

Screaction

A js script enabling you to change css values as the page scrolls
JavaScript
8
star
29

jitter

Jitt Command Line Tool
Python
7
star
30

everythingme.github.io

A showcase of our open source work, built with openspace
JavaScript
7
star
31

android-logger

Java
6
star
32

ffos-notes

Notes App for FirefoxOS
JavaScript
5
star
33

pytest-blocker

A pytest plugin to mark a test as blocker and skip all tests after it upon failure.
Python
4
star
34

sublime.me

SublimeText3 setup for Everything.me FEDS
JavaScript
3
star
35

jitt

A crowd-sourcing oriented localization service for Andorid applications
CSS
3
star
36

python-disposable-redis

Disposable Redis for your Unittests
Python
3
star
37

logstash-scribeinput

A logstash input plugin which receives scribe log entries via thrift
Java
2
star
38

generator-savedata

Photoshop plugin to save document data for BabelFish
JavaScript
2
star
39

cql_dump

Python
1
star
40

go-accuweather

Go wrapper for the Accuweather API.
Go
1
star