• Stars
    star
    155
  • Rank 235,309 (Top 5 %)
  • Language
    Ruby
  • License
    MIT License
  • Created over 13 years ago
  • Updated about 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Redis-backed statistics storage and querying library written in Ruby.

Redistat Build Status

A Redis-backed statistics storage and querying library written in Ruby.

Redistat was originally created to replace a small hacked together statistics collection solution which was MySQL-based. When I started I had a short list of requirements:

  • Store and increment/decrement integer values (counters, etc)
  • Up to the second statistics available at all times
  • Screamingly fast

Redis fits perfectly with all of these requirements. It has atomic operations like increment, and it's lightning fast, meaning if the data is structured well, the initial stats reporting call will store data in a format that's instantly retrievable just as fast.

Installation

gem install redistat

If you are using Ruby 1.8.x, it's recommended you also install the SystemTimer gem, as the Redis gem will otherwise complain.

Usage (Crash Course)

view_stats.rb:

require 'redistat'

class ViewStats
  include Redistat::Model
end

# if using Redistat in multiple threads set this
# somewhere in the beginning of the execution stack
Redistat.thread_safe = true

Simple Example

Store:

ViewStats.store('hello', {:world => 4})
ViewStats.store('hello', {:world => 2}, 2.hours.ago)

Fetch:

ViewStats.find('hello', 1.hour.ago, 1.hour.from_now).all
  #=> [{'world' => 4}]
ViewStats.find('hello', 1.hour.ago, 1.hour.from_now).total
  #=> {'world' => 4}
ViewStats.find('hello', 3.hour.ago, 1.hour.from_now).total
  #=> {'world' => 6}

Advanced Example

Store page view on product #44 from Chrome 11:

ViewStats.store('views/product/44', {'count/chrome/11' => 1})

Fetch product #44 stats:

ViewStats.find('views/product/44', 23.hours.ago, 1.hour.from_now).total
  #=> { 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }

Store a page view on product #32 from Firefox 3:

ViewStats.store('views/product/32', {'count/firefox/3' => 1})

Fetch product #32 stats:

ViewStats.find('views/product/32', 23.hours.ago, 1.hour.from_now).total
  #=> { 'count' => 1, 'count/firefox' => 1, 'count/firefox/3' => 1 }

Fetch stats for all products:

ViewStats.find('views/product', 23.hours.ago, 1.hour.from_now).total
  #=> { 'count'           => 2,
  #     'count/chrome'    => 1,
  #     'count/chrome/11' => 1,
  #     'count/firefox'   => 1,
  #     'count/firefox/3' => 1 }

Store a 404 error view:

ViewStats.store('views/error/404', {'count/chrome/9' => 1})

Fetch stats for all views across the board:

ViewStats.find('views', 23.hours.ago, 1.hour.from_now).total
  #=> { 'count'           => 3,
  #     'count/chrome'    => 2,
  #     'count/chrome/9'  => 1,
  #     'count/chrome/11' => 1,
  #     'count/firefox'   => 1,
  #     'count/firefox/3' => 1 }

Fetch list of products known to Redistat:

finder = ViewStats.find('views/product', 23.hours.ago, 1.hour.from_now)
finder.children.map { |child| child.label.me }
  #=> [ "32", "44" ]
finder.children.map { |child| child.label.to_s }
  #=> [ "views/products/32", "views/products/44" ]
finder.children.map { |child| child.total }
  #=> [ { "count" => 1, "count/firefox" => 1, "count/firefox/3" => 1 },
  #     { "count" => 1, "count/chrome"  => 1, "count/chrome/11" => 1 } ]

Terminology

Scope

A type of global-namespace for storing data. When using the Redistat::Model wrapper, the scope is automatically set to the class name. In the examples above, the scope is ViewStats. Can be overridden by calling the #scope class method on your model class.

Label

Identifier string to separate different types and groups of statistics from each other. The first argument of the #store, #find, and #fetch methods is the label that you're storing to, or fetching from.

Labels support multiple grouping levels by splitting the label string with / and storing the same stats for each level. For example, when storing data to a label called views/product/44, the data is stored for the label you specify, and also for views/product and views. You may also configure a different group separator using the Redistat.group_separator= method. For example:

Redistat.group_separator = '|'

A word of caution: Don't use a crazy number of group levels. As two levels causes twice as many hincrby calls to Redis as not using the grouping feature. Hence using 10 grouping levels, causes 10 times as many write calls to Redis.

Input Statistics Data

You provide Redistat with the data you want to store using a Ruby Hash. This data is then stored in a corresponding Redis hash with identical key/field names.

Key names in the hash also support grouping features similar to those available for Labels. Again, the more levels you use, the more write calls to Redis, so avoid using 10-15 levels.

Depth (Storage Accuracy)

Define how accurately data should be stored, and how accurately it's looked up when fetching it again. By default Redistat uses a depth value of :hour, which means it's impossible to separate two events which were stored at 10:18 and 10:23. In Redis they are both stored within a date key of 2011031610.

You can set depth within your model using the #depth class method. Available depths are: :year, :month, :day, :hour, :min, :sec

Time Ranges

When you fetch data, you need to specify a start and an end time. The selection behavior can seem a bit weird at first when, but makes sense when you understand how Redistat works internally.

For example, if we are using a Depth value of :hour, and we trigger a fetch call starting at 1.hour.ago (13:34), till Time.now (14:34), only stats from 13:00:00 till 13:59:59 are returned, as they were all stored within the key for the 13th hour. If both 13:00 and 14:00 was returned, you would get results from two whole hours. Hence if you want up to the second data, use an end time of 1.hour.from_now.

The Finder Object

Calling the #find method on a Redistat model class returns a Redistat::Finder object. The finder is a lazy-loaded gateway to your data. Meaning you can create a new finder, and modify instantiated finder's label, scope, dates, and more. It does not call Redis and fetch the data until you call #total, #all, #map, #each, or #each_with_index on the finder.

This section does need further expanding as there's a lot to cover when it comes to the finder.

Key Expiry

Support for expiring keys from Redis is available, allowing you too keep varying levels of details for X period of time. This allows you easily keep things nice and tidy by only storing varying levels detailed stats only for as long as you need.

In the below example we define how long Redis keys for varying depths are stored. Second by second stats are available for 10 minutes, minute by minute stats for 6 hours, hourly stats for 3 months, daily stats for 2 years, and yearly stats are retained forever.

class ViewStats
  include Redistat::Model

  depth :sec

  expire \
    :sec => 10.minutes.to_i,
    :min => 6.hours.to_i,
    :hour => 3.months.to_i,
    :day => 2.years.to_i
end

Keep in mind that when storing stats for a custom date in the past for example, the expiry time for the keys will be relative to now. The values you specify are simply passed to the Redis#expire method.

Internals

Storing / Writing

Redistat stores all data into a Redis hash keys. The Redis key name the used consists of three parts. The scope, label, and datetime:

{scope}/{label}:{datetime}

For example, this...

ViewStats.store('views/product/44', {'count/chrome/11' => 1})

...would store the follow hash of data...

{ 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }

...to all 12 of these Redis hash keys...

ViewStats/views:2011
ViewStats/views:201103
ViewStats/views:20110315
ViewStats/views:2011031510
ViewStats/views/product:2011
ViewStats/views/product:201103
ViewStats/views/product:20110315
ViewStats/views/product:2011031510
ViewStats/views/product/44:2011
ViewStats/views/product/44:201103
ViewStats/views/product/44:20110315
ViewStats/views/product/44:2011031510

...by creating the Redis key, and/or hash field if needed, otherwise it simply increments the already existing data.

It would also create the following Redis sets to keep track of which child labels are available:

ViewStats.label_index:
ViewStats.label_index:views
ViewStats.label_index:views/product

It should now be more obvious to you why you should think about how you use the grouping capabilities so you don't go crazy and use 10-15 levels. Storing is done through Redis' hincrby call, which only supports a single key/field combo. Meaning the above example would call hincrby a total of 36 times to store the data, and sadd a total of 3 times to ensure the label index is accurate. 39 calls is however not a problem for Redis, most calls happen in less than 0.15ms (0.00015 seconds) on my local machine.

Fetching / Reading

By default when fetching statistics, Redistat will figure out how to do the least number of reads from Redis. First it checks how long range you're fetching. If whole days, months or years for example fit within the start and end dates specified, it will fetch the one key for the day/month/year in question. It further drills down to the smaller units.

It is also intelligent enough to not fetch each day from 3-31 of a month, instead it would fetch the data for the whole month and the first two days, which are then removed from the summary of the whole month. This means three calls to hgetall instead of 29 if each whole day was fetched.

Buffer

The buffer is a new, still semi-beta, feature aimed to reduce the number of Redis hincrby that Redistat sends. This should only really be useful when you're hitting north of 30,000 Redis requests per second, if your Redis server has limited resources, or against my recommendation you've opted to use 10, 20, or more label grouping levels.

Buffering tries to fold together multiple store calls into as few as possible by merging the statistics hashes from all calls and groups them based on scope, label, date depth, and more. You configure the the buffer by setting Redistat.buffer_size to an integer higher than 1. This basically tells Redistat how many store calls to buffer in memory before writing all data to Redis.

Todo

  • More details in Readme.
  • Documentation.
  • Anything else that becomes apparent after real-world use.

Credits

Global Personals deserves a thank you. Currently the primary user of Redistat, they've allowed me to spend some company time to further develop the project.

Note on Patches/Pull Requests

  • Fork the project.
  • Make your feature addition or bug fix.
  • Add tests for it. This is important so I don't break it in a future version unintentionally.
  • Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
  • Send me a pull request. Bonus points for topic branches.

License and Copyright

Copyright (c) 2011 Jim Myhrberg.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

More Repositories

1

git-aware-prompt

Display current Git branch name in your terminal prompt when in a Git working directory.
Shell
2,139
star
2

tmux-themepack

A pack of various Tmux themes.
Go
1,564
star
3

tmuxifier

Tmuxify your Tmux. Powerful session, window & pane management for Tmux.
Shell
1,018
star
4

build-emacs-for-macos

Somewhat hacky script to automate building of Emac.app on macOS.
Go
434
star
5

emacs-builds

Self-contained Emacs.app builds for macOS, with native-compilation support.
Go
303
star
6

manservant

Browse man pages in style with your personal manservant.
Perl
164
star
7

docker-znc

Run the ZNC IRC Bouncer in a Docker container.
Shell
134
star
8

zsh-peco-history

Search shell history with peco when pressing ctrl+r.
Shell
121
star
9

PastryKit

A little-known and unreleased iPhone web-app Javascript framework developed by Apple.
JavaScript
104
star
10

.emacs.d

My personal Emacs config with any quirks, oddities, bugs, and man-eating errors I live with on a daily basis.
Emacs Lisp
98
star
11

twilight-bright-theme.el

A Emacs port of the TextMate theme by the same name with some minor tweaks and additions.
Emacs Lisp
63
star
12

stub.sh

Helpers for bash script testing to stub/fake binaries and functions. Includes support for validating number of stub calls, and/or if stub has been called with specific arguments.
Shell
53
star
13

php-rack

An implementation of the middleware execution stack from Ruby's Rack library, for PHP.
PHP
52
star
14

dotfiles

My personals dotfiles with any quirks, oddities, bugs, and man-eating errors I live with on a daily basis.
Shell
40
star
15

twilight-anti-bright-theme

A light-on-dark Emacs and TextMate theme inspired by the dark-on-light Twilight Bright TextMate theme.
Emacs Lisp
40
star
16

common-flow

An attempt to gather a sensible selection of the most common usage patterns of git into a single and concise specification.
40
star
17

birds-of-paradise-plus-theme.el

A Emacs port of Joseph Bergantine's light-on-dark theme by the same name.
Emacs Lisp
38
star
18

node-base58

Base58 encoding and decoding for Node.js
JavaScript
33
star
19

rubocopfmt.el

Emacs minor-mode to format Ruby code with RuboCop on save.
Emacs Lisp
16
star
20

go-midjourney

Go
15
star
21

760-grid-system

Use the same principals of the 960 grid system within Facebook's 760 pixel wide frame for Facebook Applications.
Ruby
15
star
22

jimeh.me-v3.0

Jekyll project of a old version my personal website and blog.
CSS
13
star
23

tomorrow-night-paradise-theme.el

A light-on-dark Emacs theme which is essentially a tweaked version of Chris Kempson's Tomorrow Night Bright theme.
Emacs Lisp
11
star
24

litemysql

Very light-weight and simple ORM-like MySQL library for PHP. Kind of like ActiveRecord's little brother.
PHP
11
star
25

amqp-failover

Add multi-server support with failover and fallback to the amqp gem.
Ruby
10
star
26

homebrew-emacs-builds

Ruby
9
star
27

jimeh.github.io

This is the source-code for my personal website.
SCSS
9
star
28

suggest_results

Easily customizable search suggestion plugin for jQuery, which suggests results directly, rather than search terms.
JavaScript
7
star
29

collecta_ruby

A light Ruby library / Rails plugin for querying the Collecta API.
Ruby
7
star
30

ansible-adguardhome

Ansible role to install and run AdGuard Home, with support for non-root operation.
Python
7
star
31

mta-sts-on-github-pages

Template repository for hosting MTA-STS (.well-known/mta-sts.txt) on GitHub Pages.
6
star
32

go-golden

Yet another Go package for working with *.golden test files, with a focus on simplicity.
Go
6
star
33

yank-indent

Emacs minor-mode that ensures pasted (yanked) text has the correct indentation level.
Emacs Lisp
6
star
34

airbrake-statsd

Extends the Airbrake gem to also report exceptions to Esty's StatsD statistics aggregator.
Ruby
6
star
35

greek_easter

Never wonder again when easter is in Greece.
Ruby
6
star
36

facebooker_plus

A Ruby on Rails plugin fixing, extending and adding features to Facebooker, possibly beyond the originally intended scope of Facebooker itself.
Ruby
5
star
37

fancy_input

Easily customizable search suggestion plugin for jQuery, which suggests results directly, rather than search terms. (Formally known as suggest_results)
JavaScript
5
star
38

casecmp

Case-insensitive string comparison, as an API. Because ¯\_(ツ)_/¯
Go
5
star
39

docker-flexget

Simple Docker container for running Flexget
Shell
4
star
40

zynapse

Rails-like MVC framework for PHP5. Currently abandoned and published for educational reasons.
JavaScript
4
star
41

twhois

Whois-like command-line tool and Ruby Gem for Twitter users
Ruby
3
star
42

undent

Go package which removes leading indentation/white-space from strings.
Go
3
star
43

Dockerfiles

Small collection of Docker and Docker Compose files I use to run stuff on my personal laptop
Shell
3
star
44

standardfmt.el

Emacs minor-mode to format JavaScript with standard / semistandard on save.
Emacs Lisp
3
star
45

time_ext

Extends the abilities of Ruby's built-in Time class by building on top of ActiveSupport.
Ruby
3
star
46

960-grid-system-plus

A clone/enhancement of the excellent 960 Grid System by Nathan Smith to fit my personal likes and dislikes.
3
star
47

tab-bar-notch

Adjust tab-bar height for MacBook Pro notch
Emacs Lisp
3
star
48

skyline

Ruby-based interactive shell tools to send terminal commands to instances in Amazon EC2 AutoScaling groups
Ruby
2
star
49

fastmail-rules

Go
2
star
50

kotaku-uk-rss

Small experimental web-scraper to provide a RSS feed of kotaku.co.uk for my own personal use.
Go
2
star
51

skyhook

A customized set of scripts and config files to deploy projects and control services on Amazon EC2.
Ruby
2
star
52

rubocopfmt

DEPRECATED! Easy formatting of Ruby code using rubocop. Analogous to gofmt.
Ruby
2
star
53

envctl

Go package providing test helper functions to temporarily change and restore environment variables.
Go
1
star
54

git-basics

A basic intro to git
1
star
55

tmux-themepack-previews

This holds nothing interesting. Go check out the Tmux Themepack project instead.
Shell
1
star
56

cloudflare-dyndns

Go
1
star
57

rands

Go package providing a suite of functions that use crypto/rand to generate cryptographically secure random strings in various formats.
Go
1
star
58

chef-btsync

A simple and quickly hacked together chef cookbook for installing and configuring BTSync on Ubuntu for my own personal needs.
Ruby
1
star
59

blank_gem

A blank/empty Ruby gem that does nothing.
Ruby
1
star
60

modern_bubbling

An Adium message style.
JavaScript
1
star
61

seedsafe

An attempt to learn about AES encryption.
Go
1
star
62

mje

Go
1
star
63

heartb.it

Ruby
1
star
64

pylight

Quick hack to create an HTTP API for accessing Pygments, and to start teaching myself Python.
Python
1
star
65

bah.io

Static landing page for bah.io
HTML
1
star
66

jimeh.me-api

Experimental project to rewrite my site and blog to run off of an EventMachine-based API. Cause it's fun ^_^
Ruby
1
star
67

play-store-notifier

Small shitty script I originally used to notify myself via email of when the Nexus 4 came back in stock. Now using it to find out when the Nexus 5 becomes available on the Play Store.
Ruby
1
star
68

rbheap

A tool to help with tracking down memory leaks in Ruby.
Go
1
star
69

ps4-20th-tool

A small tool built for educational purposes that attempts to pick apart Sony's 20 Years Of Character's website to find the secret URL.
Go
1
star
70

terraform-cloudflare-email

Terraform module to configure various email related DNS records on Cloudflare.
HCL
1
star