steveyen/moxi

moxi - a memcached/membase proxy with energy and pep

Dependencies:

   -- libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev)

   -- libconflate, http://github.com/northscale/libconflate

   -- libvbucket, http://github.com/northscale/libvbucket

As a backwards-compatible (ketama/consistent-hashing) alternative to
libvbucket, you may instead use libmemcached instead of libvbucket...

   -- libmemcached, http://tangent.org/552/libmemcached.html

To use moxi against membase, however, you'll want libvbucket.

----------

To compile moxi (assuming you want libvbucket), after you got the
dependencies built and installed:

  ./config/autorun.sh
  ./configure
  make

For example, if libevent is installed in /opt/local, you'd do...

  ./config/autorun.sh
  ./configure --with-libevent=/opt/local
  make

----------

Using moxi:

To have moxi load a "vBucket" json configuration from a REST/HTTP
server, try...

  moxi http://host:port/url/of/vBucketServerMapJSON

Against a NorthScale server, for example, this would look like...

  moxi http://host:8080/pools/default/bucketsStreamingConfig/default

The above, also, is just shorthand (assuming you have no other flags
specified), for the following explicit command-line...

  moxi -z url=http://host:8080/url/of/vBucketServerMap.json

To get more command line usage info:

  moxi -h

----------

File-based configuration:

You may also provide a configuration file to moxi that holds a static
vBucket server map.  The file (such as vbucket1.cfg) would look
like...

  11211 = {
    "hashAlgorithm": "CRC",
    "numReplicas": 0,
    "serverList": ["memcached_svr1:11311"],
    "vBucketMap":
      [
        [0],
        [0]
      ]
  }

The above configuration would tell moxi to listen on port 11211, and
proxy to memcached_svr1:11311.  To use a static configuration file,
you would start moxi like...

  moxi -z ./vbucket1.cfg

The "./" path prefix is required so that moxi knows you're passing in
a config file.

----------

Tests:

  # To test that moxi still behaves like memcached
  # and passes all the tests that memcached passes...
  #
  make test

  # To test moxi in a simple proxy topology of...
  #  client <-> moxi <-> memcached
  #
  ./t/moxi.pl

  # To test moxi in a chained proxy topology of...
  #   client <-> moxi <-> moxi <-> moxi <-> memcached
  #
  ./t/moxi.pl chain

  # To test moxi in a fanout topology of...
  #   client <---> moxi <---> memcached
  #                       |-> memcached
  #                       |-> memcached
  #                       |-> memcached
  #
  ./t/moxi.pl fanout

  # To test moxi in a fanout and back in again topology of...
  #   client <---> moxi <---> moxi <-> memcached
  #            |-> moxi <-|
  #            |-> moxi <-|
  #            |-> moxi <-|
  #
  ./t/moxi.pl fanoutin

  # To test moxi proxy cases...
  #
  ./moxi -z 11333=localhost:11311 -t 1
  python t/moxi_mock.py

----------

For vbucket development, start the following...

  ruby ./t/rest_mock.rb

Then start a "pretend" memcached server...

  ./moxi -vvv -p 11311

Then...

  ./moxi -vvv -z url=http://localhost:4567/pools/default/bucketsStreaming/default -Z port_listen=11211

Then...

  telnet localhost 11211

----------

More notes:

If using Linux, you need a kernel with epoll (it's better than select()).

epoll isn't in Linux 2.4 yet, but there's a backport at:

    http://www.xmailserver.org/linux-patches/nio-improve.html

You want the epoll-lt patch (level-triggered).

If you're using MacOS, you'll want libevent 1.1 or higher to deal with
a kqueue bug.

Also, be warned that the -k (mlockall) option to memcached might be
dangerous when using a large cache.  Just make sure the memcached machines
don't swap.  memcached does non-blocking network I/O, but not disk.  (it
should never go to disk, or you've lost the whole point of it)

The memcached website is at:

    http://www.danga.com/memcached/

----------

Using moxi with libmemcached (instead of libvbucket)...

If you want to use moxi explicitly using ketama / consistent-hashing,
you'll need to compile moxi with libmemcached.

First, you would configure moxi differently at build-time...

  ./configure --enable-moxi-vbucket=no \
     CFLAGS=<libmemcached/include> \
     LDFLAGS=<libmemcahed/lib>

After building, moxi understands the following kind of command-line...

  moxi -z <port=<more-config>>

  moxi -z <port=<memcached_host:memcached_port(,*)>>

moxi will listen on the given port and accept connections from
upstream memcached clients.  It will forward requests to downstream
memcached servers running on memcached_host:memcached_port.  For
example...

  moxi -z 11211=my_memcached_server:11222

Above, moxi will listen on port 11211 and forward requests
to the memcached running on my_memcached_server that's listening
on port 11222.

You can list more than one memcached_host:memcached_port,
separated by commas.  For example...

  moxi -z 11211=memcached_1:11211,memcached_2:11211

The default downstream port is 11211, so you can also
just use...

  moxi -z 11211=memcached_1,memcached_2

If you have some memcached servers running on the same server,
but on different ports, you can put moxi in front of them
with something like...

  moxi -z 11211=server:11222,server:11233