• Stars
    star
    33
  • Rank 771,251 (Top 16 %)
  • Language
    Lua
  • Created over 7 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Powerful memcached client with a shdict caching layer and many other features

Name

lua-resty-memcached-shdict - Powerful memcached client with a shdict caching layer and many other features

Table of Contents

Status

Experimental.

Synopsis

local shdict_memc = require "resty.memcached.shdict"

local function dlog(ctx, ...)
    ngx.log(ngx.DEBUG, "my app: ", ...)
end

local function error_log(ctx, ...)
    ngx.log(ngx.ERR, "my app: ", ...)
end

local function warn(ctx, ...)
    ngx.log(ngx.WARN, "my app: ", ...)
end

local memc_fetch, memc_store =
    shdict_memc.gen_memc_methods{
        tag = "my memcached server tag",
        debug_logger = dlog,
        warn_logger = warn,
        error_logger = error_log,

        locks_shdict_name = "some_lua_shared_dict_name",

        shdict_set = meta_shdict_set,  -- generated by lua-resty-shdict-simple or
                                       -- any other API compatible function factories

        shdict_get = meta_shdict_get,  -- ditto

        disable_shdict = false,  -- optional, default false

        memc_host = "127.0.0.1",
        memc_port = 11211,
        memc_timeout = 200,  -- in ms
        memc_conn_pool_size = 5,
        memc_fetch_retries = 2,  -- optional, default 1
        memc_fetch_retry_delay = 100, -- in ms, optional, default to 100 (ms)

        memc_conn_max_idle_time = 10 * 1000,  -- in ms, for in-pool connections,
                                              -- optional, default to nil

        memc_store_retries = 2,  -- optional, default to 1
        memc_store_retry_delay = 100,  -- in ms, optional, default to 100 (ms)

        store_ttl = 1,  -- in seconds, optional, default to 0 (i.e., never expires)
    }

-- on hot code paths:

local ctx = ngx.ctx

-- in case of failure, can return the stale data with an error message
local data, err = memc_fetch(ctx, key)

-- using the default ttl specified by "store_ttl"
local ok = memc_store(ctx, key, value)

ok = memc_store(ctx, key, value, ttl)  -- overriding the default ttl

Description

This library provides a powerful memcached client that deliver the following important features:

  1. Automatically using shm-based caching layer with OpenResty's lua_shared_dict. This local shm cache layer can also be turned off by setting the disable_shdict to true.
  2. Automatically using server-level (not worker-level) cache locks (based on lua-resty-lock) to avoid duplicate and concurrent queries to the memcached server for the same key when it is a cache miss in the local shm cache.
  3. Automatically handles and logs any errors while accessing the Memcached server or lua_shared_dict.
  4. Automatically uses connection pools for all the Memcached queries.
  5. Automatically returns stale (or expired) data item in the local shm cache when a cache miss happens and some other light threads are already querying Memcached and updating the cache for the same key.
  6. Automatically retry querying Memcached when failures happen (like intermittent network issues).

This library is mostly used for cases that use Memcached or Memcached-compatible servers as the data storage (like Kyoto Tycoon).

Atop this library, the user can further add another per-worker caching layer by employing the lua-resty-lrucache library herself, and/or add asynchronous cache updating mechanism via ngx.timer.at.

Caveats

  • Unlike the lua-resty-memcached library, this client does not escape special characters in the key by default. So it is the caller's responsibility to avoid using special characters in the keys.

Back to TOC

Dependencies

This library depends on the following Lua libraries:

Back to TOC

TODO

  • Add a test suite under t/.

Back to TOC

Author

Yichun "agentzh" Zhang (章亦春) [email protected], CloudFlare Inc.

Back to TOC

Copyright and License

This module is licensed under the BSD license.

Copyright (C) 2016, by CloudFlare Inc.

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Back to TOC

See Also

Back to TOC

More Repositories

1

openresty

High Performance Web Platform Based on Nginx and LuaJIT
C
12,238
star
2

lua-nginx-module

Embed the Power of Lua into NGINX HTTP servers
C
11,133
star
3

nginx-tutorials

Nginx Tutorials
Perl
2,855
star
4

lua-resty-redis

Lua redis client driver for the ngx_lua based on the cosocket API
Lua
1,876
star
5

openresty-systemtap-toolkit

Real-time analysis and diagnostics tools for OpenResty (including NGINX, LuaJIT, ngx_lua, and more) based on SystemTap
Perl
1,651
star
6

headers-more-nginx-module

Set, add, and clear arbitrary output headers in NGINX http servers
C
1,609
star
7

openresty.org

Code and data for the openresty.org site
HTML
1,259
star
8

luajit2

OpenResty's Branch of LuaJIT 2
C
1,196
star
9

echo-nginx-module

An Nginx module for bringing the power of "echo", "sleep", "time" and more to Nginx's config file
C
1,150
star
10

docker-openresty

Docker tooling for OpenResty
Dockerfile
922
star
11

redis2-nginx-module

Nginx upstream module for the Redis 2.0 protocol
C
901
star
12

lua-resty-limit-traffic

Lua library for limiting and controlling traffic in OpenResty/ngx_lua
Lua
808
star
13

lua-resty-core

New FFI-based API for lua-nginx-module
Lua
786
star
14

stream-lua-nginx-module

Embed the power of Lua into NGINX TCP/UDP servers
C
714
star
15

lua-resty-mysql

Nonblocking Lua MySQL driver library for ngx_lua or OpenResty
Lua
699
star
16

stapxx

Simple macro language extentions to systemtap
Perl
689
star
17

sregex

A non-backtracking NFA/DFA-based Perl-compatible regex engine matching on large data streams
C
615
star
18

lua-resty-upstream-healthcheck

Health Checker for Nginx Upstream Servers in Pure Lua
Lua
510
star
19

lua-resty-websocket

WebSocket support for the ngx_lua module (and OpenResty)
Lua
498
star
20

lua-upstream-nginx-module

Nginx C module to expose Lua API to ngx_lua for Nginx upstreams
C
496
star
21

srcache-nginx-module

Transparent subrequest-based caching layout for arbitrary nginx locations.
C
469
star
22

opm

OpenResty Package Manager
Lua
458
star
23

lua-resty-lrucache

Lua-land LRU Cache based on LuaJIT FFI
Lua
434
star
24

test-nginx

Data-driven test scaffold for Nginx C module and OpenResty Lua library development
Perl
434
star
25

lua-resty-string

String utilities and common hash functions for ngx_lua and LuaJIT
Lua
423
star
26

lua-resty-upload

Streaming reader and parser for http file uploading based on ngx_lua cosocket
Lua
396
star
27

set-misc-nginx-module

Various set_xxx directives added to nginx's rewrite module (md5/sha1, sql/json quoting, and many more)
C
384
star
28

drizzle-nginx-module

an nginx upstream module that talks to mysql and drizzle by libdrizzle
C
336
star
29

openresty-gdb-utils

GDB Utilities for OpenResty (including Nginx, ngx_lua, LuaJIT, and more)
Python
331
star
30

lua-resty-dns

DNS resolver for the nginx lua module
Lua
320
star
31

lua-resty-balancer

A generic consistent hash implementation for OpenResty/Lua
Lua
320
star
32

programming-openresty

Programming OpenResty Book
Perl
316
star
33

lua-resty-lock

Simple nonblocking lock API for ngx_lua based on shared memory dictionaries
Lua
303
star
34

openresty-devel-utils

Utilities for nginx module development
Perl
265
star
35

resty-cli

Fancy command-line utilities for OpenResty
Perl
261
star
36

replace-filter-nginx-module

Streaming regular expression replacement in response bodies
C
257
star
37

memc-nginx-module

An extended version of the standard memcached module that supports set, add, delete, and many more memcached commands.
C
212
star
38

lua-resty-memcached

Lua memcached client driver for the ngx_lua based on the cosocket API
Lua
210
star
39

encrypted-session-nginx-module

encrypt and decrypt nginx variable values
C
197
star
40

openresty-packaging

Official OpenResty packaging source and scripts for various Linux distributions and other systems
Makefile
174
star
41

rds-json-nginx-module

An nginx output filter that formats Resty DBD Streams generated by ngx_drizzle and others to JSON
C
154
star
42

xss-nginx-module

Native support for cross-site scripting (XSS) in an nginx
C
147
star
43

mockeagain

Mocking ideally slow network that only allows reading and/or writing one byte at a time
C
128
star
44

lua-resty-shell

Lua module for nonblocking system shell command executions
Perl
122
star
45

lua-tablepool

Lua table recycling pools for LuaJIT
Perl
112
star
46

lua-redis-parser

Lua module for parsing raw redis responses
C
93
star
47

openresty-survey

OpenResty Web App for OpenResty User Survey
HTML
90
star
48

lua-ssl-nginx-module

NGINX C module that extends ngx_http_lua_module for enhanced SSL/TLS capabilities
Lua
86
star
49

opsboy

A rule-based sysadmin tool that helps setting up complex environment for blank machines
Perl
83
star
50

no-pool-nginx

replace nginx's pool mechanism with plain malloc & free to help tools like valgrind
Shell
77
star
51

stream-echo-nginx-module

TCP/stream echo module for NGINX (a port of ngx_http_echo_module)
C
70
star
52

meta-lua-nginx-module

Meta Lua Nginx Module supporting both Http Lua Module and Stream Lua Module
C
66
star
53

array-var-nginx-module

Add support for array-typed variables to nginx config files
C
66
star
54

lemplate

OpenResty/Lua template framework implementing Perl's TT2 templating language
Perl
53
star
55

openresty-con

JavaScript
47
star
56

nginx-dtrace

An nginx fork that adds dtrace USDT probes
C
44
star
57

orbpf-ko

The orbpf (eBPF+) Linux kernel module from OpenResty Inc.
C
37
star
58

lua-resty-shdict-simple

Simple applicaton-oriented interface to the OpenResty shared dictionary API
Perl
32
star
59

lua-resty-signal

Lua library for killing or sending signals to UNIX processes
Perl
31
star
60

luajit2-test-suite

OpenResty's LuaJIT test suite based on Mike Pall's LuaJIT tests
Lua
29
star
61

ngx_postgres

OpenResty's fork of FRiCKLE/ngx_postgres
C
26
star
62

rds-csv-nginx-module

Nginx output filter module to convert Resty-DBD-Streams (RDS) to Comma-Separated Values (CSV)
C
22
star
63

showman-samples

Sample screenplay files for generating our public video tutorials using OpenResty Showman
20
star
64

lua-rds-parser

Resty DBD Stream (RDS) parser for Lua written in C
C
19
star
65

redis-nginx-module

8
star
66

AB-test-http

test http requests between two systems.
Perl
5
star
67

transparency

2
star