openresty/stream-lua-nginx-module openresty/stream-lua-nginx-module

  • Stars
    star
    723
  • Rank 62,657 (Top 2 %)
  • Language
    C
  • License
    BSD 2-Clause "Sim...
  • Created almost 9 years ago
  • Updated 3 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
openresty/stream-lua-nginx-module
github

openresty

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Embed the power of Lua into NGINX TCP/UDP servers

Name

ngx_stream_lua_module - Embed the power of Lua into Nginx stream/TCP Servers.

This module is a core component of OpenResty. If you are using this module, then you are essentially using OpenResty.

This module is not distributed with the Nginx source. See the installation instructions.

Table of Contents

Status

Production ready.

Version

This document describes ngx_stream_lua v0.0.8, which was released on 2 July, 2020.

Synopsis

events {
    worker_connections 1024;
}

stream {
    # define a TCP server listening on the port 1234:
    server {
        listen 1234;

        content_by_lua_block {
            ngx.say("Hello, Lua!")
        }
    }
}

Set up as an SSL TCP server:

stream {
    server {
        listen 4343 ssl;

        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5;
        ssl_certificate     /path/to/cert.pem;
        ssl_certificate_key /path/to/cert.key;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

        content_by_lua_block {
            local sock = assert(ngx.req.socket(true))
            local data = sock:receive()  -- read a line from downstream
            if data == "thunder!" then
                ngx.say("flash!")  -- output data
            else
                ngx.say("boom!")
            end
            ngx.say("the end...")
        }
    }
}

Listening on a UNIX domain socket is also supported:

stream {
    server {
        listen unix:/tmp/nginx.sock;

        content_by_lua_block {
            ngx.say("What's up?")
            ngx.flush(true)  -- flush any pending output and wait
            ngx.sleep(3)  -- sleeping for 3 sec
            ngx.say("Bye bye...")
        }
    }
}

Back to TOC

Description

This is a port of the ngx_http_lua_module to the Nginx "stream" subsystem so as to support generic stream/TCP clients.

The available Lua APIs and Nginx directives remain the same as those of the ngx_http_lua module.

Back to TOC

Directives

The following directives are ported directly from ngx_http_lua. Please check the documentation of ngx_http_lua for more details about their usage and behavior.

The send_timeout directive in the Nginx "http" subsystem is missing in the "stream" subsystem. As such, ngx_stream_lua_module uses the lua_socket_send_timeout directive for this purpose instead.

Note: the lingering close directive that used to exist in older version of stream_lua_nginx_module has been removed and can now be simulated with the newly added tcpsock:shutdown API if necessary.

Back to TOC

preread_by_lua_block

syntax: preread_by_lua_block { lua-script }

context: stream, server

phase: preread

Acts as a preread phase handler and executes Lua code string specified in lua-script for every connection (or packet in datagram mode). The Lua code may make API calls and is executed as a new spawned coroutine in an independent global environment (i.e. a sandbox).

It is possible to acquire the raw request socket using ngx.req.socket and receive data from or send data to the client. However, keep in mind that calling the receive() method of the request socket will consume the data from the buffer and such consumed data will not be seen by handlers further down the chain.

The preread_by_lua_block code will always run at the end of the preread processing phase unless preread_by_lua_no_postpone is turned on.

This directive was first introduced in the v0.0.3 release.

Back to TOC

preread_by_lua_file

syntax: preread_by_lua_file <path-to-lua-script-file>

context: stream, server

phase: preread

Equivalent to preread_by_lua_block, except that the file specified by <path-to-lua-script-file> contains the Lua code or LuaJIT bytecode to be executed.

Nginx variables can be used in the <path-to-lua-script-file> string to provide flexibility. This however carries some risks and is not ordinarily recommended.

When a relative path like foo/bar.lua is given, it will be turned into the absolute path relative to the server prefix path determined by the -p PATH command-line option given when starting the Nginx server.

When the Lua code cache is turned on (by default), the user code is loaded once at the first connection and cached. The Nginx config must be reloaded each time the Lua source file is modified. The Lua code cache can be temporarily disabled during development by switching lua_code_cache off in nginx.conf to avoid having to reload Nginx.

This directive was first introduced in the v0.0.3 release.

Back to TOC

log_by_lua_block

syntax: log_by_lua_block { lua-script }

context: stream, server

phase: log

Runs the Lua source code specified as <lua-script> during the log request processing phase. This does not replace the current access logs, but runs before.

Yielding APIs such as ngx.req.socket, ngx.socket.*, ngx.sleep, or ngx.say are not available in this phase.

This directive was first introduced in the v0.0.3 release.

Back to TOC

log_by_lua_file

syntax: log_by_lua_file <path-to-lua-script-file>

context: stream, server

phase: log

Equivalent to log_by_lua_block, except that the file specified by <path-to-lua-script-file> contains the Lua code or LuaJIT bytecode to be executed.

Nginx variables can be used in the <path-to-lua-script-file> string to provide flexibility. This however carries some risks and is not ordinarily recommended.

When a relative path like foo/bar.lua is given, it will be turned into the absolute path relative to the server prefix path determined by the -p PATH command-line option given when starting the Nginx server.

When the Lua code cache is turned on (by default), the user code is loaded once at the first connection and cached. The Nginx config must be reloaded each time the Lua source file is modified. The Lua code cache can be temporarily disabled during development by switching lua_code_cache off in nginx.conf to avoid having to reload Nginx.

This directive was first introduced in the v0.0.3 release.

Back to TOC

lua_add_variable

syntax: lua_add_variable $var

context: stream

Add the variable $var to the "stream" subsystem and makes it changeable. If $var already exists, this directive will do nothing.

By default, variables added using this directive are considered "not found" and reading them using ngx.var will return nil. However, they could be re-assigned via the ngx.var.VARIABLE API at any time.

This directive was first introduced in the v0.0.4 release.

Back to TOC

preread_by_lua_no_postpone

syntax: preread_by_lua_no_postpone on|off

context: stream

Controls whether or not to disable postponing preread_by_lua* directives to run at the end of the preread processing phase. By default, this directive is turned off and the Lua code is postponed to run at the end of the preread phase.

This directive was first introduced in the v0.0.4 release.

Back to TOC

Nginx API for Lua

Many Lua API functions are ported from ngx_http_lua. Check out the official manual of ngx_http_lua for more details on these Lua API functions.

This module fully supports the new variable subsystem inside the Nginx stream core. You may access any built-in variables provided by the stream core or other stream modules.

Only raw request sockets are supported, for obvious reasons. The raw argument value is ignored and the raw request socket is always returned. Unlike ngx_http_lua, you can still call output API functions like ngx.say, ngx.print, and ngx.flush after acquiring the raw request socket via this function.

When the stream server is in UDP mode, reading from the downstream socket returned by the ngx.req.socket call will only return the content of a single packet. Therefore the reading call will never block and will return nil, "no more data" when all the data from the datagram has been consumed. However, you may choose to send multiple UDP packets back to the client using the downstream socket.

The raw TCP sockets returned by this module will contain the following extra method:

Back to TOC

reqsock:receiveany

syntax: data, err = reqsock:receiveany(max)

context: content_by_lua*, ngx.timer.*, ssl_certificate_by_lua*

This method is similar to tcpsock:receiveany method

This method was introduced into stream-lua-nginx-module since v0.0.8.

Back to TOC

tcpsock:shutdown

syntax: ok, err = tcpsock:shutdown("send")

context: content_by_lua*

Shuts down the write part of the request socket, prevents all further writing to the client and sends TCP FIN, while keeping the reading half open.

Currently only the "send" direction is supported. Using any parameters other than "send" will return an error.

If you called any output functions (like ngx.say) before calling this method, consider use ngx.flush(true) to make sure all busy buffers are complely flushed before shutting down the socket. If any busy buffers were detected, this method will return nil will error message "socket busy writing".

This feature is particularly useful for protocols that generate a response before actually finishing consuming all incoming data. Normally, the kernel will send RST to the client when tcpsock:close is called without emptying the receiving buffer first. Calling this method will allow you to keep reading from the receiving buffer and prevents RST from being sent.

You can also use this method to simulate lingering close similar to that provided by the ngx_http_core_module for protocols in need of such behavior. Here is an example:

local LINGERING_TIME = 30 -- 30 seconds
local LINGERING_TIMEOUT = 5000 -- 5 seconds

local ok, err = sock:shutdown("send")
if not ok then
    ngx.log(ngx.ERR, "failed to shutdown: ", err)
    return
end

local deadline = ngx.time() + LINGERING_TIME

sock:settimeouts(nil, nil, LINGERING_TIMEOUT)

repeat
    local data, _, partial = sock:receive(1024)
until (not data and not partial) or ngx.time() >= deadline

Back to TOC

reqsock:peek

syntax: ok, err = reqsock:peek(size)

context: preread_by_lua*

Peeks into the preread buffer that contains downstream data sent by the client without consuming them. That is, data returned by this API will still be forwarded upstream in later phases.

This function takes a single required argument, size, which is the number of bytes to be peeked. Repeated calls to this function always returns data from the beginning of the preread buffer.

Note that preread phase happens after the TLS handshake. If the stream server was configured with TLS enabled, the returned data will be in clear text.

If preread buffer does not have the requested amount of data, then the current Lua thread will be yielded until more data is available, preread_buffer_size has been exceeded, or preread_timeout has elapsed. Successful calls always return the requested amounts of data, that is, no partial data will be returned.

When preread_buffer_size has been exceeded, the current stream session will be terminated with the session status code 400 immediately by the stream core module, with error message "preread buffer full" that will be printed to the error log.

When preread_timeout has been exceeded, the current stream session will be terminated with the session status code 200 immediately by the stream core module.

In both cases, no further processing on the session is possible (except log_by_lua*). The connection will be closed by the stream core module automatically.

Note that this API cannot be used if consumption of client data has occurred. For example, after calling reqsock:receive. If such an attempt was made, the Lua error "attempt to peek on a consumed socket" will be thrown. Consuming client data after calling this API is allowed and safe.

Here is an example of using this API:

local sock = assert(ngx.req.socket())

local data = assert(sock:peek(1)) -- peek the first 1 byte that contains the length
data = string.byte(data)

data = assert(sock:peek(data + 1)) -- peek the length + the size byte

local payload = data:sub(2) -- trim the length byte to get actual payload

ngx.log(ngx.INFO, "payload is: ", payload)

This API was first introduced in the v0.0.6 release.

Back to TOC

Back to TOC

TODO

  • Add new directives access_by_lua_block and access_by_lua_file.
  • Add lua_postpone_output to emulate the postpone_output directive.

Back to TOC

Nginx Compatibility

The latest version of this module is compatible with the following versions of Nginx:

  • 1.19.x (last tested: 1.19.3)
  • 1.17.x (last tested: 1.17.8)
  • 1.15.x (last tested: 1.15.8)
  • 1.13.x (last tested: 1.13.6)

Nginx cores older than 1.13.6 (exclusive) are not tested and may or may not work. Use at your own risk!

Back to TOC

Installation

It is highly recommended to use OpenResty releases which bundle Nginx, ngx_http_lua, ngx_stream_lua, (this module), LuaJIT, as well as other powerful companion Nginx modules and Lua libraries.

It is discouraged to build this module with Nginx yourself since it is tricky to set up exactly right.

Note that Nginx, LuaJIT, and OpenSSL official releases have various limitations and long standing bugs that can cause some of this module's features to be disabled, not work properly, or run slower. Official OpenResty releases are recommended because they bundle OpenResty's optimized LuaJIT 2.1 fork and Nginx/OpenSSL patches.

Alternatively, ngx_stream_lua can be manually compiled into Nginx:

  1. LuaJIT can be downloaded from the latest release of OpenResty's LuaJIT fork. The official LuaJIT 2.x releases are also supported, although performance will be significantly lower for reasons elaborated above
  2. Download the latest version of ngx_stream_lua HERE
  3. Download the latest supported version of Nginx HERE (See Nginx Compatibility)

Build the source with this module:

wget 'https://nginx.org/download/nginx-1.13.6.tar.gz'
tar -xzvf nginx-1.13.6.tar.gz
cd nginx-1.13.6/

# tell nginx's build system where to find LuaJIT 2.1:
export LUAJIT_LIB=/path/to/luajit/lib
export LUAJIT_INC=/path/to/luajit/include/luajit-2.1

# Here we assume Nginx is to be installed under /opt/nginx/.
./configure --prefix=/opt/nginx \
        --with-ld-opt="-Wl,-rpath,/path/to/luajit-or-lua/lib" \
        --with-stream \
        --with-stream_ssl_module \
        --add-module=/path/to/stream-lua-nginx-module

# Build and install
make -j4
make install

You may use --without-http if you do not wish to use this module with the "http" subsystem. ngx_stream_lua will work perfectly fine without the "http" subsystem.

Back to TOC

Community

Back to TOC

English Mailing List

The openresty-en mailing list is for English speakers.

Back to TOC

Chinese Mailing List

The openresty mailing list is for Chinese speakers.

Back to TOC

Code Repository

The code repository of this project is hosted on GitHub at openresty/stream-lua-nginx-module.

Back to TOC

Bugs and Patches

Please submit bug reports, wishlists, or patches by

  1. creating a ticket on the GitHub Issue Tracker,
  2. or posting to the OpenResty community.

Back to TOC

Acknowledgments

We appreciate Kong Inc. for kindly sponsoring OpenResty Inc. on the following work:

  • Compatibility with Nginx core 1.13.3.
  • Development of meta-lua-nginx-module to make code sharing between this module and lua-nginx-module possible.
  • balancer_by_lua_*, preread_by_lua_*, log_by_lua_* and ssl_certby_lua* phases support.
  • reqsock:peek API support.

Back to TOC

Copyright and License

This module is licensed under the BSD license.

Copyright (C) 2009-2019, by Yichun "agentzh" Zhang (章亦春) [email protected], OpenResty Inc.

Copyright (C) 2009-2016, by Xiaozhe Wang (chaoslawful) [email protected].

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Back to TOC

See Also

Back to TOC

More Repositories

1

openresty

High Performance Web Platform Based on Nginx and LuaJIT
C
12,431
star
2

lua-nginx-module

Embed the Power of Lua into NGINX HTTP servers
C
11,224
star
3

nginx-tutorials

Nginx Tutorials
Perl
2,864
star
4

lua-resty-redis

Lua redis client driver for the ngx_lua based on the cosocket API
Lua
1,893
star
5

openresty-systemtap-toolkit

Real-time analysis and diagnostics tools for OpenResty (including NGINX, LuaJIT, ngx_lua, and more) based on SystemTap
Perl
1,653
star
6

headers-more-nginx-module

Set, add, and clear arbitrary output headers in NGINX http servers
C
1,636
star
7

openresty.org

Code and data for the openresty.org site
HTML
1,262
star
8

luajit2

OpenResty's Branch of LuaJIT 2
C
1,229
star
9

echo-nginx-module

An Nginx module for bringing the power of "echo", "sleep", "time" and more to Nginx's config file
C
1,162
star
10

docker-openresty

Docker tooling for OpenResty
Dockerfile
934
star
11

redis2-nginx-module

Nginx upstream module for the Redis 2.0 protocol
C
902
star
12

lua-resty-limit-traffic

Lua library for limiting and controlling traffic in OpenResty/ngx_lua
Lua
815
star
13

lua-resty-core

New FFI-based API for lua-nginx-module
Lua
797
star
14

lua-resty-mysql

Nonblocking Lua MySQL driver library for ngx_lua or OpenResty
Lua
703
star
15

stapxx

Simple macro language extentions to systemtap
Perl
692
star
16

sregex

A non-backtracking NFA/DFA-based Perl-compatible regex engine matching on large data streams
C
616
star
17

lua-resty-upstream-healthcheck

Health Checker for Nginx Upstream Servers in Pure Lua
Lua
513
star
18

lua-resty-websocket

WebSocket support for the ngx_lua module (and OpenResty)
Lua
503
star
19

lua-upstream-nginx-module

Nginx C module to expose Lua API to ngx_lua for Nginx upstreams
C
499
star
20

srcache-nginx-module

Transparent subrequest-based caching layout for arbitrary nginx locations.
C
474
star
21

opm

OpenResty Package Manager
Lua
458
star
22

test-nginx

Data-driven test scaffold for Nginx C module and OpenResty Lua library development
Perl
438
star
23

lua-resty-lrucache

Lua-land LRU Cache based on LuaJIT FFI
Lua
437
star
24

lua-resty-string

String utilities and common hash functions for ngx_lua and LuaJIT
Lua
426
star
25

lua-resty-upload

Streaming reader and parser for http file uploading based on ngx_lua cosocket
Lua
401
star
26

set-misc-nginx-module

Various set_xxx directives added to nginx's rewrite module (md5/sha1, sql/json quoting, and many more)
C
388
star
27

drizzle-nginx-module

an nginx upstream module that talks to mysql and drizzle by libdrizzle
C
336
star
28

openresty-gdb-utils

GDB Utilities for OpenResty (including Nginx, ngx_lua, LuaJIT, and more)
Python
335
star
29

lua-resty-balancer

A generic consistent hash implementation for OpenResty/Lua
Lua
325
star
30

lua-resty-dns

DNS resolver for the nginx lua module
Lua
321
star
31

programming-openresty

Programming OpenResty Book
Perl
317
star
32

lua-resty-lock

Simple nonblocking lock API for ngx_lua based on shared memory dictionaries
Lua
304
star
33

openresty-devel-utils

Utilities for nginx module development
Perl
266
star
34

resty-cli

Fancy command-line utilities for OpenResty
Perl
263
star
35

replace-filter-nginx-module

Streaming regular expression replacement in response bodies
C
258
star
36

memc-nginx-module

An extended version of the standard memcached module that supports set, add, delete, and many more memcached commands.
C
213
star
37

lua-resty-memcached

Lua memcached client driver for the ngx_lua based on the cosocket API
Lua
210
star
38

encrypted-session-nginx-module

encrypt and decrypt nginx variable values
C
198
star
39

openresty-packaging

Official OpenResty packaging source and scripts for various Linux distributions and other systems
Makefile
176
star
40

rds-json-nginx-module

An nginx output filter that formats Resty DBD Streams generated by ngx_drizzle and others to JSON
C
153
star
41

xss-nginx-module

Native support for cross-site scripting (XSS) in an nginx
C
148
star
42

mockeagain

Mocking ideally slow network that only allows reading and/or writing one byte at a time
C
129
star
43

lua-resty-shell

Lua module for nonblocking system shell command executions
Perl
124
star
44

lua-tablepool

Lua table recycling pools for LuaJIT
Perl
112
star
45

lua-redis-parser

Lua module for parsing raw redis responses
C
93
star
46

openresty-survey

OpenResty Web App for OpenResty User Survey
HTML
90
star
47

lua-ssl-nginx-module

NGINX C module that extends ngx_http_lua_module for enhanced SSL/TLS capabilities
Lua
86
star
48

opsboy

A rule-based sysadmin tool that helps setting up complex environment for blank machines
Perl
85
star
49

no-pool-nginx

replace nginx's pool mechanism with plain malloc & free to help tools like valgrind
Shell
77
star
50

stream-echo-nginx-module

TCP/stream echo module for NGINX (a port of ngx_http_echo_module)
C
69
star
51

meta-lua-nginx-module

Meta Lua Nginx Module supporting both Http Lua Module and Stream Lua Module
C
66
star
52

array-var-nginx-module

Add support for array-typed variables to nginx config files
C
66
star
53

lemplate

OpenResty/Lua template framework implementing Perl's TT2 templating language
Perl
54
star
54

openresty-con

JavaScript
48
star
55

nginx-dtrace

An nginx fork that adds dtrace USDT probes
C
44
star
56

orbpf-ko

The orbpf (eBPF+) Linux kernel module from OpenResty Inc.
C
37
star
57

lua-resty-memcached-shdict

Powerful memcached client with a shdict caching layer and many other features
Lua
33
star
58

lua-resty-signal

Lua library for killing or sending signals to UNIX processes
Perl
32
star
59

lua-resty-shdict-simple

Simple applicaton-oriented interface to the OpenResty shared dictionary API
Perl
32
star
60

luajit2-test-suite

OpenResty's LuaJIT test suite based on Mike Pall's LuaJIT tests
Lua
29
star
61

ngx_postgres

OpenResty's fork of FRiCKLE/ngx_postgres
C
26
star
62

rds-csv-nginx-module

Nginx output filter module to convert Resty-DBD-Streams (RDS) to Comma-Separated Values (CSV)
C
22
star
63

showman-samples

Sample screenplay files for generating our public video tutorials using OpenResty Showman
20
star
64

lua-rds-parser

Resty DBD Stream (RDS) parser for Lua written in C
C
19
star
65

redis-nginx-module

8
star
66

AB-test-http

test http requests between two systems.
Perl
5
star
67

transparency

2
star