• This repository has been archived on 05/Jan/2021
  • Stars
    star
    118
  • Rank 283,867 (Top 6 %)
  • Language
    Lua
  • License
    BSD 2-Clause "Sim...
  • Created over 9 years ago
  • Updated about 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

dynamic nginx configuration

NOTICE

This project is no longer maintained and has been archived. This repository will be deleted on March 4th, 2021.


redx

Redx (or redis-nginx) is an embedded lua based approach of having a dynamic configuration of nginx of frontends and backends with redis as its data store. It was inspired by hipache. It has a RESTful API (that runs within nginx itself) to manage the many-to-one relationships between frontends to backends, set configurations, and more.

One of the main benefits of redx is the ability to update your nginx config without needing to reload nginx. This is useful for environments with a highly dynamic topology where backends and frontends are added/removed several times a second (ie PaaS). Also, this allows you to have a single nginx config across multiple nginx servers making it easier to have true high availability and scalability on your load balancing layer.

Redx is licensed under the 2-Clause BSD License.

Project Status

At RStudio, we are using it in production serving all web traffic for shinyapps.io.

How it works

The Components

Redx is composed of two components; the api and main. The api is a restful api embedded in lua and runs within the nginx process. It runs on its own port (for ease of firewall security) and is manages the frontends and backends in the nginx config by editing the redis database.

The other component is main, and this is what takes regular traffic from your users. It looks up the proper backend to proxy the request to based on the host and path.

The fallback

In the event that there isn't a frontend or backend match for an incoming request OR the backend server (ie a cache miss), the request was proxied to a server that isn't responding, the request is sent to the fallback. This is typically your application server, which handles these failure scenario. Several headers are added to the request to help your application server understand the context in which the request is coming in (ie cache miss vs. port not open). In some cases, you may want to update redx by hitting the API and insert the missing frontend and/or backend and send them back to nginx to try again, or maybe you want to forward them to a custom 404 page. Its up to you to decide what the behavior you want it to be.

Performance

At RStudio, we find that redx performs slightly slower than regular redis config files. Of course, this makes sense, as you're now querying a remote redis server to lookup frontends and backends, instead of caching all that in local memory. In our unofficial benchmarks, we see a 10ms increase in response time with using a third party redis hosting service. Of course latency to your redis server is a big factor. Redx though, keeps a pool of connections that are reused instead of making separate calls on each request.

That being said, the payoff of having dynamic configuration and being able to easily do high availability with active active nginx load balancers is well worth the 10ms cost in my opinion. Each environment is different and has different requirements, goals, etc. So its up to you to decide what is worth what.

Requirements

lapis version 1.0.4-1

openresty 1.7.2 or greater

A redis server

Setup Dev Environment

Setup and start vagrant

  vagrant plugin install vagrant-berkshelf --plugin-version '>= 2.0.1'
  vagrant plugin install vagrant-omnibus
  vagrant up

The redx code on your local workstation is run within vagrant (due to sharing the redx directory with vagrant at /home/vagrant/redx). As you make code changes, they should take affect immediately and do not require reloading nginx. You will however need to reload nginx when you change the nginx config located vagrant://etc/nginx/sites-available/redx.conf. To see redx logs, see /var/log/nginx/[access,error].log

Git Hooks

It is recommended that you setup two git hooks.

The pre-commit hook (./git/hooks/pre-commit) should be used to ensure you don't make changes to the moonscript code without recompiling the lua code.

#!/bin/sh

moonc -t lua/ .

The pre-push hook (./git/hooks/pre-push) should be used to ensure you've run all tests and they all pass.

#!/bin/sh

busted lua/spec/

Testing

Redx uses a testing framework, busted, to run integration tests. To run these tests, execute busted lua/spec. Continuous integration is setup with travis ci.

Configuration

The configuration of redx is loaded from lua/conf/config.lua. This file consists of the following configuration options

redis_host

An IP address or hostname to a redis server

redis_port

An port number of the redis server to connect to

redis_password

If you have a redis password, put it here. If you don't leave it an empty string (ie '')

redis_timeout

This is the connection timeout for any redis connection, in milliseconds

redis_keepalive_pool_size

The pool size of keepalive connections to maintain, per nginx worker.

redis_keepalive_max_idle_timeout

max idle timeout for keepalive connection, in milliseconds

max_path_length

The max number of parts to the path to look up. This defines the max length of a path to be to looked up to a corresponding frontend in redis.

Say in your service, your path always consists of an account name and service name (ie http://sasservice.com/jdoe/app1). So the max path length you want to support for your application here is 2. If the user, comes into nginx with more to the path than that (ie http://sasservice.com/jdoe/app1/static/js/base.js), but when request comes in, redx will only search for "sasservice.com/jdoe/app1" first and "sasservice.com/jdoe" second and "sasservice.com" third, for frontends in the database.

For another example, say you only want to route traffic based on the domain (ie 'chad.myserver.com'). Setting the max_path_length to 0 will cause redx to only look for frontends on the domain with no path.

plugins

A list of plugins to enable. Plugins are executed in the order they are given. If you wish to pass a parameter to a plugin, make a plugin an array, where the first element is the plugin name and the second is the parameter.

Here is an example.

M.plugins = {
  'stickiness',
  { 'score', 'most' },
  'random'
}
session_length

The amount of time (in seconds) you wish the session cookie to keep alive. This is applicable when using cookies as a user specific persistence datastore (ie stickiness).

default_score

The default score is the score that is inserted into backends in the case where a score is not provided (ie batch updating). If this config option isn't specified, it defaults to 0 (zero).

API

The api is documented here

Plugins

Redx has a plugin architecture to allow others to easily expand its capability to do more (publicly or privately).

See the plugins documentation

More Repositories

1

cheatsheets

Posit Cheat Sheets - Can also be found at https://posit.co/resources/cheatsheets/.
TeX
5,382
star
2

shiny

Easy interactive web applications with R
R
5,139
star
3

rstudio

RStudio is an integrated development environment (IDE) for R
Java
4,432
star
4

bookdown

Authoring Books and Technical Documents with R Markdown
JavaScript
3,525
star
5

rmarkdown

Dynamic Documents for R
R
2,737
star
6

shiny-examples

JavaScript
1,909
star
7

gt

Easily generate information-rich, publication-quality tables from R
R
1,853
star
8

blogdown

Create Blogs and Websites with R Markdown
R
1,691
star
9

reticulate

R Interface to Python
R
1,604
star
10

webinars

Code and slides for RStudio webinars
HTML
1,510
star
11

rticles

LaTeX Journal Article Templates for R Markdown
TeX
1,402
star
12

plumber

Turn your R code into a web API.
R
1,352
star
13

tensorflow

TensorFlow for R
R
1,313
star
14

renv

renv: Project environments for R.
R
949
star
15

shinydashboard

Shiny Dashboarding framework
CSS
852
star
16

pagedown

Paginate the HTML Output of R Markdown with CSS for Print
R
850
star
17

keras

R Interface to Keras
R
811
star
18

pointblank

Data quality assessment and metadata reporting for data frames and database tables
R
794
star
19

flexdashboard

Easy interactive dashboards for R
JavaScript
778
star
20

leaflet

R Interface to Leaflet Maps
JavaScript
775
star
21

rmarkdown-book

R Markdown: The Definitive Guide (published by Chapman & Hall/CRC in July 2018)
RMarkdown
738
star
22

ggvis

Interactive grammar of graphics for R
R
709
star
23

rstudio-conf

Materials for rstudio::conf
HTML
694
star
24

learnr

Interactive Tutorials with R Markdown
R
692
star
25

shiny-server

Host Shiny applications over the web.
JavaScript
686
star
26

RStartHere

A guide to some of the most useful R Packages that we know about
R
655
star
27

py-shiny

Shiny for Python
Python
627
star
28

DT

R Interface to the jQuery Plug-in DataTables
JavaScript
577
star
29

rmarkdown-cookbook

R Markdown Cookbook. A range of tips and tricks to make better use of R Markdown.
RMarkdown
562
star
30

blastula

Easily send great-looking HTML email messages from R
R
522
star
31

r2d3

R Interface to D3 Visualizations
R
506
star
32

bookdown-demo

A minimal book example using bookdown
CSS
476
star
33

hex-stickers

RStudio hex stickers
R
426
star
34

distill

Distill for R Markdown
HTML
418
star
35

bslib

Tools for theming Shiny and R Markdown via Bootstrap 3, 4, or 5.
SCSS
408
star
36

packrat

Packrat is a dependency management system for R
R
394
star
37

tufte

Tufte Styles for R Markdown Documents
R
385
star
38

dygraphs

R interface to dygraphs
JavaScript
358
star
39

revealjs

R Markdown Format for reveal.js Presentations
JavaScript
316
star
40

pins-r

Pin, Discover and Share Resources
R
295
star
41

fontawesome

Easily insert FontAwesome icons into R Markdown docs and Shiny apps
R
288
star
42

profvis

Visualize R profiling data
JavaScript
285
star
43

crosstalk

Inter-htmlwidget communication for R (with and without Shiny)
JavaScript
283
star
44

config

config package for R
R
248
star
45

pool

Object Pooling in R
R
241
star
46

thematic

Theme ggplot2, lattice, and base graphics based on a few simple settings.
R
237
star
47

Intro

Course materials for "Introduction to Data Science with R", a video course by RStudio and O'Reilly Media
R
234
star
48

shinytest

Automated testing for shiny apps
JavaScript
227
star
49

tinytex-releases

Windows/macOS/Linux binaries and installation methods of TinyTeX
PowerShell
226
star
50

httpuv

HTTP and WebSocket server package for R
C
217
star
51

nomnoml

Sassy 'UML' Diagrams for R
JavaScript
216
star
52

shinymeta

Record and expose Shiny app logic using metaprogramming
R
212
star
53

htmltools

Tools for HTML generation and output
R
201
star
54

shinyuieditor

A GUI for laying out a Shiny application that generates clean and human-readable UI code
JavaScript
198
star
55

promises

A promise library for R
R
191
star
56

vetiver-r

Version, share, deploy, and monitor models
R
170
star
57

rstudioapi

Safely access RStudio's API (when available)
R
161
star
58

gradethis

Tools for teachers to use with learnr
R
161
star
59

concept-maps

Concept maps for all things data science
HTML
158
star
60

master-the-tidyverse

Course contents for Master the Tidyverse
158
star
61

shinythemes

Themes for Shiny
R
152
star
62

ShinyDeveloperConference

Materials collected from the First Shiny Developer Conference Palo Alto, CA January 30-31 2016
HTML
151
star
63

chromote

Chrome Remote Interface for R
R
143
star
64

shiny-gallery

Code and other documentation for apps in the Shiny Gallery ✨
HTML
139
star
65

sortable

R htmlwidget for Sortable.js
R
124
star
66

rsconnect

Publish Shiny Applications, RMarkdown Documents, Jupyter Notebooks, Plumber APIs, and more
R
120
star
67

reactlog

Shiny Reactivity Visualizer
JavaScript
119
star
68

bigdataclass

Two-day workshop that covers how to use R to interact databases and Spark
R
113
star
69

shinyapps

Deploy Shiny applications to ShinyApps
110
star
70

shinyloadtest

Tools for load testing Shiny applications
HTML
109
star
71

webshot2

Take screenshots of web pages from R
R
108
star
72

shinyvalidate

Input validation package for the Shiny web framework
JavaScript
108
star
73

r-docker

Docker images for R
Dockerfile
105
star
74

miniUI

R
102
star
75

sass

Sass compiler package for R
C++
102
star
76

r-system-requirements

System requirements for R packages
Shell
100
star
77

shinytest2

R
96
star
78

keras-customer-churn

Customer Churn Shiny Application
R
96
star
79

r-builds

an opinionated environment for compiling R
Shell
91
star
80

addinexamples

An R package showcasing how RStudio addins can be registered and used.
R
85
star
81

r-manuals

A re-styled version of the R manuals
R
83
star
82

shinyapps-package-dependencies

Collection of bash scripts that install R package system dependencies
R
74
star
83

markdown

The first generation of Markdown rendering for R (born in 2012). Originally based on the C library sundown. Now based on commonmark. Note that this package is markdown, not *rmarkdown*.
R
72
star
84

R-Websockets

HTML 5 Websockets implementation for R
R
68
star
85

webdriver

WebDriver client in R
R
68
star
86

beyond-dashboard-fatigue

Materials for the RStudio webinar 'Beyond Dashboard Fatigue'
R
66
star
87

cloudml

R interface to Google Cloud Machine Learning Engine
R
65
star
88

shinylive

Run Shiny on Python (compiled to wasm) in the browser
TypeScript
61
star
89

rstudio-conf-2022-program

rstudio::conf(2022, "program")
R
61
star
90

rstudio-docker-products

Docker images for RStudio Professional Products
Just
59
star
91

bookdown.org

Source documents to generate the bookdown.org website
R
59
star
92

education.rstudio.com

CSS
57
star
93

tfestimators

R interface to TensorFlow Estimators
R
57
star
94

sparkDemos

HTML
54
star
95

vetiver-python

Version, share, deploy, and monitor models.
Python
54
star
96

tfprobability

R interface to TensorFlow Probability
R
54
star
97

shiny-incubator

Examples and ideas that don't belong in the core Shiny package and aren't officially supported.
JavaScript
53
star
98

connections

https://rstudio.github.io/connections/
R
52
star
99

swagger

Swagger is a collection of HTML, Javascript, and CSS assets that dynamically generate beautiful documentation from a Swagger-compliant API.
HTML
51
star
100

leaflet.mapboxgl

Extends the R Leaflet package with a Mapbox GL JS plugin to allow easy drawing of vector tile layers.
R
49
star