overload-protection
Load detection and shedding capabilities for http, express, restify, and koa
About
overload-protection provides integration for your framework of choice.
If a threshold is crossed for a given metric, overload-protection
will send an HTTP 503 Service Unavailable response, with (by default)
a Retry-After header, instructing the client (e.g. a browser or load balancer) to
retry after a given amount of seconds.
Current supported metrics are:
- event loop delay (is the JavaScript thread blocking too long)
- Used Heap Memory
- Total Resident Set Size
For a great explanation of Used Heap Memory vs Resident Set Size see Daniel Khans article at https://www.dynatrace.com/blog/understanding-garbage-collection-and-hunting-memory-leaks-in-node-js
Usage
Create a config object for your thresholds (and other overload-protection)
options.
const protectCfg = {
production: process.env.NODE_ENV === 'production', // if production is false, detailed error messages are exposed to the client
clientRetrySecs: 1, // Retry-After header, in seconds (0 to disable) [default 1]
sampleInterval: 5, // sample rate, milliseconds [default 5]
maxEventLoopDelay: 42, // maximum detected delay between event loop ticks [default 42]
maxHeapUsedBytes: 0, // maximum heap used threshold (0 to disable) [default 0]
maxRssBytes: 0, // maximum rss size threshold (0 to disable) [default 0]
errorPropagationMode: false, // dictate behavior: take over the response
// or propagate an error to the framework [default false]
logging: false, // set to string for log level or function to pass data to
logStatsOnReq: false // set to true to log stats on every requests
}Then pass the framework we're integrating with along with the configuration object.
For instance with Express we would do:
const app = require('express')()
const protect = require('overload-protection')('express', protectCfg)
app.use(protect)With middleware based frameworks, always put the overload-protection middleware
first. In default mode this means overload-protection will take over the response
and prevent any other middleware from executing (thus taking further potential pressure off
of the process).
Restify, and Koa all work in much the same way, call the overload-protection
module with the name of the framework, a config object and pass the resulting protect
instance to app.use β e.g. Koa would be:
const Koa = require('koa')
const protect = require('overload-protection')('koa', protectCfg)
const app = new Koa()
app.use(protect)For pure core HTTP the overload-protection instance can be called
at the top of the request handler function. With two arguments (just req and res)
the function will return true if protection/shedding has been provided, or false
if not. If overload-protection has taken over (the true case), then we should
exit the function and do no further work:
const http = require('http')
const protect = require('overload-protection')('http', protectCfg)
http.createServer(function (req, res) {
if (protect(req, res) === true) return
res.end('content')
})With three arguments (the third argument being a callback), the rest of the work should be done within the supplied callback.
const http = require('http')
const protect = require('overload-protection')('http', protectCfg)
http.createServer(function (req, res) {
protect(req, res, function () {
// when errorPropagationMode mode is false, will *only*
// be called if load shedding didn't occur
// (if it was true we'd need to check for an Error object as first arg)
res.end('content')
})
})Installation
npm install overload-protection --saveTests
npm install
npm testBenchmark
The overhead of using overload-protection is minimal, run the benchmarks to conduct
comparative profiling of using overload-protection versus not using it for each supported framework.
npm run benchmarksAPI
require('overload-protection') => (framework, opts) => instance
The framework argument is non-optional. It's a string and may be one of:
- express
- koa
- restify
- http
The opts argument is optional, as are all properties.
Options (particularly thresholds) are quite sensitive and highly relevant on a case by case basis. Possible options are as follows:
production: process.env.NODE_ENV === 'production'
The production option determines whether the client receives an error message
detailing the surpassed threshold(s). (It may also be used in future for other such
good practices or performance trade-offs).
clientRetrySecs: 1
By default, overload-protection will add a header to the 503 response
called Retry-After. It's up to the client to honour this header, which
instructs the client on how many seconds to wait between retries.
Defaults to 1 seconds.
sampleInterval: 5
In order to establish whether a threshold has been crossed, the metrics are sampled at a regular interval. The interval defaults to 5 milliseconds.
maxEventLoopDelay: 42
Synchronous work causes the event loop to freeze, when this happens an interval timer (which is our sampler) will be delayed by the amount of time the event loop was stalled for while the thread processed synchronous work. We can measure this with timestamp comparison. This option sets a threshold for the maximum amount of stalling between intervals we'll accept before our service begins responding with 503 codes to requests. Defaults to 42 milliseconds.
When set to 0 this threshold will be disabled.
maxHeapUsedBytes: 0
Disabled by default (set to 0), this defines maximum V8 (Node's JavaScript engine) used heap size.
If the Used Heap size exceeds the threshold the server will begin return 503 error codes until it crosses back under the threshold.
See https://www.dynatrace.com/blog/understanding-garbage-collection-and-hunting-memory-leaks-in-node-js for more info on Used Heap from a V8 context.
maxRssBytes: 0
Disabled by default (set to 0) maximum process Resident Set Size. If the RSS exceeds the threshold the server will begin return 503 error codes until it crosses back under the threshold.
errorPropagationMode: false
This is relevant to middleware integration only
By default, overload-protection will handle and end the response,
without calling any subsequent configured middleware. The point here
is to avoid any further processing for an already (by definition)
over loaded process.
However, it could be argued, from a puritanical perspective, that middleware should defer to the framework and that any HTTP code of 500 or above should be generated by propagating an error through the framework.
This option prevents overload-protection from manually ended the response and
instead generates an Error object (with additional properties as per http-errors as used by Express and Koa)
and propagates it through the framework (either by throwing it in Koa, or passing through the next callback).
logging: false
The logging option can be set to a string or a function.
If logging is set to a string, the string should indicate the desired log
level for notifying that a 503 response was given. When logging is a string
a request bound Log4j-style logger is assumed. This means the req object (or the ctx object in the case of Koa)
should have a log object which contains methods corresponding to log levels. So if logging
was set to warn (logging: 'warn') then req.log.warn is expected to be present
and be a function. A number of logging libraries follow this pattern, such as
bunyan-express and all of the pino
middleware loggers (express-pino-logger, koa-pino-logger,
restify-pino-logger, pino-http).
If the application isn't using a request bound Log4j-style logger, the logging
option can be set to a function which receives a log message. This function is
then responsible for writing the log. We could also simply set it to one of
the console methods, e.g. logging: console.warn.
This is primarily for usage when errorPropagationMode is false. If errorPropagationMode
is set to true, we may want to instead log once the error has propagated to a handler.
logStatsOnReq: false
Set logStatsOnReq to true log the profiled stats on every request. In order to use this option, the logging option must not be false. Bear in mind that using this option will
add extra pressure on the event loop in itself, so use with caution.
instance.overload
The returned instance (which in many cases is passed as middleware to app.use),
has an overload property. This begins as false. If any of the thresholds have
been passed this will be set to true. Once all metrics are below their thresholds
this would become false again.
This allows for any heavy load detection required outside of a framework.
instance.eventLoopOverload
The returned instance (which in many cases is passed as middleware to app.use),
has an eventLoopOverload property. This begins as false. If the maxEventLoopDelay
threshold is passed this will be set to true. Once it's below the configured threshold
this would become false again.
This allows for any event loop delay detection necessary outside of a framework.
instance.heapUsedOverload
The returned instance (which in many cases is passed as middleware to app.use),
has a heapUsedOverload property. This begins as false. If the maxHeapUsedBytes
threshold is passed this will be set to true. Once it's below the configured threshold
this would become false again.
This allows for any heap used threshold detection necessary outside of a framework.
instance.rssOverload
The returned instance (which in many cases is passed as middleware to app.use),
has a rssOverload property. This begins as false. If the maxRssBytes
threshold is passed this will be set to true. Once it's below the configured threshold
this would become false again.
This allows for any heap used threshold detection necessary outside of a framework.
instance.eventLoopDelay
The delay in milliseconds (with additional decimal precision) since the last sample.
If maxEventLoopDelay is 0, the event loop is not measured, so eventLoopDelay will always
be 0 in that case.
instance.maxEventLoopDelay
Corresponds to the opts.maxEventLoopDelay option.
instance.maxHeapUsedBytes
Corresponds to the opts.maxHeapUsedBytes option.
instance.maxRssBytes
Corresponds to the opts.maxRssBytes option.
Dependencies
- loopbench: Benchmark your event loop
Dev Dependencies
- autocannon: Fast HTTP benchmarking tool written in Node.js
- express: Fast, unopinionated, minimalist web framework
- koa: Koa web app framework
- koa-router: Router middleware for koa. Provides RESTful resource routing.
- pre-commit: Automatically install pre-commit hooks for your npm modules.
- restify: REST framework
- standard: JavaScript Standard Style
- tap: A Test-Anything-Protocol library
License
MIT
Acknowledgements
Kindly sponsored by nearForm