rotating-file-stream

Description

Creates a stream.Writable to a file which is rotated. Rotation behaviour can be deeply customized; optionally, classical UNIX logrotate behaviour can be used.

Usage

const rfs = require("rotating-file-stream");
const stream = rfs.createStream("file.log", {
  size: "10M", // rotate every 10 MegaBytes written
  interval: "1d", // rotate daily
  compress: "gzip" // compress rotated files
});

Installation

With npm:

$ npm install --save rotating-file-stream

Upgrading from v2 to v3
Upgrading from v1 to v2
API
Rotation logic
Under the hood
Compatibility
TypeScript
Licence
Bugs
ChangeLog
Donating

Upgrading from v2 to v3

In v3 the package was completely refactored using async / await.

TypeScript types for events and the external event were added.

Breaking change: by default the .gz extension is added to the rotated compressed files.

Breaking change: the way the external compression command is executed was slightly changed; possible breaking change.

To maintain back compatibility upgrading from v2 to v3, just follow this rules:

using a file name generator or not using options.compress: nothing to do
using a file name and using options.compress: use options.omitExtension or check how rotated files are treated.

Upgrading from v1 to v2

There are two main changes in package interface.

In v1 the default export of the packege was directly the RotatingFileStream constructor and the caller have to use it; while in v2 there is no default export and the caller should use the createStream exported function and should not directly use RotatingFileStream class. This is quite easy to discover: if this change is not applied, nothing than a runtime error can happen.

The other important change is the removal of option rotationTime and the introduction of intervalBoundary. In v1 the time argument passed to the filename generator function, by default, is the time when rotaion job started, while if options.interval option is used, it is the lower boundary of the time interval within rotaion job started. Later I was asked to add the possibility to restore the default value for this argument so I introduced options.rotationTime option with this purpose. At the end the result was something a bit confusing, something I never liked. In v2 the time argument passed to the filename generator function is always the time when rotaion job started, unless options.intervalBoundary option is used. In a few words, to maintain back compatibility upgrading from v1 to v2, just follow this rules:

using options.rotation: nothing to do
not using options.rotation:
- not using options.interval: nothing to do
- using options.interval:
  - using options.rotationTime: to remove it
  - not using options.rotationTime: then use options.intervalBoundary.

API

const rfs = require("rotating-file-stream");

rfs.createStream(filename[, options])

filename <string> | <Function> The name of the file or the function to generate it, called file name generator. See below for details.
options <Object> Rotation options, See below for details.
Returns: <RotatingFileStream> The rotating file stream!

This interface is inspired to fs.createWriteStream one. The file is rotated following options rules.

filename

The most complex problem about file name is: "how to call the rotated file name?"

The answer to this question may vary in many forms depending on application requirements and/or specifications. If there are no requirements, a string can be used and default rotated file name generator will be used; otherwise a Function which returns the rotated file name can be used.

Note: if part of returned destination path does not exists, the rotation job will try to create it.

filename(time[, index])

time <Date>
- By default: the time when rotation job started;
- if both options.interval and intervalBoundary options are enabled: the start time of rotation period.
If null, the not-rotated file name must be returned.
index <number> The progressive index of rotation by size in the same rotation period.

An example of a complex rotated file name generator function could be:

const pad = num => (num > 9 ? "" : "0") + num;
const generator = (time, index) => {
  if (!time) return "file.log";

  var month = time.getFullYear() + "" + pad(time.getMonth() + 1);
  var day = pad(time.getDate());
  var hour = pad(time.getHours());
  var minute = pad(time.getMinutes());

  return `${month}/${month}${day}-${hour}${minute}-${index}-file.log`;
};

const rfs = require("rotating-file-stream");
const stream = rfs.createStream(generator, {
  size: "10M",
  interval: "30m"
});

Note: if both of options.interval options.size are used, returned rotated file name must be function of both arguments time and index.

filename(index)

index <number> The progressive index of rotation. If null, the not-rotated file name must be returned.

If classical logrotate behaviour is enabled (by options.rotate), rotated file name is only a function of index.

Class: RotatingFileStream

Extends stream.Writable. It should not be directly used. Exported only to be used with instanceof operator and similar.

Event: 'external'

stdout <string> The standard output of the external compression command.
stderr <string> The standard error of the external compression command.

The external event is emitted once an external compression command completes its execution to give access to the command output streams.

Event: 'history'

The history event is emitted once the history check job is completed.

Event: 'open'

filename <string> Is constant unless options.immutable is true.

The open event is emitted once the not-rotated file is opened.

Event: 'removed'

filename <string> The name of the removed file.
number <boolean>
- true if the file was removed due to options.maxFiles
- false if the file was removed due to options.maxSize

The removed event is emitted once a rotated file is removed due to options.maxFiles or options.maxSize.

Event: 'rotation'

The rotation event is emitted once the rotation job is started.

Event: 'rotated'

filename <string> The rotated file name produced.

The rotated event is emitted once the rotation job is completed.

Event: 'warning'

error <Error> The non blocking error.

The warning event is emitted once a non blocking error happens.

options

compress: <boolean> | <string> | <Function> Specifies compression method of rotated files. Default: null.
encoding: <string> Specifies the default encoding. Default: 'utf8'.
history: <string> Specifies the history filename. Default: null.
immutable: <boolean> Never mutate file names. Default: null.
initialRotation: <boolean> Initial rotation based on not-rotated file timestamp. Default: null.
interval: <string> Specifies the time interval to rotate the file. Default: null.
intervalBoundary: <boolean> Makes rotated file name with lower boundary of rotation period. Default: null.
maxFiles: <number> Specifies the maximum number of rotated files to keep. Default: null.
maxSize: <string> Specifies the maximum size of rotated files to keep. Default: null.
mode: <number> Proxied to fs.createWriteStream. Default: 0o666.
omitExtension: <boolean> Omits the .gz extension from compressed rotated files. Default: null.
path: <string> Specifies the base path for files. Default: null.
rotate: <number> Enables the classical UNIX logrotate behaviour. Default: null.
size: <string> Specifies the file size to rotate the file. Default: null.
teeToStdout: <boolean> Writes file content to stdout as well. Default: null.

encoding

Specifies the default encoding that is used when no encoding is specified as an argument to stream.write().

mode

Proxied to fs.createWriteStream.

path

If present, it is prepended to generated file names as well as for history file.

teeToStdout

If true, it makes the file content to be written to stdout as well. Useful for debugging puposes.

size

Accepts a positive integer followed by one of these possible letters:

B: Bites
K: KiloBites
M: MegaBytes
G: GigaBytes

  size: '300B', // rotates the file when size exceeds 300 Bytes
                // useful for tests

  size: '300K', // rotates the file when size exceeds 300 KiloBytes

  size: '100M', // rotates the file when size exceeds 100 MegaBytes

  size: '1G', // rotates the file when size exceeds a GigaByte

interval

Accepts a positive integer followed by one of these possible letters:

s: seconds. Accepts integer divider of 60.
m: minutes. Accepts integer divider of 60.
h: hours. Accepts integer divider of 24.
d: days. Accepts integer.
M: months. Accepts integer. EXPERIMENTAL

  interval: '5s', // rotates at seconds 0, 5, 10, 15 and so on
                  // useful for tests

  interval: '5m', // rotates at minutes 0, 5, 10, 15 and so on

  interval: '2h', // rotates at midnight, 02:00, 04:00 and so on

  interval: '1d', // rotates at every midnight

  interval: '1M', // rotates at every midnight between two distinct months

intervalBoundary

If set to true, the argument time of filename generator is no longer the time when rotation job started, but the lower boundary of rotation interval.

Note: this option has effect only if options.interval is used.

compress

For historical reasons external compression can be used, but the best choice is to use the value "gzip".

To enable external compression, a function can be used or simply the boolean true value to use default external compression. The function should accept source and dest file names and must return the shell command to be executed to compress the file. The two following code snippets have exactly the same effect:

var rfs = require("rotating-file-stream");
var stream = rfs.createStream("file.log", {
  size: "10M",
  compress: true
});

var rfs = require("rotating-file-stream");
var stream = rfs.createStream("file.log", {
  size: "10M",
  compress: (source, dest) => "cat " + source + " | gzip -c9 > " + dest
});

Note: this option is ignored if options.immutable is used.

Note: the shell command to compress the rotated file should not remove the source file, it will be removed by the package if rotation job complete with success.

omitExtension

From v3 the package adds by default the .gz extension to the rotated compressed files. Simultaneously this option was added: set this option to true to not add the extension, i.e. to keep backward compatibility.

initialRotation

When program stops in a rotation period then restarts in a new rotation period, logs of different rotation period will go in the next rotated file; in a few words: a rotation job is lost. If this option is set to true an initial check is performed against the not-rotated file timestamp and, if it falls in a previous rotation period, an initial rotation job is done as well.

Note: this option has effect only if both options.interval and options.intervalBoundary are used.

Note: this option is ignored if options.rotate is used.

rotate

If specified, classical UNIX logrotate behaviour is enabled and the value of this option has same effect in logrotate.conf file.

Note: if this optoin is used following ones take no effect: options.history, options.immutable, options.initialRotation, options.intervalBoundary, options.maxFiles, options.maxSize.

immutable

If set to true, names of generated files never changes. New files are immediately generated with their rotated name. In other words the rotated file name generator is never called with a null time argument unless to determinate the history file name; this can happen if options.history is not used while options.maxFiles or options.maxSize are used. The filename argument passed to 'open' event evaluates now as the newly created file name.

Useful to send logs to logstash through filebeat.

Note: if this option is used, options.compress is ignored.

Note: this option is ignored if options.interval is not used.

history

Due to the complexity that rotated file names can have because of the filename generator function, if number or size of rotated files should not exceed a given limit, the package needs a file where to store this information. This option specifies the name history file. This option takes effect only if at least one of options.maxFiles or options.maxSize is used. If null, the not rotated filename with the '.txt' suffix is used.

maxFiles

If specified, it's value is the maximum number of rotated files to be kept.

maxSize

If specified, it's value must respect same syntax of option.size and is the maximum size of rotated files to be kept.

Rotation logic

Regardless of when and why rotation happens, the content of a single stream.write will never be split among two files.

by size

Once the not-rotated file is opened first time, its size is checked and if it is greater or equal to size limit, a first rotation happens. After each stream.write, the same check is performed.

by interval

The package sets a Timeout to start a rotation job at the right moment.

Under the hood

Logs should be handled so carefully, so this package tries to never overwrite files.

At stream creation, if the not-rotated log file already exists and its size exceeds the rotation size, an initial rotation attempt is done.

At each rotation attempt a check is done to verify that destination rotated file does not exists yet; if this is not the case a new destination rotated file name is generated and the same check is performed before going on. This is repeated until a not existing destination file name is found or the package is exhausted. For this reason the rotated file name generator function could be called several times for each rotation job.

If requested through options.maxFiles or options.maxSize, at the end of a rotation job, a check is performed to ensure that given limits are respected. This means that while rotation job is running both the limits could be not respected. The same can happen till the end of first rotation job if options.maxFiles or options.maxSize are changed between two runs. The first check performed is the one against options.maxFiles, in case some files are removed, then the check against options.maxSize is performed, finally other files can be removed. When options.maxFiles or options.maxSize are enabled for first time, an history file can be created with one rotated filename (as returned by filename generator function) at each line.

Once an error event is emitted, nothing more can be done: the stream is closed as well.

Compatibility

Requires Node.js v14.

The package is tested under all Node.js versions currently supported accordingly to Node.js Release.

To work with the package under Windows, be sure to configure bash.exe as your script-shell.

> npm config set script-shell bash.exe

TypeScript

TypeScript types are distributed with the package itself.

License

MIT License

Bugs

Do not hesitate to report any bug or inconsistency @github.

ChangeLog

Donating

If you find useful this package, please consider the opportunity to donate some satoshis to this bitcoin address: 12p1p5q7sK75tPyuesZmssiMYr4TKzpSCN

iccicci/rotating-file-stream

iccicci

Reviews

Repository Details