ReGrid
ReGrid is a method of storing large files inside a RethinkDB database.
Using a SQL database? Try SQLGrid
Features
- Reliable - Files are replicated across the cluster, benefiting from RethinkDB's automatic failover.
- Scalable - Easily store large files in RethinkDB, distributed across the cluster.
- Consistent - Sha256 hashes are calculated when the file is written, and verified when read back out.
- Realtime - Watch the file system for changes and be notified immediately.
- Fast - Supports in-memory caching so your server doesn't fall over when something goes viral.
- Current - Uses RethinkDB's changefeeds to evict cached items. ReGrid doesn't serve stale files.
View the Changelog
The ReGrid spec is an open specification free for anyone to implement and use.
The spec is very out of date
Need help?
Join Slack here, then meet us on the #regrid channel.
Special thanks to Arthur Andrew Medical for sponsoring this project.
Arthur Andrew Medical manufactures products with ingredients that have extensive clinical research for safety and efficacy. We specialize in Enzymes, Probiotics and Antioxidants.
Installation
Supports node v8.0+
npm install --save rethinkdb-regrid
TL;DR
var ReGrid = require('rethinkdb-regrid')
var bucket = ReGrid({db: 'example'})
// initBucket creates tables and indexes if they don't exist, returns a promise.
bucket.initBucket().then(function () {
// We are now ready to read and write files
})
API Documentation
ReGrid([connectionOptions, options])
Parameters
key | default | type | description |
---|---|---|---|
connectionOptions | {} | Object | connectionOptions is passed directly to rethinkdbdash |
options | {} | Object | Optional parameters listed below |
Options
key | default | type | description |
---|---|---|---|
bucketName | fs |
String | The name of the bucket. Table names are prefixed by this. |
chunkSizeBytes | 1024 * 255 |
Number | The default chunk size, in bytes. |
concurrency | 10 |
Number | When writing/reading a file, the number of concurrent queries in flight for a given stream. |
cacheSize | 400 |
Number | The maximum number of objects to keep in memory. Setting to 0 will disable caching. |
returns
Bucket instance
Description
Creates a new ReGrid bucket instance.
Example
var ReGrid = require('rethinkdb-regrid')
var bucket = ReGrid({db: 'mydatabase'}, {bucketName: 'mybucket'})
initBucket()
Parameters
none
returns
Promise
Description
Verifies required tables and indexes exist and will create them if missing.
Example
bucket.initBucket().then(function () {
// bucket is ready for use.....
})
writeFile(options)
Options
key | default | type | description |
---|---|---|---|
filename | required | String | The name of the file. |
buffer | required | Buffer | A buffer of file contents. |
chunkSizeBytes | The chunkSizeBytes setting for the bucket. |
Number | Size of each chunk in bytes. |
metadata | {} | Object | Metadata object allowing you to store custom keys on files. |
returns
Promise
Description
Returns a promise that resolves to the newly written file.
Example
let fileBuffer = fs.readFileSync('./myVid.mp4')
let newFile = await bucket.writeFile({filename: '/videos/myVid.mp4', buffer: fileBuffer})
createWriteStream(options)
Options
key | default | type | description |
---|---|---|---|
filename | required | String | The name of the file. |
chunkSizeBytes | The chunkSizeBytes setting for the bucket. |
Number | Size of each chunk in bytes. |
metadata | {} | Object | Metadata object allowing you to store custom keys on files. |
returns
WriteStream
Description
Returns a write stream for storing a file in ReGrid.
Example
var writeStream = bucket.createWriteStream({
filename: '/videos/myVid.mp4',
chunkSizeBytes: 1024 * 255,
metadata: {topic: 'cats'}
})
writeStream.on('finish', function () {
// File is now stored in ReGrid
})
fs.createReadStream('./myVid.mp4').pipe(writeStream)
getFile(options)
Options
key | default | type | description |
---|---|---|---|
id | Null | String | The id of the file to retrieve. |
filename | Null | String | Ignored if id != null . The filename of the file to retrieve |
revision | -1 |
Number | Ignored if id != null . The revision of the file to retrieve. If multiple files are uploaded under the same filename they are considered revisions. This may be a positive or negative number. (see chart below) |
How revision numbers work
If there are five versions of a file, the below chart would be the revision numbers
Number | Description |
---|---|
0 or -5 |
The original file |
1 or -4 |
The first revision |
2 or -3 |
The second revision |
3 or -2 |
The second most recent revision |
4 or -1 |
The most recent revision |
Description
Returns a promise that resolves to the files information.
Example
let file1 = bucket.getFile({id: 'ca608825-15c0-44b5-9bef-3ccabf061bab'})
let file2 = bucket.getFile({filename: 'catVideo.mp4', revision: 2})
readFile(options)
Options
key | default | type | description |
---|---|---|---|
id | Null | String | The id of the file to retrieve. |
filename | Null | String | Ignored if id != null . The filename of the file to retrieve |
revision | -1 |
Number | Ignored if id != null . The revision of the file to retrieve. If multiple files are uploaded under the same filename they are considered revisions. This may be a positive or negative number. (see chart below) |
seekStart | Null | Number | The start of the byte range. |
seekEnd | Null | Number | The end of the byte range. If omitted the stream will continue to the end of file. |
How revision numbers work
If there are five versions of a file, the below chart would be the revision numbers
Number | Description |
---|---|
0 or -5 |
The original file |
1 or -4 |
The first revision |
2 or -3 |
The second revision |
3 or -2 |
The second most recent revision |
4 or -1 |
The most recent revision |
Description
Returns a promise that resolves to the files information and contents.
Example
let file1 = bucket.readFile({id: 'ca608825-15c0-44b5-9bef-3ccabf061bab'})
let file2 = bucket.readFile({filename: 'catVideo.mp4', revision: 2})
createReadStream(options)
Options
key | default | type | description |
---|---|---|---|
id | required | String | The id of the file to retrieve |
seekStart | Null | Number | The start of the byte range. |
seekEnd | Null | Number | The end of the byte range. If omitted the stream will continue to the end of file. |
returns
ReadStream
Description
Returns a read stream for reading a file from ReGrid.
Example
var readStream = bucket.createReadStream({id: 'ca608825-15c0-44b5-9bef-3ccabf061bab'})
readStream.pipe(fs.createWriteStream('./mySavedVideo.mp4'))
listRegex(pattern[, options])
Parameters
key | default | type | description |
---|---|---|---|
pattern | required | String | A regular expression matched against filename |
options | {} | Object | Optional parameters listed below |
Options
key | default | type | description |
---|---|---|---|
sort | 'DESC' | String | Sort results by filename . Valid values are ASC and DESC . |
skip | undefined | Number | Skip results, useful for pagination. |
limit | undefined | Number | Limit results. |
showAll | false | Boolean | Show all revisions of matched files. Normally duplicate filenames are hidden and only the latest revision of each filename is returned. |
returns
ReadStream in objectMode
. The stream emits objects, not buffers.
Description
Returns a read stream for finding files via regular expression. The stream emits objects as they are found, and can be a long running operation.
You may call toArray()
on the returned stream to coerce it to an array. This may fail if the result set is very large. You may optionally call with limit to prevent this.
Example
var videoStream = bucket.listRegex('^/videos/', {limit: 100})
videoStream.on('data', function (video) {
console.log(video)
})
// OR
bucket.listRegex('^/videos/', {limit: 100}).toArray().then(function (videos) {
// list the first 100 videos in the `/videos` directory
console.log(videos)
})
listFilename(filename[, options])
Parameters
key | default | type | description |
---|---|---|---|
filename | required | String | The name of the file. |
options | {} | Object | Optional parameters listed below |
Options
key | default | type | description |
---|---|---|---|
sort | undefined | String | Sort results by finishedAt (The date the file was uploaded). Valid values are ASC and DESC . |
skip | undefined | Number | Skip results, useful for pagination. |
limit | undefined | Number | Limit results. |
returns
ReadStream in objectMode
. The stream emits objects, not buffers.
Description
Returns a read stream for finding files by filename. The stream emits objects as they are found, and can be a long running operation. However, this method uses an index and is very efficient.
You may call toArray()
on the returned stream to coerce it to an array. This may fail if the result set is very large. You may optionally call with limit to prevent this.
Example
var revisionStream = bucket.listFilename('/videos/editedVideo.mp4')
revisionStream.on('data', function (revision) {
console.log(revision)
})
// OR
bucket.listFilename('/videos/editedVideo.mp4').toArray().then(function (revisions) {
// list all revisions of '/videos/editedVideo.mp4'
console.log(revisions)
})
listMetadata(metadata[, options])
Parameters
key | default | type | description |
---|---|---|---|
metadata | required | Object | An object to match against the metadata field of ReGrid files. |
options | {} | Object | Optional parameters listed below |
Options
key | default | type | description |
---|---|---|---|
skip | undefined | Number | Skip results, useful for pagination. |
limit | undefined | Number | Limit results. |
returns
ReadStream in objectMode
. The stream emits objects, not buffers.
Description
Returns a read stream for finding files by metadata. The stream emits objects as they are found, and can be a long running operation.
You may call toArray()
on the returned stream to coerce it to an array. This may fail if the result set is very large. You may optionally call with limit to prevent this.
Example
var catStream = bucket.listMetadata({topic: 'cats'})
catStream.on('data', function (catVideo) {
console.log(catVideo)
})
// OR
bucket.listMetadata({topic: 'cats'}).toArray().then(function (catVideos) {
// list all videos under the 'cat' topic.
console.log(catVideos)
})
watchRegex(pattern)
Parameters
key | default | type | description |
---|---|---|---|
pattern | required | String | A regular expression matched against filename. |
returns
A RethinkDB changeFeed cursor
Description
Allows you to be notified of changes to a file if the filename
property matches your regular expression.
Example
bucket.watchRegex('^/videos').then(function (cursor) {
cursor.each((err, video) => {
console.log(video)
})
})
watchFilename(filename)
Parameters
key | default | type | description |
---|---|---|---|
filename | required | String | The filename to watch for changes. |
returns
A RethinkDB changeFeed cursor
Description
Allows you to be notified of changes to a file if the filename
property ===
your string.
Example
bucket.watchFilename('/videos/myVid.mp4').then(function (cursor) {
cursor.each((err, changes) => {
console.log(changes)
})
})
watchMetadata(metadata)
Parameters
key | default | type | description |
---|---|---|---|
metadata | required | Object | The metadata to watch for changes. |
returns
A RethinkDB changeFeed cursor
Description
Allows you to be notified of changes to a file if the metadata
property matches your object.
Example
bucket.watchMetadata({topic: 'cats'}).then(function (cursor) {
cursor.each((err, catVideoChanges) => {
console.log(catVideoChanges)
})
})
deleteId(fileId)
Parameters
key | default | type | description |
---|---|---|---|
fileId | required | String | The id of the file to delete |
hard | false | Boolean | Permanently delete a file instead of just flagging |
returns
Returns a promise that resolves to a boolean, depending on whether the operation was successful.
Description
Marks a file as deleted in ReGrid
Example
bucket.deleteId('ca608825-15c0-44b5-9bef-3ccabf061bab', {hard: true})
renameId(fileId, filename)
Parameters
key | default | type | description |
---|---|---|---|
fileId | required | String | The id of the file to rename. |
filename | required | String | The new filename for the selected file. |
returns
Returns a promise that resolves to a boolean, depending on whether the operation was successful.
Description
Renames a file in ReGrid
Example
bucket.renameId('ca608825-15c0-44b5-9bef-3ccabf061bab', 'newName.mp4')
getMetadata(fileId)
Parameters
key | default | type | description |
---|---|---|---|
fileId | required | String | The id of the file to read. |
returns
Returns a promise that resolves to the file metadata.
Description
Retrieves the metadata for a given fileId.
Example
bucket.getMetadata('ca608825-15c0-44b5-9bef-3ccabf061bab')
replaceMetadata(fileId, metadata)
Parameters
key | default | type | description |
---|---|---|---|
fileId | required | String | The id of the file to rename. |
metadata | required | Object | The new metadata for the selected file. |
returns
Returns a promise that resolves to a boolean, depending on whether the operation was successful.
Description
Replaces a file's metadata in ReGrid
Example
bucket.replaceMetadata('ca608825-15c0-44b5-9bef-3ccabf061bab', {topic: 'other cat stuff'})