thunderclap
Thunderclap is a key-value, indexed JSON, and graph database plus function oriented server designed specifically for Cloudflare. It runs on top of the Cloudflare KV store. Its query capability is supported by JOQULAR (JavaScript Object Query Language), which is similar to, but more extensive than, the query language associated with MongoDB. In addition to having more predicates than MongoDB, JOQULAR extends pattern matching to object properties, e.g.
// match all objects with properties starting with the letter "a" containing the value 1
db.query({Object:{[/a.*/]:{$eq: 1}}}) Thunderclap uses a JavaScript client client to support:
-
full text indexing and search in addition to regular property indexing
A URL fetch (CURL) capability is also supported.
Like MongoDB, Thunderclap is open-sourced under the Server Side Public License. This means licencees are free to use and modify the code for internal applications or public applications that are not primarily a means of providing Thunderclap as a hosted service. In order to provide Thunderclap as a hosted service you must either secure a commercial license from AnyWhichWay or make all your source code available, including the source of non-derivative works that support the monitoring and operation of Thunderclap as a service.
Important Notes
Thunderclap is currently in ALPHA because:
-
Workers KV from Cloudflare recently came out of beta and is missing a few key features that are "patched" by Thunderclap.
-
Security measures are incomplete and have not yet been vetted by a third party.
-
Although there are many unit tests, application level functional testing has been limited.
-
The source code could do with a lot more comments.
-
Project structure does not currently have a clean separation between what people might want to change for their own use vs submit as a pull request. In general, changes to file in the
srcdirectory are candidates for pull requests and with the exception of this README those outside are not. -
It has not been performance tuned.
-
It is highly likely you will need to re-create your NAMESPACES with every new ALPHA release.
-
APIs are not yet stable.
-
It could do with contributors!
Installation and Deployment
Clone the repository https://www.github.com/anywhichway/thunderclap.
Run npm install.
Production
NOTE: While the software is in ALPHA state, you should probably not deploy to a production Cloudflare server that is not behind Cloudflare's paid Access management interface.
When Thunderclap is running in production, it will be available at thunderclap.<your-domain>. When it is running in
development mode, it will be available at <dev-host-prefix>-thunderclap.<your-domain>. You can choose
the dev-host-prefix.
You can deploy and use Thunderclap immediately after creating and populating a thunderclap.json configuration file,
creating a KV namespace, and establishing a CNAME alias thunderclap in the Cloudflare DNS control panel. This can
just point to your root, you do not need a distinct IP address, Cloudflare's smart routers will send requests to the
Thunderclap Cloudflare Worker.
Create a Cloudflare Workers KV namespace using the Cloudflare Workers control panel. By convention, the following name form is recommended so that the name parallels the name of thw worker script generated by Thunderclap.
thunderclap-<primaryHostName>-<com|org|...>
e.g. thunderclap-mydomain-com is the KV namespace and script name associated with thunderclap.mydomain.com
You will need to populate a file thunderclap.json with many of your Cloudflare ids or keys. Copy the file
thunderclap.template.json to thunderclap.json and replace the placeholder values. This will contain secret keys,
so you may want to move it out of your project directory to avoid having it checked-in. The thunderclap script in
webpack.config.js assumes the file is up one level in the directory tree. If you do leave it in the project
directory .gitignore is configured to not check it in. But, you will need to edit webpack.config.js so that it can
find thunderclap.json.
The .gitignore is also set to ignore dbo.js, which contains the default dbo password and keys.js which contains
a function definition for iterating over keys on the server that requires special credentials. None of this data is
built into the browser software.
You will need to place the file db.json at the root of your web server's public directory and thunderclap.js
in your normal JavaScript resources directory. For convenience, db.json and thunderclap.js are located in the docs
subdirectory of the repository so you can host them using GitHub Pages if you wish.
You can use Thunderclap without making any modifications by setting the mode in thunderclap.json to production
and running npm run thunderclap. This will deploy the Thunderclap worker and a route. Don't forget to deploy db.json
and thunderclap.js also.
See the files in docs and docs/test for examples of using Thunderclap.
Data Manipulation top
Data in Thunderclap can be manipulated using a JavaScript client or CURL.
JavaScript Client top
<script src="thunderclap.js"></script>
<script>
const endpoint = "https://thunderclap.mydomain.com/db.json",
username = "<get from somewhere>",
password = "<get from somewhere>",
// there is a default user `dbo`, with a default password `dbo`.
db = new Thunderclap({endpoint,user:{username,password}});
</script>boolean async addRoles(string userName,Array [string role,...]=[]) - Assigns roles to the named user. Returns true on success. By default,
only a user with the role dbo can call this function.
undefined async clear(string prefix="") - Deletes items whose keys start with prefix. By default it can only be
called by a user with the dbo role. See the section on Access Control to change this.
string async changePassword(string userName,string password,string oldPassword) - Changes the password from
oldPassword to password for the user with name userName and old password oldPassword. If password is not
provided, a random 10 character password is generated and returned. If the currently authenticated user has the role
dbo and is not the user for whom the password was being changed, oldPassword can be omitted. If a password
reset process has been inititiated with resetPassword, then oldPassword should be the temporary password or mobile
code.
User async createUser(string userName,string password,object extras={},reAuth) - Creates a user. The password is
stored on the server as an SHA-256 hash and salt. createUser can be called even if Thunderclap is started without
a username and password. If this is done and createUser succeeds, the Thunderclap instance is bound to the new user
for immediate authenticated use. The extras argument can be any additional data to be stored in the user object except roles.
If reAuth is truthy and the Thunderclap instance was already authenticated, the Thunderclap instance will also be re-bound
to the new user. You can implement access control and account creation logic on the server to prevent the creation of
un-authorized accounts. See the section on Access Control.
boolean async deleteUser(string userName) - Deletes a user. Returns true if user was deleted or did not exist. By default,
only a user with the role dbo can call this function.
Array async entries(string prefix="",{number batchSize,string cursor}) - Returns an array or arrays for keys,
values and optionally expirations of data with keys that start with prefix. By default,
only a user with the role dbo can call this function. Expirations are in Unix epoch milliseconds,
e.g. entries("Person@") might return:
[["Person@jxmc9cc1kswqak4ga",{"name":"joe"},1562147669820],["Person@jxmcnqkx9irjhrz4p",{"name":"joe"}]]Entries can be used in a loop just like keys below.
Array entry(string key) - Returns the entry for a key as a two or three element array or undefined. By default,
only a user with the role dbo can call this function. For example, entry("Person@jxmc9cc1kswqak4ga")
might return:
["Person@jxmc9cc1kswqak4ga",{"name":"joe"},1562147669820]Edge async get(string|Array path) - Returns an Edge in a graph data store. The path can be an Array or a dot delimited string, e.g.
["people","joe"] is the same as "people.joe".
User async getUser(string userName) - Returns the User with the userName or undefined. By default can only be executed
by a user with the role dbo or the named user itself.
any async getItem(string key) - Gets the value at key. Returns undefined if no value exists.
boolean async hasKey(string key) - Returns true if key exists.
Array async keys(prefix="",{number batchSize=1000,string cursor,boolean expanded}) - Returns an Array of the next batchSize
keys in database than match the prefix every time it is called. By convention, the last value in the array is the cursor.
By default it can only be called by a user with the dbo role. A loop can be used to process all keys:
let cursor;
do {
keys = await mythunder.keys("",{cursor});
cursor = keys.pop();
keys.forEach((key) => dosomething(key));
} while(cursor && keys.length>0)any async putItem(object instance,options={}) - Adds a unique id on property "#" of instance, if one does not exist,
indexes the object and stores it with setItem using the id as the key. In most cases, the unique id will be of the form
<className>@xxxxxxxxxxxxx. The options can be one of: {expiration: secondsSinceEpoch} or {expirationTtl: secondsFromNow}.
boolean async removeItem(string|object keyOrObject) - Removes the keyOrObject. If the argument is an indexed object
or a key that resolves to an indexed object, the index entries are also removed from the database so long as the user has
the appropriate privileges. If the key exists but can't be removed, the function returns false. If the key does not exist
or removal succeeds, the function returns true.
boolean async removeRoles(string userName,Array [string role,...]=[]) - Removes roles from the named user. Returns true on success. By default,
only a user with the role dbo can call this function.
string async resetPassword(string userName,string method="email"||"mobile") - COMING SOON. Initiates a password reset process for
the userName. method defaults to "email". The User object stored in the database must have an email or mobile property. The "mobile"
option requires Neutrino keys in thunderclap.json and "email" requires Mailgun keys. You must build a UI that calls changePassword
with the code sent to the user as the oldPassword argument. Mobile codes are good for 5 minutes. Email codes are good for 30 minutes. The
current password on the user is not changed until changePassword is called. By default resetPassword can only be executed by a dbo or
the current user for for themself.
boolean async sendMail({to:Array [string emailAddress,...],{cc:Array},{bcc:Array},{subject:string},{body:string}) -
Send's email. Requires providing Mailgun keys in thunderclap.json. The from for the e-mail is always the currently authenticated user's e-mail.
This function should be access controlled in secure.js. By default, only users with the role dbo can send mail. Developers might
consider adding a role mailsenders. Returns true if mail was successfuly sent, false if there was no e-mail address for the
autheticated user, and error text with a 500 HTTP status if there was some other cause of failure.
any async setItem(string key,any value,options={}) - Sets the key to value. If the value is an object it is
NOT indexed. Options can one of: {expiration: secondsSinceEpoch} or {expirationTtl: secondsFromNow}.
Array async query(object {<className>:JOQULARPattern},{boolean partial,number limit) - Query the database using JOQULARPattern.
If partial is truthy, then only those properties used in the query will actuall be returned. The limit defaults internally to
1000 items. See JOQULAR below.
boolean async unique(object|string cnameOrIdOrObject,property,value) - Returns true if the value on property is or will be unique for
the provided cnameOrIdOrObject. If a string class name is provided and true is returned, the the value will be unique for the class name
and property when added to the database. If a full object id, e.g. Object@jy34s5bz1fkqseh8j is provided, then either the value will
be unique when the object is added or the object exists and has the unique value. Passing in an actuall object just has its id pulled
for use in the call to the server.
User async updateUser(string userName,properties={}) - COMING SOON. Update the user with the provided properties. The currently
autheticated user must be a dbo or the target user. To delete values, use a value of undefined for a property. The properties
role, password, hash, and salt properties are ignored. By default resetPassword can only be executed by a dbo or
the current user for for themself.
any async value(string|Array path [,any value [,object options={}]) - With no optional arguments, returns the value stored at
the edge found at path in a graph data store. The path can be an Array or a dot delimited string, e.g. ["people","joe"]
is the same as "people.joe". With the optional argument value, sets the value at the edge. The options can be one
of: {expiration: secondsSinceEpoch} or {expirationTtl: secondsFromNow}.
Array async values(string prefix="",{number batchSize,string cursor}) - Returns all the data associated with keys that
start with prefix. By default it can only be called by a user with the dbo role. It can be used in a loop just like
keys above.
CURL Requests top
To be written
Special Storage top
Most JavaScript data stores do not support special values like undefined, Infinity and NaN. Thunderclap
serializes these as special strings, e.g. @Infinity. However, this is transparent to API calls via the JavaScript
client and should only be of concern to those who are customizing or extending Thuderclap.
The Thunderclap client also serializes dates as Date@<timestamp> and restores them to full-fledged dates after
transport.
Built-in Classes top
User top
Thunderclap provides a basic User class accessable via new Thunderclap.User({string userName,object roles={user:true}})
and createUser(string userName,string password,[object extras],[boolean reauth]). Developers are free to add other
properties and values to the constructor argument or extras object. Additional role keys may also be added to the roles
sub-object. The only built-in roles are user and dbo. See Access Control(#access-control) for more detail. To actually
create a user and store it in the database use createUser.
Edge top
The Edge object is used to support graph database options. The graph API is similar to, but no identical to, the
GunDB graph API. It has a number of methods:
Edge async add(string|Array path,any data[,options={}]) - Adds data to a Set at path.
number async delete(string|Array path) - Deletes the sub-graph, if any, at path. Returns the number of nodes deleted.
Edge async get(string|Array path) - Gets the sub-edge at path.
object async put(object data) - Explodes the object into a sub-graph on the current Edge.
Edge async remove(string|Array path,any data) - Removes data from a Set at path.
any async value(string|Array path [,any value [,object options={}]) - Get's or sets the value associated with the Edge.
Coordinates top
For convenience, Thunderclap exposes a Coordinates object with the same properties as the JavaScript browser standard
interface {latitude,longitude,altitude,accuracy,altitudeAccuracy,heading,speed}. Coordinates can be created directly with:
new Thunderclap.Coordinates({Coordinates coords,number timestamp});There is also an asynchronous Thunderclap.Coordinates.create([Coordinates coords]). If the
optional argument is not provided, then the browser navigator.geolocation.getCurrentPosition will be called
to get the values to create the Coordinates. This makes it easy to deploy clients that automatically collect and store
location data.
Position top
For convenience, Thunderclap exposes a Position object with the same properties as the JavaScript browser standard
interface coords and timestamp. Positions can be created
directly with:
new Thunderclap.Position({Coordinates coords,number timestamp});There is also an asynchronous Thunderclap.Position.create([{Coordinates coords,number timestamp}]). If the
optional argument is not provided, then the browser
navigator.geolocation.getCurrentPosition
will be called to get the values to create the Position. This makes it easy to deploy clients that automatically collect and store
location data.
JOQULAR top
Thunderclap supports a subset of JOQULAR. It is simlar to the MongoDB query language but more extensive. You can see many
examples in the unit test file test/index.js.
Here is a basic query that returns all Users of age 21 or greater in zipcode 98101:
const db = new Thunderclap({endpoint,user:{username:"<username>",password:"<password>"}}),
results = await db.query({User:{age:{$gte: 21},address:{zipcode:98101}}});Thuderclap also suppport pattern matching on property names:
db.query({Object:{[/a.*/]:{$eq: 1}}}) // match all Objects with properties starting with the letter "a" containing the value 1You may have noted above, the top level property names in a query should be class names. In MongoDB, these would
be collections. Using different top level keys you can query multiple "collections" at the sam etime. If you use the
top leve wild card key _, all "collections" will be searched.
The supported patterns are described below. All examples assume these two objects exist in the database:
const o1 = {
#:"User@jxxtlym2fxbmg0pno",
userName:"joe",
age:21,
email: "[email protected]",
SSN: "555-55-5555",
registeredIP: "127.0.0.1",
address:{city:"Seattle",zipcode:"98101"},
registered:"Tue, 15 Jan 2019 05:00:00 GMT",
favoritePhrase:"to be or not to be, that is the question"
},
o2 = {
#:"User@jxxviym2fxbmg0pcr",
userName:"mary",
age:20,
address:{city:"Bainbridge Island",zipcode:"98110"},
registered:"Tue, 15 Jan 2019 10:00:00 GMT",
favoritePhrase:"premum non nocere"
};The supported patterns include the below. (If a pattern is not documented, it may not have been tested yet. See the
unit test file docs/test/index.js to confirm.):
Math and String Comparisons top
{$lt: number|string value} - A value in a property is less than the one provided, e.g. {age:{$lt:21}} matches o2.
{$lte: number|string value} - A value in a property is less or equal the one provided, e.g. {age:{$lte:21}} matches o1 and o2.
{$eq: number|string value} - A value in a property is relaxed equal the one provided, e.g. {age:{$eq:21}} and {age:{$eq:"21"}} match o1.
{$eeq: number|string value} - A value in a property is exactly equal the one provided, e.g. {age:{$eeq:21}} matches o1 but and {age:{$eeq:"21"}} does not.
{$neq: number|string value} - A value in a property is relaxed equal the one provided, e.g. {age:{$neq:21}} matches o2.
{$gte: number|string value} - A value in a property is greater than or equal the one provided, e.g. {age:{$gte:20}} matches o1 and o2.
{$gt: number|string value} - A value in a property is greater than the one provided, e.g. {age:{$gt:20}} matches o1 and o2.
String Tests
{$startsWith: string value} - A value in a property starts with the one provided.
{$endsWith: string value} - A value in a property ends with the one provided.
Logical Operators top
{$and: Array} - Ands multiple conditions, e.g. {age:{$and:[{$gt:20},{$lt: 30}]} matches o1 and o2. Typically not required
because this produces the same result, {age:{$gt:20,$lt: 30}}.
{$not: JOQULARExpression} - Negates the contained condition, e.g. {age:{$not:{$gt:20}}} matches o2.
{$or: Array} - Ors multiple conditions, e.g. {age:{$or:[{$eq:20},{$eq: 21}]} matches o1 and o2. Allow repeating
the same predicate for a single property. The nested form is also supported, {age:{$eq:20,$or:{$eq: 21}}}
{$xor: Array} - Exclusive ors multiple conditions.
Date and Time top
The full range of methods available for extracting parts from Date on a native JavaScript Date instance are also available as predicates:
{$date: number dayOfMonth} - {$date: 14} matches o1 in EST.
{$day: number dayOfWeek} - {$day: 2} matches o1 in EST.
{$fullYear: number fourDigitYear} - {$fullYear: 2019} matches o1 in EST.
{$hours: number hours} - {$hours: 5} matches o1 in EST.
{$milliseconds: number ms} - {$milliseconds: 0} matches o1.
{$minutes: number minutes} - {$minutes: 0} matches o1.
{$month: number month} - {$month: 0} matches o1.
{$seconds: number seconds} - {$seconds: 0} matches o1.
{$time: number time} - {$time: 1547528400000} matches o1.
{$UTCDate: number dayofMonth} - {$date: 14} matches o1.
{$UTCDay: number dayOfWeek} - {$day: 2} matches o1.
{$UTCFullYear: number fourDigitYear} - {$fullYear: 2019} matches o1.
{$UTCHours: number hours} - {$hours: 5} matches o1.
{$UTCMilliseconds: number ms} - {$milliseconds: 0} matches o1.
{$UTCMinutes: number minutes} - {$minutes: 0} matches o1.
{$UTCMonth: number month} - {$month: 0} matches o1.
{$UTCSeconds: number seconds} - {$seconds: 0} matches o1.
{$year: number 2digitYear} - ${year: 19} matches o1.
Membership top
{$in: Array|string container} - A value in a property is in the provided container, e.g. {age:{$in:[20,21,22]}} matches o1 and o2.
{$nin: Array|string container} - A value in a property is not in the provided container, e.g. {age:{$nin:[21,22,23]}} matches o2.
{$includes: boolean|number|string included} - A value in a property (Array or string) includes included.
{$intersects: Array|string container} - A value in a property (Array or string) intersects container.
$disjoint -
Ranges top
{$between: Array [number|string bound1,number|string bound2,inclusive]} - A value in a property in between the two provided
limits. The limits can be in any order, e.g. {age:{$between:[19,21]}} or {age:{$between:[21,19]}} matches o2. Optionally,
the limits can be inclusive, e.g. {age:{$between:[19,21,true]}} matches o1 and o2.
{$outside: Array [number|string bound1, number|string bound2]} - A value in a property in outside the two provided limits.
The limits can be in any order, e.g. {age:{$outside:[19,20]}} or {age:{$between:[20,19]}} matches o1.
{$near: Array [number target,number|string absoluteOrPercent]} - A value in a property is near the provided number either
from an absolute or percentage perspective, e.g. {age:{$near:[21,1]}} matches both o1 and o2 as does
{age:{$near:[21,"5%"]}} since 1 is 4.7% of 21.
Regular Expression top
{$matches: RegExp|string pattern} - A value in a property matches the provided regular expression. The regular expression
can be a string that looks like a regular expression or an actual regular expression, e.g. {userName:{$matches:/a.*/}}
or {userName:{$matches:"/a.*/"}}
Special Tests top
{$instanceof: string className} - A value in a property is an instanceof the class denoted by the string argument. The
class must be registered on the server. Currently this includes Object, Array, Data, User, Schema, Position, and Coordinates.
You can add more classes by modifying the file classes.js.
{$isa: string className} - A value in a property is of the class provided by the string argument. Note, this is not
an instanceof test, it does not walk the inheritance tree.
Note that other special tests typically take true as an argument. This is an artifact of JSON format that does not allow
empty properties. Passing anything else will cause them to fail. You may occassionaly want to pass false to match
things that do not satisfy the test.
{$isCreditCard: boolean value} - A value in a property is a valid credit card based on a regular expression and Luhn algorithm.
{$isEmail: boolean value} - A value in a property is a valid e-mail address by format, e.g. {email:{$isEmail: true}}. Note:
e-mail addresses are remarkably hard to validate without actually trying to send and e-mail. This will address
all reasonable cases.
{$isEven: boolean value} - A value in a property is even, e.g. {age:{$isEven: true}} matches o2.
{$isFloat: boolean value} - A value in a property is a float, e.g. {age:{$isFloat: true}} will not match either o1 or o2. Note,
0 and 0.0 are both treated as 0 by JavaScript, so 0 will never satisfy $isFloat.
{$isIPAddress: boolean value} - A value in a property is a dot delimited IP address, e.g. `{registeredIP:{$isIPAddress: true}}
{$isInt: boolean value} - A value in a property is a dot delimited IP address, e.g. `{registeredIP:{$isIPAddress: true}} matches o1.
{$isNaN: boolean value} - A value in a property is a not a number, e.g. `{address:{zipcode:{$isNaN: true}}} matches o1. Note, $isNaN
will fail when there is no value since it is no known whether the target is a number or not.
{$isOdd: boolean value} - A value in a property is odd, e.g. {age:{$isOdd: true}} matches o1.
{$isSSN: boolean value} - A value in a property looks like a Social Security Number, e.g. {SSN:{$isSSN: true}} matches o1. Note,
unlike $isCreditCard no validation is done beyond textual format.
Text Search top
{$echoes: string soundALike} - A value in a property sounds like the provided value, e.g. {userName:{$echoes: "jo"}} matches o1.
{$search: string searchPhrase} - Does a full text trigram based search, e.g. {favoritePhrase:{$search:"question"}} matches o1.
If no second argument is provided, the search is fuzzy at 80%, e.g. {favoritePhrase:{$search:"questin"}} also
matches o1 whereas {favoritePhrase:{$search:["questin",.99]}}, which requires a 99% match does not. The search
phrase can contain multiple space separated words.
Special Predicates
{$_:any value} - Matches any property that has value.
{"$.":[string functionName,...args]} - Calls the functionName on the value in the property, e.g. {name:{$startsWith:"ma"}}
is the same as {name:{"$.":["startsWith","ma"]}}. By default, queries that use this predicate will return a HTTP status code 403
for users that do not have a dbo role.
{"$.<functionName>":[...args]} - Similar to $., except the function name is part of the property. By default, queries that
use this predicate will return a HTTP status code 403 for users that do not have a dbo role.
Access Control top
The Thunderclap security mechanisms support the application of role based read and write access rules for functions, objects, properties, keys, and edges.
If a user is not authorized read access to an object, key value or edge, it will not be returned. If a user is not authorized access to a particular property, the property will be stripped from the object before the object is returned. Additionally, a query for an object using properties to which a user does not have access will automatically drop the properties from the selection process to prevent data leakage through inference.
If a user is not authorized write access to specific properties on an object, update attempts will
fall back to partial updates on just those properties for which write access is allowed. If write access to a
key, entire object, or edge is not allowed, the write will simply fail and return undefined.
At the moment, by default, all keys, objects, and properties are available for read and write unless specifically
controlled in the secure.js file in the root of the Thunderclap repository. A future release will support defaulting
to prevent read and write unless specifically permitted.
If the user is not authorized to execute a function a 403 status will be returned.
The default secure.js file is show below.
(function() {
module.exports = {
"Function@": {
securedTestFunction: { // for testing purposes
execute: [] // no execution allowed
},
addRoles: { // only dbo can add roles to a user
execute: ["dbo"]
},
clear: { // only dbo can clear
execute: ["dbo"]
},
deleteUser: {
execute: ["dbo"]
},
entries: { // only dbo can list entries
execute: ["dbo"]
},
entry: {
execute: ["dbo"]
},
keys: { // only dbo can list keys
execute: ["dbo"]
},
removeRoles: {
execute: ["dbo"]
},
resetPassword: { // only user themself or dbo can start a password reset
execute({argumentsList,user}) {
return user.roles.dbo || argumentsList[0]===user.userName
}
},
sendMail: { // only dbo can send mail
execute: ["dbo"]
},
updateUser: { // only user themself or dbo can update user properties
execute({argumentsList,user}) {
return user.roles.dbo || argumentsList[0]===user.userName
}
},
values: { // only dbo can list values
execute: ["dbo"]
}
},
"User@": { // key to control, use <cname>@ for classes
// read: ["<role>",...], // array or map of roles to allow get, not specifying means all have get
// write: {<role>:true}, // array or map of roles to allow set, not specifying means all have set
// a filter function can also be used
// action with be "get" or "set", not returning anything will result in denial
// not specifying a filter function will allow all get and set, unless controlled above
// a function with the same call signature can also be used as a property value above
filter({action,user,data,request}) {
// very restrictive, don't return a user record unless requested by the dbo or data subject
if(user.roles.dbo || user.userName===data.userName) {
return data;
}
},
keys: { // only applies to objects
roles: {
// only dbo's and data subject can get roles
get({action,user,object,key,request}) { return user.roles.dbo || object.userName===user.userName; },
},
hash: {
// only dbo's can get password hashes
read: ["dbo"],
// only the dbo and data subject can set a hash
set({action,user,object,key,request}) { return user.roles.dbo || object.userName===user.userName; },
},
salt: {
// example of alternate control form, only dbo's can get password salts
read: {
dbo: true
},
// only the dbo and data subject can set a salt
set({action,user,object,key,request}) { return user.roles.dbo || object.userName===user.userName; },
},
name({action,user,data,request}) { return data; } // example, same as no access control
}
/* keys could also be a function
keys({action,user,data,key,request})
*/
},
securedTestReadKey: { // for testing purposes
read: [] // no gets allowed
},
securedTestWriteKey: { // for testing purposes
write: [] // no sets allowed
},
[/\!.*/]: { // prevent direct index access by anyone other than a dbo, changing this may create a data inference leak
read: ["dbo"],
write: ["dbo"]
}
/* Edges are just nested keys or wildcards, e.g.
people: {
_: { // matches any sub-edge
secretPhrase: { // matches secrePhrase edge
read(...) { ... },
write(...) { ... }
}
}
}
*/
}
}).call(this);
Roles can also be established in a tree that is automatically applied at runtime. See the file roles.js.
When Thunderclap is first initialized, a special user User@dbo with the user name dbo the role dbo and the
dbo password defined in thunderclap.json is created. It also has the unique id User@dbo.
You can create additional accounts with the createUser method and change passwords with the changePassword
method.
Inline Analytics & Hooks top
Inline analytics and hooks are facilitated by the use of JOQULAR patterns or edge specficications and tranform or hook
calls in the file when.js. The transforms and hooks can be invoked from the browser, a service worker, or in the cloud.
They are not currently access controlled in the browser or a service worker. In the cloud, transforms are invoked after
it is determined primary key access is allowed but before data property access is assesed and the data is written.
This security is applied to the transformed data. Hooks are called after the data is written. If you need to transform
something and call a hook, but not write to the database either call the hook as the last action in the transform and
return nothing, or use a before trigger.
Below is an example.
(function() {
module.exports = {
client: [
{
when: {testWhenBrowser:{$eq:true}},
transform({data,pattern,user,request}) {
Object.keys(data).forEach((key) => { if(!pattern[key]) delete data[key]; });
return data;
},
call({data,pattern,user,request}) {
}
}
],
worker: [
// not yet implemented
],
cloud: [
{
when: {testWhen:{$eq:true}},
transform({data,pattern,user,request}) {
Object.keys(data).forEach((key) => { if(!pattern[key]) delete data[key]; });
return data;
},
call({data,pattern,user,request,db}) {
}
}
]
}
}).call(this);Triggers top
Triggers can get invoked before and after key value or indexed object properties change or get deleted. The triggers are configured
in the file on.js. Any asynchronous triggers will be awaited. before triggers must return truthy for execution to
continue, i.e. a before on set that returns false will result in the set aborting. before triggers are fired immediately
before security checks. Triggers are not access controlled.
Triggers can be executed in the browser, a service worker, or the cloud.
(function() {
module.exports = {
client: {
},
worker: {
},
cloud: {
"User@": {
read({value,key,user,request}) {
; // called after get
},
write({value,key,oldValue,user,request}) {
// if value!==oldValue it is a change
// if oldValue===undefined it is new
// if value===undefined it is delete
; // called after set
},
execute({value,key,args,user,request}) {
; // called after execute, value is the result, key is the function name
},
keys: {
password: {
write({value,key,oldValue,user,request}) {
; // called after set
}
}
}
}
/* Edges are just nested keys or wildcards, e.g.
people: {
_: { // matches any sub-edge
secretPhrase: { // matches secrePhrase edge
read(...) { ... },
write(...) { ... }
}
}
}
*/
}
}
}).call(this);Functions top
Exposing server functions to the JavaScript client in the browser is simple. Just define the functions in
the file functions.js. Any asynchronous functions will be awaited.
(function() {
module.exports = {
client: { // added only to the client and invoked only there
},
worker: {
}
cloud: { // added to the client, but invoked on the server
securedTestFunction() {
return "If you see this, there may be a security leak";
},
getDate() {
return new Date();
}
}
}
}).call(this);The functions will automatically become available in the admin client docs/thunderclap. Function execution in the cloud can
be access controlled in secure.js.
Indexing top
All properties of objects inserted using putItem are indexed for direct match, with the exception of properties
containing strings over 64 characters in length. Strings longer than 64 characters can be matched use $search.
Objects that are just a value to setItem are not indexed. The index partitioned per class, but searches can be conducted
across all classes by the use of the wildcard key _.
The root index node can be accessed via keys("!"). Direct access is restricted to users with the role dbo.
Indexes in Thunderclap consume very little RAM, they are primarily composed of specially formed and partitioned keys
pointing to just the value 1. This means the performance of Thunderclap is heavily dependent on the performance of
the Cloudflare KV with respect to iterating keys. It also means that Thunderclap can have an unlimited number of objects
indexed. The largest object that can be stored is 2MB (the same as Cloudflare KV).
Full Text Indexing top
Any strings containg spaces are automatically added to a full-text index based on trigrams
after stop words such as and, but, or have been removed. These can be searched using the {$search: <phrase>} pattern.
Schema top
The use of schema is optional with Thunderclap. They can be used to validate data in all tiers of an application: browser,
worker, or cloud. The built in classes User, Position, and Coordinates all have schema. If schema are present, they
are automatically used to validate data prior to insert in the cloud. They can be optionally applied in the browser.
Below is an example for User. Note, by convention Schema are attached to classes as a static property.
User.schema = {
userName: {required:true, type: "string", unique:true},
roles: {type: "object"}
}The following constraints are supported:
matches:RegExp - Checks to see if a property value matches the provided regular expression.
noindex:boolean - If present and truthy, prevents indexing of a property.
oneof:Array - Checks to see if a property value is in the provided array.
required:boolean - Ensures the property has a value.
unique:boolean - Does a database look-up to ensure no other entity of the same class has the same property value. (Not yet implemented).
validate:function - Calls a custom validation function with the signature (object object,string key,any value,Array errors,Thunderclap db).
The function is responsible for pushing any errors into the provided errors array.
Development top
If you wish to modify Thunderclap, you must subscribe to the Cloudflare Argo tunneling service on the domain where you wish to use Thunderclap.
Create a Cloudflare Workers KV namespace using the Cloudflare Workers control panel. By convention the following name form is recommended so that the name parrallels the name of the worker script generated by the Thunderclap.
<devHost>-thunderclap-<primaryHostName>-<com|org|...>
e.g. myname-thunderclap-mydomain-com is the KV namespace and script name associated with myname-thunderclap.mydomain.com
You do not need a CNAME record for your dev host, Argo manages this for you.
Run the thunderclap script:
npm run thunderclap
If the 'mode' in 'thunderclap.jsonis set todevelopment, then in addition to deploying the worker script to -thunderclap--<com|org|...>with a route, a local web server is started with an Argo tunnel to access-thunderclap.` via your web browser.
If you access https://<dev-host-prefix>-thunderclap.<your-domain>/test/ via your web browser, the unit test file
will load.
When in dev mode, files are watched by webpack and any changes cause a re-bundling and deployment of the worker script to Cloudflare.
Admin UI top
When in development mode, there is a primitive UI for making one-off requests at
https://<dev-host-prefix>-thunderclap.<your-domain>/thunderclap.html. This UI exposes all of the functions
available via the Javascript client.
History and Roadmap top
Many of the concepts in Thunderclap were first explored in ReasonDB. ReasonDB development has been suspended for now, but many features found in ReasonDB will make their way into Thunderclap if interest is shown in the software.
Change Log (reverse chronological order) top
2019-07-24 v0.0.33a Added graph add and remove for set operations on values.
2019-07-24 v0.0.32a Minor documentation fixes.
2019-07-23 v0.0.31a Slight performance inmprovements. Fixed broken $search.
2019-07-22 v0.0.30a Security now works on graph paths.
2019-07-22 v0.0.29a Started adding graph database capability. Not yet tied to triggers, security, etc. Reworked triggers, security, so that they will work across all of key-value, JSON, and graph storage. If you are using any, they will need substantive re-work.
2019-07-16 v0.0.28a Multiple classes can now be queried at the same time.
2019-07-15 v0.0.27a Added many user management functions.
2019-07-14 v0.0.26a Modified indexing and query approach to use classes at top level, i.e. {<cname>:<pattern>}
instead of <pattern>. NAMESPACES must be recreated. The ability to query across classes will be re-introduced in
a subsequent release using {_:}. This change will improve performance is real-world cases by further
partitioning keys and also making unique key look-up/verification much faster.
2019-07-13 v0.0.25a Added unique(cnameOrIdOrObject,property,value).
2019-07-13 v0.0.24a Fixed $instanceof and added $isa. Eliminated gloabal leak in unit test for Schema validation.
2019-07-12 v0.0.23a Full text search repaired. Optimized inserts and deletes. NAMESPACES must be recreated.
2019-07-12 v0.0.22a Ehanced documentation. Completely re-worked indexing to allow for more object storage. Full text search currently broken. NAMESPACES must be re-created.
2019-07-11 v0.0.21a Ehanced documentation.
2019-07-10 v0.0.20a Ehanced documentation. Added Position and Coordinates.
2019-07-09 v0.0.19a Server was throwing errors on date predicates. Fixed. Added support for un-indexing nested objects. Unindexing full-text not yet implemented. Added a short term cache to improve performance. Unit tests for removeItem are failing as a result. Not sure why.
2019-07-06 v0.0.18a Added nested object indexing (unindex does not yet work).
2019-07-04 v0.0.17a Added full text indexing with {$search: string terms} or {$search: [string terms, number pctMatch]}
2019-07-03 v0.0.16a Added changePassword(userName,password,oldPassword).
2019-07-02 v0.0.15a Added clear(prefix), entries(prefix,options), hasKey(key), values(prefix,options).
All are limited to dbo access. Reverted to two level index for now to address performance. Limits number of
entries per index due to 128MB limit of Cloudflare Workers.
2019-06-30 v0.0.14a Ehanced triggers and functions to allow browser, service worker, or cloud execution.
Added when capability. Service worker support will operate once service workers are generated during the
build process.
2019-06-26 v0.0.13a Indexing optimized to reeuce RAM usage. Substantive performance drop.
2019-06-26 v0.0.12a Indexing now extends to 3 levels to provide more data spread. Sub-objects still not indexed as direct paths. Added support for expiring keys and listing keys.
2019-06-25 v0.0.11a Code optimizations and bug fixes.
2019-06-24 v0.0.10a Custom function support added.
2019-06-22 v0.0.9a Triggers on put, update, remove.
2019-06-22 v0.0.8a Triggers now working for putItem.
2019-06-22 v0.0.7a Added JOQULAR pattern $near:[target,range]. Range can be a number, in which case it is
added/substracted or a string ending in the % sign, in which case the percentage is add/substracted. Added
stress tests up to 1000 items. Started support for RegExp as acl keys. Enhanced doucmentation.
2019-06-21 v0.0.6a Documentation improvements.
2019-06-21 v0.0.5a ACL improvements. More of unit tests.
2019-06-20 v0.0.4a Added a large number of unit tests