scripting/nodeStorage

nodeStorage

A simple server-side JavaScript storage system using Twitter identity, running in Node.js.

Overview

1. Public and private data.

2. Store data in local filesystem or on Amazon S3.

3. User identity through Twitter.

4. The back-end of an open source blogging system, with MyWord Editor as the front-end.

5. Written by Dave Winer, first deployed in June 2014, supports a small set of browser-based JavaScript software. Hope to expand to become a community of apps.

How to set up

Here's quick guide to setting up a nodeStorage server on Ubuntu, but probably works on many other Unix systems. It doesn't assume you have Node.js installed or a Git client. It shows you how to set up a connection with Twitter. It stores user data, both public and private, in the filesystem of the Ubuntu machine.

It's a good guide for installing on any Unix system, but has been tested on Ubuntu.

Links

1. There are a lot of docs about nodeStorage on the 1999-project site.

2. I wrote a backgrounder that explains the philosophy of nodeStorage, what it can be used for and where it's likely to go as it evolves.

3. 1999.io, my new blogging system, is a nodeStorage app and it has streamlined setup instructions.

4. A blog post provides another perspective.

5. Andrew Shell wrote a guide to setting up a new nodeStorage server.

6. Marco Fabbri wrote a howto for Heroku server setup.

7. Why use Twitter for identity?

Demo app #1

1. A minimal demo app.

2. The full source for the demo app is available on GitHub. MIT License.

Demo app #2

1. MyWord Editor is a simple and beautiful blogging system.

2. It's fully open source, MIT License.

3. The back-end is nodeStorage.

API

api.js file provides glue for browser-based JavaScript apps.

You can also access it in api.nodestorage.io and include it from apps if you want.

Updates

v0.9.17 -- 10/2/20 by DW

nodeStorage becomes a real package. See package.md in the repo for an explanation and background.

v0.9.11 - 4/14/20 by DW

Add new /myfiles call, requires authentication, and returns a zip archive containing the user's files. Only works if the storage is local, not on S3.

v0.9.10 - 3/6/20 by DW

Add new /disconnect call, requires authentication, and removes the user from the screen name cache. The need for this came up in the development of feedBase. Little Outliner calls this endpoint.

v0.9.9 - 3/5/20 by DW

In the /getfilelist call, if it's the local file system, we return extra info about each file: when it was created, modified, its size. No changes for S3 storage. This was used to implement the new File/Open command in Little Outliner.

v0.9.7 - 9/19/19 by DW

If process.env.PORT is set when we launch, use that instead of what's in config.json.

v0.9.5 - 2/27/18 by DW

New config option -- flUsePortInRedirect. Default true. See this post for an explanation.

v0.9.4 - 8/15/17 by DW

Changed to NPM-friendly version numbering.

Added a new routine to api.js, nodeStorageApp. It factors out some of the complexity of writing a nodeStorage-able app, based on what we know now, vs what we knew then. ;-)

The demo app for the new interface is MacWrite.

All the previous api.js routines work as before, no changes.

v0.95b - 7/20/16 by DW

Replaced calls to "encode" in encodeURIComponent with handleHttpRequest. This helped resolve an issue Don Park was having on his system.

v0.95a - 7/17/16 by DW

A problem showed up in Safari only, on a customized menu, on a site with a custom URL.

When we get the HTML of the menu from the server, it redirects from the long URL to the short one, as if we were a user coming in through a web browser.

This works fine in Chrome, the $.ajax call in the browser just redirects, but not in Safari. A search turns up several Stack Exchange articles that say that Apple believes the HTML 5 spec tells them to work this way. Whatever.

So I added support for a new param on a request, noredirect, if it's true, we don't do the redirect, we just return the content of the object requested (the HTML of the menu in this case) without redirecting.

Amazingly it worked. ;-)

Here's the page that illustrated the problem, but I've seen it on other sites before, didn't have the time to chase it down.

v0.94w - 6/25/16 by DW

New config.json value -- url404page. It points to an HTML page that's returned when a 404 error is generated.

Previously we'd return a 500 error where we should have returned 404, and the page we displayed showed the contents of a JSON struct. Now It gives you a plain 404 "not found" message, and gives you the opportunity to override the default choice.

Here's an example of a config.json that specified a 404 page.

v0.93 - 4/3/16 by DW

Added a call to ping the "community server" when the RSS feed updates in addition to the rssCloud server. See lib/rss.js for details.

v0.92 - 3/26/16 by DW

Sorry for the lack of update notes. What's been going on has been the development of nodeStorage as a server for 1999.io.

At this point the server should be as easy to install as it was previously, after some docs are written, and from here-out I plan to keeop the update notes current with the development work.

v0.79 - 9/9/15 by DW

In the new chat functionality, when we return from a longpoll, we used to send back the entire chatlog. This is not very efficient, esp over a mobile connection. Now we only send back the new item that caused us to return. See postChatMessage for the change.

v0.78 - 9/3/15 by DW

New routines support a basic chat capability.

Slack-compatible incoming and outgoing webhooks.

Can be configured to watch for a change to the mod date of storage.js, which causes it to quit (presumably to be relaunched by the OS or forever). Makes installing updates automatic.

By default, longpolls timeout after 60 seconds, previously it was 20 seconds.

The everyMinute script now runs at the top of the minute. Previously it was a function of when the app was launched. It also announces itself with the current time and the version of nodeStorage.

v0.77 - 7/29/15 by DW

We can now store user data in the local file system. Previous versions could only store data in Amazon S3. The changes are documented in this post.

v0.75 - 7/15/15 by DW

New functionality to support named outlines.

v0.74 - 5/20/15 by DW

Fixed an error in the way whitelists are specified.

v0.73 - 5/10/15 by DW

A new way to configure nodeStorage, with a config.json file.

v0.72 - 3/5/15 by DW

The /getfilelist endpoint now returns an array of objects, one for each file, as before. The objects contain three values, path, whenLastChange and ctChars.

Previously we were returning the array that S3 returns to us. The file paths it returns are not easily used by a client.

There is a possibility of breakage if you had built on this endpoint, but given its previous state, it's hard to imagine anyone doing that. ;-)

v0.66 -1/29/15 by DW

New stats for longpolling, in serverStats.json. Example. Gives us a way to see how many people are watching resources managed by a nodeStorage server.

v0.65 -1/24/15 by DW

Added a readout of free memory as the second item in the server log.

In this screen shot the arrow points to the free memory readout.

Questions, problems

Please post a note on the server-snacks mail list.