scripting/river4

River4

River4 is a JavaScript river-of-news aggregator running in Node.js

Important: River5 has been released. It's simpler to set up and runs faster.

How to install

Here are the instructions for setting up River4 on a system running Node.js, using the local file system for storage.

Links

1. Welcome to River4.

2. What is a River of News aggregator?

3. Why every news organization should have a river.

4. The River4 support mail list.

5. How to ask for help with software.

6. The Hello World of Rivers.

7. Editing subscription lists.

8. Installing River4 on Ubuntu.

Updates

v0.120 -- 8/21/15 by DW

Watch for a change to the mod date of the river4.js file. Quit if it changes. But only if the feature is enabled via config.flWatchAppDateChange boolean.

If you changed the name of the River4 app, you can provide the new name in config.fnameApp.

Here's an example of a config.json for this setup.

v0.118 -- 8/13/15 by DW

Now handles feeds whose address begins with feed://. We just translate it to http://. Previously there would be an error reading the feed.

Also created a module, utils.js, instead of including the source in the main body of river.js. River4 was the first Node app I shipped, I've since learned how to modularize Node apps.

v0.117 -- 7/20/15 by DW

Lots of small changes to make River4 easier to setup for newbies. ;-)

1. If you haven't set any path variables, or set up config.json, River4 will run with the data stored in a river4data sub-folder of the folder containing river4.js, which it automatically creates.

2. We don't announce each step of the startup process on the console.

3. If there's an ENOENT error reading prefsAndStats.json, we don't report an error, since River4 automatically creates the file the first time it runs. This confused some users, unnecessarily.

4. We create the lists folder automatically.

5. We no longer install an index.html file in the river4data folder. The file we installed didn't actually work, and it's no longer necessary since the home page of the server is now a perfectly good way to browse the rivers on the server.

6. Re-wrote the howto for setup, eliminating two complicated and potentially error-prone steps. The old howto is still there but with a bold statement at the top saying you should use the new one.

7. There's a 15-minute video that shows how to set up a River4 installation.

v0.116 -- 7/4/15 by DW

The home page of the River4 server now shows you the rivers being maintained by the server. There's a menu that links to the dashboard, the blog, mail list, and GitHub repo.

v0.115 -- 6/21/15 by DW

New feature: Callback scripts that run when River4 adds an item to the river.

v0.114 -- 6/16/15 by DW

Fixed JSON encoding problem reported by Andrew Shell.

v0.113 -- 6/7/15 by DW

We now record the current time in each item in the calendar structure. This is used when building a river to set the whenLastUpdate field.

v0.112 -- 6/5/15 by DW

Now when we receive a message saying that a feed updated, we read the feed and rebuild all rivers that it's part of. I wanted to test the framework before going this step.

Again, a careful code review and testing by others would be appreciated.

v0.111 -- 6/4/15 by DW

Added support for rssCloud. Now if a feed has a <cloud> element, we contact the server and go through the subscription protocol. If it all works, we'll be notified of updates to the feed before we poll.

The rssCloud support is largely untested. However I have upgraded all my copies of River4 to run the new version, and it seems to be functioning well. Code review of the new functionality would be much appreciated.

We also remove items from each feed's history array when the item no longer appears in the feed. This reduces the size of some of the files in the data folder, in general making the software more efficient.

Fixed an error that would cause River4 to crash when there were no OPML subscription list files in the lists folder.

v0.110 -- 5/10/15 by DW

A new way to configure River4, using a config.son file in the same directory as river4.js.

v0.109 -- 4/21/15 by DW

Fixed a bug that would cause generated rivers to be empty immediately after date rollover.

The fix was to write out an empty array in the calendar structure when the date rollover occurs. The problem was that until there was a new item saved for the day, the first read of the calendar, when building a river would fail, causing the build to finish.

The problem was discovered in podcatch.com, and written up on the River4 blog.

v0.108 -- 12/5/14 by DW

There's now a River4 Console app, at http://river4.io/ that allows you to edit subscription lists in an outliner, and set some of the server preferences remotely. It's documented on the River4 blog.

v0.100 -- 11/2/14 by DW

Fixed a bug in file name processing.

v0.99 -- 11/1/14 by DW

Two fixes for local file system use. 1. Only read lists whose names end with .opml -- there were invisible files on the Mac that would cause problems. 2. When running on Windows and writing to the local file system, there are more illegal characters. Replace them with underscores.

v0.97 -- 10/16/14 by DW

Apparently there was a change in format in the FeedParser module, in the way it represents <source:outline> elements. This release handles the change in format so outlines now pass through in a way that's understandable to the RiverBrowser software.

v0.96 -- 9/24/14 by DW

This version can be configured to store its data in the local filesystem instead of S3. See the blog post for details.

v0.95 -- 9/11/14 by DW

New /ping endpoint, available to be called by a publisher, on behalf of a user, to indicate that a feed has updated, and should be read immediately. Radio3 has this facility as of today, as does Fargo.

v0.94 -- 8/6/14 by DW

Fixed a problem that caused rivers to display only old stories. Full explanation on the blog.

v0.93 -- 7/31/14 by DW

Added more fields to the struct the /status call returns. It now says what the s3path is, what port the server is running on, and if you've defined a s3defaultAcl (see v0.91) what the value of that parameter is.

v0.91 -- 7/19/14 by DW

A new environment variable, s3defaultAcl, if present specifies the permissions on S3 files we create. The default is public-read. With this parameter, it may be possible to run a private installation of River4.

v0.90 -- 7/19/14 by DW

New <source:outline> elements flow through River4. See the docs for the source namespace for details.

v0.89 -- 6/19/14 by DW

One small change to package.json, and no changes to the JavaScript code.

v0.88 -- 6/17/14 by DW

A subscription list can now contain an include node, so you can have a list of lists. Full explanation in this blog post.

Changed the package.json file to require Node v0.8.x. Previously it was 0.6.x. This should make it possible to deploy on Nodejitsu without modification, per Dave Seidel's report.

v0.87 -- 6/09/14 by DW

Fixed a bug that would cause River4 to crash when processing an item with a null title.

v0.86 -- 6/09/14 by DW

Fixed a bug that would cause River4 to crash when reading an item from a subscription list that didn't have an xmlUrl attribute.

v0.85 -- 6/07/14 by DW

Two fixes, explained here.

v0.84 -- 6/06/14 by DW

Two fixes, explained here.

v0.83 -- 6/06/14 by DW

Now if there's an error in any JSON code we try to parse, we display an error message in the console, along with the path to the S3 file we were trying to read.

v0.81 -- 6/02/14 by DW

serverData.stats now has a copy of the last story added to the river. The dashboard page displays it.

v0.80 -- 6/02/14 by DW

New "dashboard" feature. If your server is running at aggregator.mydomain.com, if you go to:

http://aggregator.mydomain.com/dashboard

You'll get a real-time readout of what your aggregator is doing.

The HTML source for the dashboard page is in dashboard.opml in the opml folder in the repository.

Docker

There is an experimental Docker installer. Notes about using it are on the wiki.

Amazon S3, Heroku

When we started developing River4, we were targeting Heroku, because it was so easy and inexpensive to start with. They have since changed their pricing, so it's not as attractive, so we're now recommending the filesystem configuration, above.

1. The howto with instructions for installing River4 using Amazon S3 for storage.

2. Heroku How To -- get a Heroku server running with Fargo Publisher, the back-end for Fargo.

3. Bare-bones Heroku do -- checklist for setting up a Heroku server running Node.js from a Mac desktop.

Thanks!

Thanks to two developer friends, Dan MacTough and Eric Kidd, who helped this Node.js newbie get this app up and running.

Specifically thanks to Dan for writing the excellent feedparser and opmlparser packages that are incorporated in River4.

Notes

1. I edit code in an outliner, which is then turned into JavaScript. The "opml" folder in the repository contains the versions of the code that I edit. The comments are stripped out of the code before it's converted to raw JS, so there is information for developers in the OPML that isn't in the main files (though all the running code is in both).

2. The first released version is 0.79. They will increment by one one-hundredth every release. At some point I'll call it 1.0, then subsequent releases will be 1.01, 1.02 etc.

Questions, comments?

Please post a note on the River4 mail list.