charlesroelli/org-board

org-board
=========

Last updated:  Wed 30 May 2018 20:06:55 CEST


* Motivation

 org-board is a bookmarking and web archival system for Emacs Org
 mode, building on ideas from Pinboard <https://pinboard.in>.  It
 archives your bookmarks so that you can access them even when
 you're not online, or when the site hosting them goes down.
 `wget' is used as a backend for archival, so any of its options
 can be used directly from org-board.  This means you can download
 whole sites for archival with a couple of keystrokes, while
 keeping track of your archives from a simple Org file.

* Summary

 In org-board, a bookmark is represented by an Org heading of any
 level, with a `URL' property containing one or more URLs.  Once
 such a heading is created, a call to `org-board-archive' creates a
 unique ID and directory for the entry via `org-attach', archives
 the contents and requisites of the page(s) listed in the `URL'
 property using `wget', and saves them inside the entry's
 directory.  A link to the (timestamped) root archive folder is
 created in the property `ARCHIVED_AT'.  Multiple archives can be
 made for each entry.  Additional options to pass to `wget' can be
 specified via the property `WGET_OPTIONS'.  The variable
 `org-board-after-archive-functions' (defaulting to nil) holds a
 list of functions to run after each archival operation.

* User commands

 `org-board-archive' archives the current entry, creating a unique
   ID and directory via `org-attach' if necessary.

 `org-board-archive-dry-run' shows the `wget' invocation that will
   run for this entry in the echo area.

 `org-board-new' prompts for a URL to add to the current entry's
   properties, then archives the entry immediately.

 `org-board-delete-all' deletes all the archives for this entry by
   deleting the `org-attach' directory.

 `org-board-open' opens the bookmark at point in a browser.
   Default to the built-in browser, `eww', and with prefix, the
   native operating system browser.

 `org-board-diff' uses `zdiff' (if available) or `ediff' to
   recursively diff two archives of the same entry.

 `org-board-diff3' uses `ediff' to recursively diff three archives
   of the same entry.

 `org-board-cancel' cancels the current org-board archival process.

 `org-board-run-after-archive-function' prompts for a function and
   an archive in the current entry, and applies the function to the
   archive.

 These are all bound in the `org-board-keymap' variable (not bound
 to any key by default).

* Customizable options

 `org-board-wget-program' is the path to the wget program.

 `org-board-wget-switches' are the command line options to use with
 `wget'.  By default these are included as:

   "-e robots=off"      ignores robots.txt files.
   "--page-requisites"  downloads all page requisites (CSS, images).
   "--adjust-extension" add a ".html" extension where needed.
   "--convert-links"    convert external links to internal.

 `org-board-agent-header-alist' is an alist mapping agent names to
 their respective header/user-agent arguments.  Set a
 `WGET_OPTIONS' property to a key of this alist (say,
 `Mac-OS-10.8') and org-board will replace the key with its
 corresponding value before calling wget. This is useful for some
 sites that refuse to serve pages to `wget'.

 `org-board-wget-show-buffer' controls whether the archival process
 buffer is shown in a window (defaults to true).

 `org-board-log-wget-invocation' controls whether to log the
 archival process command in the root of the archival directory
 (defaults to true).

 `org-board-domain-regexp-alist' applies certain options when a
 domain matches a regular expression.  See the docstring for
 details.  As an example, this is used to make sure that `wget'
 does not send a User Agent string when archiving from Google
 Cache, which will not normally serve pages to it.

 `org-board-after-archive-functions' (default nil) holds a list of
 functions to run after an archival takes place.  This is intended
 for user extensions to `org-board'.  The functions receive three
 arguments: a list of URLs downloaded, the folder name where they
 were downloaded and the process filter event string (see the Elisp
 manual for details on the possible values of this string).  For an
 example use of `org-board-after-archive-functions', see the
 "Example usage" section below.

* Known limitations

 Options like "--header: 'Agent X" cannot be specified as
 properties, because the property API splits on spaces, and such an
 option has to be passed to `wget' as one argument.  To work around
 this, add these types of options to `org-board-agent-header-alist'
 instead, where the property API is not involved.

 At the moment, only one archive can be done at a time.

* Example usage

** Archiving

 I recently found a list of articles on linkers that I wanted to
 bookmark and keep locally for offline reading.  In a dedicated org
 file for bookmarks I created this entry:

 ** TODO Linkers (20-part series)
 :PROPERTIES:
 :URL:          http://a3f.at/lists/linkers
 :WGET_OPTIONS: --recursive -l 1 --span-hosts
 :END:

 Where the `URL' property is a page that already lists the URLs
 that I wanted to download.  I specified the recursive property for
 `wget' along with a depth of 1 ("-l 1") so that each linked page
 would be downloaded.  With point inside the entry, I run "M-x
 org-board-archive".  An `org-attach' directory is created and
 `wget' starts downloading the pages to it.  Afterwards, the end
 the entry looks like this:

 ** TODO Linkers (20-part series)
 :PROPERTIES:
 :URL:          http://a3f.at/lists/linkers
 :WGET_OPTIONS: --recursive -l 1 --span-hosts
 :ID:           D3BCE79F-C465-45D5-847E-7733684B9812
 :ARCHIVED_AT:  [2016-08-30-Tue-15-03-56]
 :END:

 The value in the `ARCHIVED_AT' property is a link that points to
 the root of the timestamped archival directory.  The ID property
 was automatically generated by `org-attach'.

** Diffing

 You can diff between two archives done for the same entry using
 `org-board-diff', so you can see how a page has changed over time.
 The diff recurses through the directory structure of an archive
 and will highlight any changes that have been made.  `ediff' is
 used if `zdiff' is not available (both are capable of recursing
 through a directory structure, but `zdiff' is possibly more
 intuitive to use).  `org-board-diff3' also offers diffing between
 three different archive directories.

** `org-board-after-archive-functions'

 `org-board-after-archive-functions' is a list of functions run
 after an archive is finished.  You can use it to do anything you
 like with newly archived pages.  For example, you could add a
 function that copies the new archive to an external hard disk, or
 opens the archived page in your browser as soon as it is done
 downloading.  You could also, for instance, copy all of the media
 files that were downloaded to your own media folder, and pop up a
 Dired buffer inside that folder to give you the chance to
 organize them.

 Here is an example function that copies the archived page to an
 external service called `IPFS' <http://ipfs.io/>, a decentralized
 versioning and storage system geared towards web content (thanks
 to Alan Schmitt):

 (defun org-board-add-to-ipfs (urls output-folder event &rest _rest)
   "Add the downloaded site to IPFS."
   (unless (string-match "exited abnormally" event)
     (let* ((parsed-url (url-generic-parse-url (car urls)))
            (domain (url-host parsed-url))
            (path (url-filename parsed-url))
            (output (shell-command-to-string
 		     (concat "ipfs add -r "
 			     (concat output-folder domain))))
            (ipref
 	     (nth 1 (split-string
 		     (car (last (split-string output "\n" t))) " "))))
       (with-current-buffer (get-buffer-create "*org-board-post-archive*")
         (princ (format "your file is at %s\n"
 			(concat "http://localhost:8080/ipfs/" ipref path))
 		(current-buffer))))))

 (eval-after-load "org-board"
   '(add-hook 'org-board-after-archive-functions 'org-board-add-to-ipfs))

 Note that for forward compatibility, it's best to add to a final
 `&rest' argument to every function listed in
 `org-board-after-archive-functions', since a future update may
 provide each function with additional arguments (like a marker
 pointing to a buffer position where the archive was initiated, for
 example).

 For more information on `org-board-after-archive-functions', see
 its docstring and the docstring of
 `org-board-test-after-archive-function'.

 You can also interactively run an after-archive function with the
 command `org-board-run-after-archive-function'.  See its docstring
 for details.


* Getting started

** Installation

 There are two ways to install the package.  One way is to clone
 this repository and add the directory to your load-path manually.

 (add-to-list 'load-path "/path/to/org-board")
 (require 'org-board)

 Alternatively, you can download the package directly from Emacs
 using MELPA <https://github.com/melpa/melpa>.  M-x
 package-install RET org-board RET will take care of it.

** Keybindings

 The following keymap is defined in `org-board-keymap':

 | Key | Command                              |
 | a   | org-board-archive                    |
 | r   | org-board-archive-dry-run            |
 | n   | org-board-new                        |
 | k   | org-board-delete-all                 |
 | o   | org-board-open                       |
 | d   | org-board-diff                       |
 | 3   | org-board-diff3                      |
 | c   | org-board-cancel                     |
 | x   | org-board-run-after-archive-function |
 | O   | org-attach-reveal-in-emacs           |
 | ?   | Show help for this keymap.           |

 To install the keymap give it a prefix key, e.g.:

 (global-set-key (kbd "C-c o") org-board-keymap)

 Then typing `C-c o a' would run `org-board-archive', for example.

* Miscellaneous

 The location of `wget' should be picked up automatically from the
 `PATH' environment variable.  If it is not, then the variable
 `org-board-wget-program' can be customized.

 Other options are already set so that archiving bookmarks is done
 pretty much automatically.  With no `WGET_OPTIONS' specified, by
 default `org-board-archive' will just download the page and its
 requisites (images and CSS), and nothing else.

** Support for org-capture from Firefox (thanks to Alan Schmitt):

 On the Firefox side, install org-capture from here:

   http://chadok.info/firefox-org-capture/

 Alternatively, you can do it manually by following the
 instructions here:

   http://weblog.zamazal.org/org-mode-firefox/
     (in the “The advanced way” section)

 When org-capture is installed, add `(require 'org-protocol)' to
 your init file (`~/.emacs').

 Then create a capture template like this:

   (setq org-board-capture-file "my-org-board.org")

   (setq org-capture-templates
         `(...
           ("c" "capture through org protocol" entry
             (file+headline ,org-board-capture-file "Unsorted")
             "* %?%:description\n:PROPERTIES:\n:URL: %:link\n:END:\n\n Added %U")
           ...))

 And add a hook to `org-capture-before-finalize-hook':

   (defun do-org-board-dl-hook ()
     (when (equal (buffer-name)
             (concat "CAPTURE-" org-board-capture-file))
       (org-board-archive)))

   (add-hook 'org-capture-before-finalize-hook 'do-org-board-dl-hook)

* Acknowledgements

 Thanks to Alan Schmitt for the code to combine `org-board' and
 `org-capture', and for the example function used in the
 documentation of `org-board-after-archive-functions' above.