clingon

clingon is a command-line options parser system for Common Lisp.

A summary of the features supported by clingon is provided below.

• Native support for sub-commands
• Support for command aliases
• Short and long option names support
• Related options may be grouped into categories
• Short options may be collapsed as a single argument, e.g. -xyz
• Long options support both notations - --long-opt arg and --long-opt=arg.
• Automatic generation of help/usage information for commands and sub-commands
• Out of the box support for --version and --help flags
• Support for various kinds of options like string, integer, boolean, switches, enums, list, counter, filepath, etc.
• Sub-commands can lookup global options and flags defined in parent commands
• Support for options, which may be required
• Options can be initialized via environment variables
• Single interface for creating options using CLINGON:MAKE-OPTION
• Generate documentation for your command-line app
• Support for pre-hook and post-hook actions for commands, which allows invoking functions before and after the respective handler of the command is executed
• Support for Bash and Zsh shell completions
• clingon is extensible, so if you don’t find something you need you can extend it by developing a new option kind, or even new mechanism for initializing options, e.g. by looking up an external key/value store.

Scroll to the demo section in order to see some examples of clingon in action.

Other Common Lisp option parser systems, which you might consider checking out.

• unix-opts
• adopt
• clon

Quick Example

Here’s a really quick example of a simple CLI application, which greets people.

(in-package :cl-user)
(defpackage :clingon.example.greet
  (:use :cl)
  (:import-from :clingon)
  (:export
   :main))
(in-package :clingon.example.greet)

(defun greet/options ()
  "Returns the options for the `greet' command"
  (list
   (clingon:make-option
    :string
    :description "Person to greet"
    :short-name #\u
    :long-name "user"
    :initial-value "stranger"
    :env-vars '("USER")
    :key :user)))

(defun greet/handler (cmd)
  "Handler for the `greet' command"
  (let ((who (clingon:getopt cmd :user)))
    (format t "Hello, ~A!~%" who)))

(defun greet/command ()
  "A command to greet someone"
  (clingon:make-command
   :name "greet"
   :description "greets people"
   :version "0.1.0"
   :authors '("John Doe <[email protected]")
   :license "BSD 2-Clause"
   :options (greet/options)
   :handler #'greet/handler))

(defun main ()
  "The main entrypoint of our CLI program"
  (let ((app (greet/command)))
    (clingon:run app)))

This small example shows a lot of details about how apps are structured with clingon.

You can see there’s a main function, which will be the entrypoint for our ASDF system. Then you can find the greet/command function, which creates and returns a new command.

The greet/options functions returns the options associated with our sample command.

And we also have the greet/handler function, which is the function that will be invoked when users run our command-line app.

This way of organizing command, options and handlers makes it easy to re-use common options, or even handlers, and wire up any sub-commands anyway you prefer.

You can find additional examples included in the test suite for clingon.

Demo

You can also build and run the clingon demo application, which includes the greet command introduced in the previous section, along with other examples.

Clone the clingon repo in your Quicklisp local-projects directory.

git clone https://github.com/dnaeon/clingon

Register it to your local Quicklisp projects.

CL-USER> (ql:register-local-projects)

Building the Demo App

You can build the demo app using SBCL with the following command.

LISP=sbcl make demo

Build the demo app using Clozure CL:

LISP=ccl make demo

In order to build the demo app using ECL you need to follow these instructions, which are ECL-specific. See Compiling with ASDF from the ECL manual for more details. First, load the :clingon.demo system.

(ql:quickload :clingon.demo)

And now build the binary with ECL:

(asdf:make-build :clingon.demo
                 :type :program
                 :move-here #P"./"
                 :epilogue-code '(clingon.demo:main))

This will create a new executable clingon-demo, which you can now execute.

Optionally, you can also enable the bash completions support.

APP=clingon-demo source extras/completions.bash

In order to activate the Zsh completions, install the completions script in your ~/.zsh-completions directory (or anywhere else you prefer) and update your ~/.zshrc file, so that the completions are loaded.

Make sure that you have these lines in your ~/.zshrc file.

fpath=(~/.zsh-completions $fpath)
autoload -U compinit
compinit

The following command will generate the Zsh completions script.

./clingon-demo zsh-completion > ~/.zsh-completions/_clingon-demo

Use the --help flag to see some usage information about the demo application.

./clingon-demo --help

Requirements

• Quicklisp

Installation

The clingon system is not yet part of Quicklisp, so for now you need to install it in your local Quicklisp projects.

Clone the repo in your Quicklisp local-projects directory.

(ql:register-local-projects)

Then load the system.

(ql:quickload :clingon)

Step By Step Guide

In this section we will implement a simple CLI application, and explain at each step what and why we do the things we do.

Once you are done with it, you should have a pretty good understanding of the clingon system and be able to further extend the sample application on your own.

We will be developing the application interactively and in the REPL. Finally we will create an ASDF system for our CLI app, so we can build it and ship it.

The code we develop as part of this section will reside in a file named intro.lisp. Anything we write will be sent to the Lisp REPL, so we can compile it and get quick feedback about the things we’ve done so far.

You can find the complete code we’ll develop in this section in the clingon/examples/intro directory.

Start the REPL

Start up your REPL session and let’s load the clingon system.

CL-USER> (ql:quickload :clingon)
To load "clingon":
  Load 1 ASDF system:
    clingon
; Loading "clingon"

(:CLINGON)

Create a new package

First, we will define a new package for our application and switch to it.

(in-package :cl-user)
(defpackage :clingon.intro
  (:use :cl)
  (:import-from :clingon)
  (:export :main))
(in-package :clingon.intro)

We have our package, so now we can proceed to the next section and create our first command.

Creating a new command

The first thing we’ll do is to create a new command. Commands are created using the CLINGON:MAKE-COMMAND function.

Each command has a name, description, any options that the command accepts, any sub-commands the command knows about, etc.

The command in clingon is represented by the CLINGON:COMMAND class, which contains many other slots as well, which you can lookup.

(defun top-level/command ()
  "Creates and returns the top-level command"
  (clingon:make-command
   :name "clingon-intro"
   :description "my first clingon cli app"
   :version "0.1.0"
   :license "BSD 2-Clause"
   :authors '("John Doe <[email protected]>")))

This is how our simple command looks like. For now it doesn’t do much, and in fact it won’t execute anything, but we will fix that as we go.

What is important to note, is that we are using a convention here to make things easier to understand and organize our code base.

Functions that return new commands will be named <name>/command. A similar approach is taken when we define options for a given command, e.g. <name>/options and for sub-commands we use <name>/sub-commands. Handlers will use the <name>/handler notation.

This makes things easier later on, when we introduce new sub-commands, and when we need to wire things up we can refer to our commands using the established naming convention. Of course, it’s up to you to decide which approach to take, so feel free to adjust the layout of the code to your personal preferences. In this guide we will use the afore mentioned approach.

Commands can be linked together in order to form a tree of commands and sub-commands. We will talk about that one in more details in the later sections of this guide.

Adding options

Next, we will add a couple of options. Similar to the previous section we will define a new function, which simply returns a list of valid options. Defining it in the following way would make it easier to re-use these options later on, in case you have another command, which uses the exact same set of options.

clingon exposes a single interface for creating options via the CLINGON:MAKE-OPTION generic function. This unified interface will allow developers to create and ship new option kinds, and still have their users leverage a common interface for the options via the CLINGON:MAKE-OPTION interface.

(defun top-level/options ()
  "Creates and returns the options for the top-level command"
  (list
   (clingon:make-option
    :counter
    :description "verbosity level"
    :short-name #\v
    :long-name "verbose"
    :key :verbose)
   (clingon:make-option
    :string
    :description "user to greet"
    :short-name #\u
    :long-name "user"
    :initial-value "stranger"
    :env-vars '("USER")
    :key :user)))

Let’s break things down a bit and explain what we just did.

We’ve defined two options – one of :COUNTER kind and another one, which is of :STRING kind. Each option specifies a short and long name, along with a description of what the option is meant for.

Another important thing we did is to specify a :KEY for our options. This is the key which we will later use in order to get the value associated with our option, when we use CLINGON:GETOPT.

And we have also defined that our --user option can be initialized via environment variables. We can specify multiple environment variables, if we need to, and the first one that resolves to something will be used as the initial value for the option.

If none of the environment variables are defined, the option will be initialized with the value specified by the :INITIAL-VALUE initarg.

Before we move to the next section of this guide we will update the definition of our TOP-LEVEL/COMMAND function, so that we include our options.

(defun top-level/command ()
  "Creates and returns the top-level command"
  (clingon:make-command
   :name "clingon-intro"
   ...
   :usage "[-v] [-u <USER>]"      ;; <- new code
   :options (top-level/options))) ;; <- new code

Defining a handler

A handler in clingon is a function, which accepts an instance of CLINGON:COMMAND and is responsible for performing some work.

The single argument a handler receives will be used to inspect the values of parsed options and any free arguments that were provided on the command-line.

A command may or may not specify a handler. Some commands may be used purely as namespaces for other sub-commands, and it might make no sense to have a handler for such commands. In other situations you may still want to provide a handler for the parent commands.

Let’s define the handler for our top-level command.

(defun top-level/handler (cmd)
  "The top-level handler"
  (let ((args (clingon:command-arguments cmd))
        (user (clingon:getopt cmd :user))
        (verbose (clingon:getopt cmd :verbose)))
    (format t "Hello, ~A!~%" user)
    (format t "The current verbosity level is set to ~A~%" verbose)
    (format t "You have provided ~A arguments~%" (length args))
    (format t "Bye.~%")))

We are introducing a couple of new functions, which we haven’t described before.

We are using CLINGON:COMMAND-ARGUMENTS, which will give us the free arguments we’ve provided to our command, when we invoke it on the command-line.

We also use the CLINGON:GETOPT function to lookup the values associated with our options. Remember the :KEY initarg we’ve used in CLINGON:MAKE-OPTION when defining our options?

And we will again update our TOP-LEVEL/COMMAND definition, this time with our handler included.

(defun top-level/command ()
  "Creates and returns the top-level command"
  (clingon:make-command
   :name "clingon-intro"
   ...
   :handler #'top-level/handler)) ;; <- new code

At this point we are basically done with our simple application. But before we move to the point where build our binary and start playing with it on the command-line we can test things out on the REPL, just to make sure everything works as expected.

Testing things out on the REPL

Create a new instance of our command and bind it to some variable.

INTRO> (defparameter *app* (top-level/command))
*APP*

Inspecting the returned instance would give you something like this.

#<CLINGON.COMMAND:COMMAND {1004648293}>
--------------------
Class: #<STANDARD-CLASS CLINGON.COMMAND:COMMAND>
--------------------
 Group slots by inheritance [ ]
 Sort slots alphabetically  [X]

All Slots:
[ ]  ARGS-TO-PARSE    = NIL
[ ]  ARGUMENTS        = NIL
[ ]  AUTHORS          = ("John Doe <[email protected]>")
[ ]  CONTEXT          = #<HASH-TABLE :TEST EQUAL :COUNT 0 {1004648433}>
[ ]  DESCRIPTION      = "my first clingon cli app"
[ ]  EXAMPLES         = NIL
[ ]  HANDLER          = #<FUNCTION TOP-LEVEL/HANDLER>
[ ]  LICENSE          = "BSD 2-Clause"
[ ]  LONG-DESCRIPTION = NIL
[ ]  NAME             = "clingon-intro"
[ ]  OPTIONS          = (#<CLINGON.OPTIONS:OPTION-BOOLEAN-TRUE short=NIL long=bash-completions> #<CLINGON.OPTIONS:OPTION-BOOLEAN-TRUE short=NIL long=version> #<CLINGON.OPTIONS:OPTION-BOOLEAN-TRUE short=NIL long=help> #<CLINGON.OPTIONS:OPTION-COUNTER short=v long=verbose> #<CLINGON.OPTIONS::OPTION-STRING short=u long=user>)
[ ]  PARENT           = NIL
[ ]  SUB-COMMANDS     = NIL
[ ]  USAGE            = "[-v] [-u <USER>]"
[ ]  VERSION          = "0.1.0"

[set value]  [make unbound]

You might also notice that besides the options we’ve defined ourselves, there are few additional options, that we haven’t defined at all.

These options are automatically added by clingon itself for each new command and provide flags for --help, --version and --bash-completions for you automatically, so you don’t have to deal with them manually.

Before we dive into testing out our application, first we will check that we have a correct help information for our command.

INTRO> (clingon:print-usage *app* t)
NAME:
  clingon-intro - my first clingon cli app

USAGE:
  clingon-intro [-v] [-u <USER>]

OPTIONS:
      --help              display usage information and exit
      --version           display version and exit
  -u, --user <VALUE>      user to greet [default: stranger] [env: $USER]
  -v, --verbose           verbosity level [default: 0]

AUTHORS:
  John Doe <[email protected]>

LICENSE:
  BSD 2-Clause

NIL

This help information will make it easier for our users, when they need to use it. And that is automatically handled for you, so you don’t have to manually maintain an up-to-date usage information, each time you introduce a new option.

Time to test out our application on the REPL. In order to test things out you can use the CLINGON:PARSE-COMMAND-LINE function by passing it an instance of your command, along with any arguments that need to be parsed. Let’s try it out without any command-line arguments.

INTRO> (clingon:parse-command-line *app* nil)
#<CLINGON.COMMAND:COMMAND name=clingon-intro options=5 sub-commands=0>

The CLINGON:PARSE-COMMAND-LINE function will (as the name suggests) parse the given arguments against the options associated with our command. Finally it will return an instance of CLINGON:COMMAND.

In our simple CLI application, that would be the same instance as our *APP*, but things look differently when we have sub-commands.

When we start adding new sub-commands, the result of CLINGON:PARSE-COMMAND-LINE will be different based on the arguments it needs to parse. That means that if our input matches a sub-command you will receive an instance of the sub-command that matched the given arguments.

Internally the clingon system maintains a tree data structure, describing the relationships between commands. This allows a command to be related to some other command, and this is how the command and sub-commands support is implemented in clingon.

Each command in clingon is associated with a context. The context or environment provides the options and their values with respect to the command itself. This means that a parent command and a sub-command may have exactly the same set of options defined, but they will reside in different contexts. Depending on how you use it, sub-commands may shadow a parent command option, but it also means that a sub-command can refer to an option defined in a global command.

The context of a command in clingon is available via the CLINGON:COMMAND-CONTEXT accessor. We will use the context in order to lookup our options and the values associated with them.

The function that operates on command’s context and retrieves values from it is called CLINGON:GETOPT.

Let’s see what we’ve got for our options.

INTRO> (let ((c (clingon:parse-command-line *app* nil)))
         (clingon:getopt c :user))
"dnaeon"
T

The CLINGON:GETOPT function returns multiple values – first one specifies the value of the option, if it had any, the second one indicates whether or not that option has been set at all on the command-line, and the third value is the command which provided the value for the option, if set.

If you need to simply test things out and tell whether an option has been set at all you can use the CLINGON:OPT-IS-SET-P function instead.

Let’s try it out with a different input.

INTRO> (let ((c (clingon:parse-command-line *app* (list "-vvv" "--user" "foo"))))
         (format t "Verbose is ~A~%" (clingon:getopt c :verbose))
         (format t "User is ~A~%" (clingon:getopt c :user)))
Verbose is 3
User is foo

Something else, which is important to mention here. The default precedence list for options is:

• The value provided by the :INITIAL-VALUE initarg
• The value of the first environment variable, which successfully resolved, provided by the :ENV-VARS initarg
• The value provided on the command-line when invoking the application.

Play with it using different command-line arguments. If you specify invalid or unknown options clingon will signal a condition and provide you a few recovery options. For example, if you specify an invalid flag like this:

INTRO> (clingon:parse-command-line *app* (list "--invalid-flag"))

We will be dropped into the debugger and be provided with restarts we can choose from, e.g.

Unknown option --invalid-flag of kind LONG
   [Condition of type CLINGON.CONDITIONS:UNKNOWN-OPTION]

Restarts:
 0: [DISCARD-OPTION] Discard the unknown option
 1: [TREAT-AS-ARGUMENT] Treat the unknown option as a free argument
 2: [SUPPLY-NEW-VALUE] Supply a new value to be parsed
 3: [RETRY] Retry SLY mREPL evaluation request.
 4: [ABORT] Return to sly-db level 1.
 5: [RETRY] Retry SLY mREPL evaluation request.
 --more--
...

This is similar to the way other Common Lisp options parsing systems behave such as adopt and unix-opts.

Also worth mentioning again here is that CLINGON:PARSE-COMMAND-LINE is meant to be used within the REPL, and not called directly by handlers.

Adding a sub-command

Sub-commands are no different than regular commands, and in fact are created exactly the way we did it for our top-level command.

(defun shout/handler (cmd)
  "The handler for the `shout' command"
  (let ((args (mapcar #'string-upcase (clingon:command-arguments cmd)))
        (user (clingon:getopt cmd :user))) ;; <- a global option
    (format t "HEY, ~A!~%" user)
    (format t "~A!~%" (clingon:join-list args #\Space))))

(defun shout/command ()
  "Returns a command which SHOUTS back anything we write on the command-line"
  (clingon:make-command
   :name "shout"
   :description "shouts back anything you write"
   :usage "[options] [arguments ...]"
   :handler #'shout/handler))

And now, we will wire up our sub-command making it part of the top-level command we have so far.

(defun top-level/command ()
  "Creates and returns the top-level command"
  (clingon:make-command
   :name "clingon-intro"
   ...
   :sub-commands (list (shout/command)))) ;; <- new code

You should also notice here that within the SHOUT/HANDLER we are actually referencing an option, which is defined somewhere else. This option is actually defined on our top-level command, but thanks’s to the automatic management of relationships that clingon provides we can now refer to global options as well.

Let’s move on to the final section of this guide, where we will create a system definition for our application and build it.

Packaging it up

One final piece which remains to be added to our code is to provide an entrypoint for our application, so let’s do it now.

(defun main ()
  (let ((app (top-level/command)))
    (clingon:run app)))

This is the entrypoint which will be used when we invoke our application on the command-line, which we’ll set in our ASDF definition.

And here’s a simple system definition for the application we’ve developed so far.

(defpackage :clingon-intro-system
  (:use :cl :asdf))
(in-package :clingon-intro-system)

(defsystem "clingon.intro"
  :name "clingon.intro"
  :long-name "clingon.intro"
  :description "An introduction to the clingon system"
  :version "0.1.0"
  :author "John Doe <[email protected]>"
  :license "BSD 2-Clause"
  :depends-on (:clingon)
  :components ((:module "intro"
                :pathname #P"examples/intro/"
                :components ((:file "intro"))))
  :build-operation "program-op"
  :build-pathname "clingon-intro"
  :entry-point "clingon.intro:main")

Now we can build our application and start using it on the command-line.

sbcl --eval '(ql:quickload :clingon.intro)' \
     --eval '(asdf:make :clingon.intro)' \
     --eval '(quit)'

This will produce a new binary called clingon-intro in the directory of the clingon.intro system.

This approach uses the ASDF program-op operation in combination with :entry-point and :build-pathname in order to produce the resulting binary.

If you want to build your apps using buildapp, please check the Buildapp section from this document.

Testing it out on the command-line

Time to check things up on the command-line.

$ ./clingon-intro --help
NAME:
  clingon-intro - my first clingon cli app

USAGE:
  clingon-intro [-v] [-u <USER>]

OPTIONS:
      --help              display usage information and exit
      --version           display version and exit
  -u, --user <VALUE>      user to greet [default: stranger] [env: $USER]
  -v, --verbose           verbosity level [default: 0]

COMMANDS:
  shout  shouts back anything you write

AUTHORS:
  John Doe <[email protected]>

LICENSE:
  BSD 2-Clause

Let’s try out our commands.

$ ./clingon-intro -vvv --user Lisper
Hello, Lisper!
The current verbosity level is set to 3
You have provided 0 arguments
Bye.

And let’s try our sub-command as well.

$ ./clingon-intro --user stranger shout why are yelling at me?
HEY, stranger!
WHY ARE YELLING AT ME?!

You can find the full code we’ve developed in this guide in the clingon/examples directory of the repo.

Exiting

When a command needs to exit with a given status code you can use the CLINGON:EXIT function.

Handling SIGINT (CTRL-C) signals

clingon by default will provide a handler for SIGINT signals, which when detected will cause the application to immediately exit with status code 130.

If your commands need to provide some cleanup logic as part of their job, e.g. close out all open files, TCP session, etc., you could wrap your clingon command handlers in UNWIND-PROTECT to make sure that your cleanup tasks are always executed.

However, using UNWIND-PROTECT may not be appropriate in all cases, since the cleanup forms will always be executed, which may or may not be what you need.

For example if you are developing a clingon application, which populates a database in a transaction you would want to use UNWIND-PROTECT, but only for releasing the database connection itself.

If the application is interrupted while it inserts or updates records, what you want to do is to rollback the transaction as well, so your database is left in a consistent state.

In those situations you would want to use the WITH-USER-ABORT system, so that your clingon command can detect the SIGINT signal and act upon it, e.g. taking care of rolling back the transaction.

Generating Documentation

clingon can generate documentation for your application by using the CLINGON:PRINT-DOCUMENTATION generic function.

Currently the documentation generator supports only the Markdown format, but other formats can be developed as separate extensions to clingon.

Here’s how you can generate the Markdown documentation for the clingon-demo application from the REPL.

CL-USER> (ql:quickload :clingon.demo)
CL-USER> (in-package :clingon.demo)
DEMO> (with-open-file (out #P"clingon-demo.md" :direction :output)
        (clingon:print-documentation :markdown (top-level/command) out))

You can also create a simple command, which can be added to your clingon apps and have it generate the documentation for you, e.g.

(defun print-doc/command ()
  "Returns a command which will print the app's documentation"
  (clingon:make-command
   :name "print-doc"
   :description "print the documentation"
   :usage ""
   :handler (lambda (cmd)
              ;; Print the documentation starting from the parent
              ;; command, so we can traverse all sub-commands in the
              ;; tree.
              (clingon:print-documentation :markdown (clingon:command-parent cmd) t))))

Above command can be wired up anywhere in your application.

Make sure to also check the clingon-demo app, which provides a print-doc sub-command, which operates on the top-level command and generates the documentation for all sub-commands.

You can also find the generated documentation for the clingon-demo app in the docs/ directory of the clingon repo.

Generate tree representation of your commands in Dot

Using CLINGON:PRINT-DOCUMENTATION you can also generate the tree representation of your commands in Dot format.

Make sure to check the clingon.demo system and the provided clingon-demo app, which provides an example command for generating the Dot representation.

The example below shows the generation of the Dot representation for the clingon-demo command.

> clingon-demo dot
digraph G {
  node [color=lightblue fillcolor=lightblue fontcolor=black shape=record style="filled, rounded"];
  "clingon-demo" -> "greet";
  "clingon-demo" -> "logging";
  "logging" -> "enable";
  "logging" -> "disable";
  "clingon-demo" -> "math";
  "clingon-demo" -> "echo";
  "clingon-demo" -> "engine";
  "clingon-demo" -> "print-doc";
  "clingon-demo" -> "sleep";
  "clingon-demo" -> "zsh-completion";
  "clingon-demo" -> "dot";
}

We can generate the resulting graph using graphviz.

> clingon-demo dot > clingon-demo.dot
> dot -Tpng clingon-demo.dot > clingon-demo-tree.png

This is what the resulting tree looks like.

Command Hooks

clingon allows you to associate pre and post hooks with a command.

The pre and post hooks are functions which will be invoked before and after the respective command handler is executed. They are useful in cases when you need to set up or tear things down before executing the command’s handler.

An example of a pre-hook might be to configure the logging level of your application based on the value of a global flag. A post-hook might be responsible for shutting down any active connections, etc.

The pre-hook and post-hook functions accept a single argument, which is an instance of CLINGON:COMMAND. That way the hooks can examine the command’s context and lookup any flags or options.

Hooks are also hierachical in the sense that they will be executed based on the command’s lineage.

Consider the following example, where we have a CLI app with three commands.

main -> foo -> bar

In above example the bar command is a sub-command of foo, which in turn is a sub-command of main. Also, consider that we have added pre- and post-hooks to each command.

If a user executed the following on the command-line:

$ main foo bar

Based on the above command-line clingon would do the following:

• Execute any pre-hook functions starting from the least-specific up to the most-specific node from the commands’ lineage
• Execute the command’s handler
• Execute any post-hook functions starting from the most-specific down to the least-specific node from the command’s lineage

In above example that would be:

> main (pre-hook)
>> foo (pre-hook)
>>> bar (pre-hook)
>>>> bar (handler)
>>> bar (post-hook)
>> foo (post-hook)
> main (post-hook)

Associating hooks with commands is done during instantiation of a command. The following example creates a new command with a pre-hook and post-hook.

(defun foo/pre-hook (cmd)
  "The pre-hook for `foo' command"
  (declare (ignore cmd))
  (format t "foo pre-hook has been invoked~&"))

(defun foo/post-hook (cmd)
  "The post-hook for `foo' command"
  (declare (ignore cmd))
  (format t "foo post-hook has been invoked~&"))

(defun foo/handler (cmd)
  (declare (ignore cmd))
  (format t "foo handler has been invoked~&"))

(defun foo/command ()
  "Returns the `foo' command"
  (clingon:make-command
   :name "foo"
   :description "the foo command"
   :authors '("John Doe <[email protected]>")
   :handler #'foo/handler
   :pre-hook #'foo/pre-hook
   :post-hook #'foo/post-hook
   :options nil
   :sub-commands nil))

If we have executed above command we would see the following output.

foo pre-hook has been invoked
foo handler has been invoked
foo post-hook has been invoked

Custom Errors

The CLINGON:BASE-ERROR condition may be used as the base for user-defined conditions.

The CLINGON:RUN method will invoke CLINGON:HANDLE-ERROR for conditions which sub-class CLINGON:BASE-ERROR. The implementation of CLINGON:HANDLE-ERROR allows the user to customize the way errors are being reported and handled.

The following example creates a new custom condition.

(in-package :cl-user)
(defpackage :my.clingon.app
  (:use :cl)
  (:import-from :clingon)
  (:export :my-app-error))
(in-package :my.clingon.app)

(define-condition my-app-error (clingon:base-error)
  ((message
    :initarg :message
    :initform (error "Must specify message")
    :reader my-app-error-message))
  (:documentation "My custom app error condition"))

(defmethod clingon:handle-error ((err my-app-error))
  (let ((message (my-app-error-message err)))
    (format *error-output* "Oops, an error occurred: ~A~%" message)))

You can now use the MY-APP-ERROR condition anywhere in your command handlers and signal it. When this condition is signalled clingon will invoke the CLINGON:HANDLE-ERROR generic function for your condition.

Customizing the parsing logic

The default implementation of CLINGON:RUN provides error handling for the most common user-related errors, such as handling of missing arguments, invalid options/flags, catching of SIGINT signals, etc.

Internally CLINGON:RUN relies on CLINGON:PARSE-COMMAND-LINE for the actual parsing. In order to provide custom logic during parsing, users may provide a different implementation of either CLINGON:RUN and/or CLINGON:PARSE-COMMAND-LINE by subclassing the CLINGON:COMMAND class.

An alternative approach, which doesn’t need a subclass of CLINGON:COMMAND is to provide AROUND methods for CLINGON:RUN.

For instance, the following code will treat unknown options as free arguments, while still using the default implementation of CLINGON:RUN.

(defmethod clingon:parse-command-line :around ((command clingon:command) arguments)
  "Treats unknown options as free arguments"
  (handler-bind ((clingon:unknown-option
                   (lambda (c)
                     (clingon:treat-as-argument c))))
    (call-next-method)))

See this issue for more examples and additional discussion on this topic.

Options

The clingon system supports various kinds of options, each of which is meant to serve a specific purpose.

Each builtin option can be initialized via environment variables, and new mechanisms for initializing options can be developed, if needed.

Options are created via the single CLINGON:MAKE-OPTION interface.

The supported option kinds include:

• counter
• integer
• string
• boolean
• boolean/true
• boolean/false
• flag
• choice
• enum
• list
• list/integer
• filepath
• list/filepath
• switch
• etc.

Counters Options

A counter is an option kind, which increments every time it is set on the command-line.

A good example for counter options is to provide a flag, which increases the verbosity level, depending on the number of times the flag was provided, similar to the way ssh(1) does it, e.g.

ssh -vvv user@host

Here’s an example of creating a counter option.

(clingon:make-option
 :counter
 :short-name #\v
 :long-name "verbose"
 :description "how noisy we want to be"
 :key :verbose)

The default step for counters is set to 1, but you can change that, if needed.

(clingon:make-option
 :counter
 :short-name #\v
 :long-name "verbose"
 :description "how noisy we want to be"
 :step 42
 :key :verbose)

Boolean Options

The following boolean option kinds are supported by clingon.

The :boolean kind is an option which expects an argument, which represents a boolean value.

Arguments true and 1 map to T in Lisp, anything else is considered a falsey value and maps to NIL.

(clingon:make-option
 :boolean
 :description "my boolean"
 :short-name #\b
 :long-name "my-boolean"
 :key :boolean)

This creates an option -b, --my-boolean <VALUE>, which can be provided on the command-line, where <VALUE> should be true or 1 for truthy values, and anything else maps to NIL.

The :boolean/true option kind creates a flag, which always returns T.

The :boolean/false option kind creates a flag, which always returns NIL.

The :flag option kind is an alias for :boolean/true.

Integer Options

Here’s an example of creating an option, which expects an integer argument.

(clingon:make-option
 :integer
 :description "my integer opt"
 :short-name #\i
 :long-name "int"
 :key :my-int
 :initial-value 42)

Choice Options

choice options are useful when you have to limit the arguments provided on the command-line to a specific set of values.

For example:

(clingon:make-option
 :choice
 :description "log level"
 :short-name #\l
 :long-name "log-level"
 :key :choice
 :items '("info" "warn" "error" "debug"))

With this option defined, you can now set the logging level only to info, warn, error or debug, e.g.

-l, --log-level [info|warn|error|debug]

Enum Options

Enum options are similar to the choice options, but instead of returning the value itself they can be mapped to something else.

For example:

(clingon:make-option
 :enum
 :description "enum option"
 :short-name #\e
 :long-name "my-enum"
 :key :enum
 :items '(("one" . 1)
          ("two" . 2)
          ("three" . 3)))

If a user specifies --my-enum=one on the command-line the option will be have the value 1 associated with it, when being looked up via CLINGON:GETOPT.

The values you associate with the enum variant, can be any object.

This is one of the options being used by the clingon-demo application, which maps user input to Lisp functions, in order to perform some basic math operations.

(clingon:make-option
 :enum
 :description "operation to perform"
 :short-name #\o
 :long-name "operation"
 :required t
 :items `(("add" . ,#'+)
          ("sub" . ,#'-)
          ("mul" . ,#'*)
          ("div" . ,#'/))
 :key :math/operation)

List / Accumulator Options

The :list option kind accumulates each argument it is given on the command-line into a list.

For example:

(clingon:make-option
 :list
 :description "files to process"
 :short-name #\f
 :long-name "file"
 :key :files)

If you invoke an application, which uses a similar option like the one above using the following command-line arguments:

$ my-app --file foo --file bar --file baz

When you retrieve the value associated with your option, you will get a list of all the files specified on the command-line, e.g.

(clingon:getopt cmd :files) ;; => '("foo" "bar" "baz")

A similar option exists for integer values using the :list/integer option, e.g.

(clingon:make-option
 :list/integer
 :description "list of integers"
 :short-name #\l
 :long-name "int"
 :key :integers)

Switch Options

:SWITCH options are a variation of :BOOLEAN options with an associated list of known states that can turn a switch on or off.

Here is an example of a :SWITCH option.

(clingon:make-option
 :switch
 :description "my switch option"
 :short-name #\s
 :long-name "state"
 :key :switch)

The default states for a switch to be considered as on are:

• on, yes, true, enable and 1

The default states considered to turn the switch off are:

• off, no, false, disable and 0

You can customize the list of on and off states by specifying them using the :ON-STATES and :OFF-STATES initargs, e.g.

(clingon:make-option
 :switch
 :description "engine switch option"
 :short-name #\s
 :long-name "state"
 :on-states '("start")
 :off-states '("stop")
 :key :engine)

These sample command-line arguments will turn a switch on and off.

my-app --engine=start --engine=stop

The final value of the :engine option will be NIL in the above example.

Persistent Options

An option may be marked as persistent. A persistent option is such an option, which will be propagated from a parent command to all sub-commands associated with it.

This is useful when you need to provide the same option across sub-commands.

The following example creates one top-level command (demo in the example), which has two sub-commands (foo and bar commands). The foo command has a single sub-command, qux in the following example.

The top-level command has a single option (persistent-opt in the example), which is marked as persistent.

(defun qux/command ()
  "Returns the `qux' command"
  (clingon:make-command
   :name "qux"
   :description "the qux command"
   :handler (lambda (cmd)
              (declare (ignore cmd))
              (format t "qux has been invoked"))))

(defun foo/command ()
  "Returns the `foo' command"
  (clingon:make-command
   :name "foo"
   :description "the foo command"
   :sub-commands (list (qux/command))
   :handler (lambda (cmd)
              (declare (ignore cmd))
              (format t "foo has been invoked"))))

(defun bar/command ()
  "Returns the `bar' command"
  (clingon:make-command
   :name "bar"
   :description "the bar command"
   :handler (lambda (cmd)
              (declare (ignore cmd))
              (format t "bar has been invoked"))))

(defun top-level/command ()
  "Returns the top-level command"
  (clingon:make-command
   :name "demo"
   :description "the demo app"
   :options (list
             (clingon:make-option
              :string
              :long-name "persistent-opt"
              :description "an example persistent option"
              :persistent t
              :key :persistent-opt))
   :sub-commands (list
                  (foo/command)
                  (bar/command))))

Since the option is marked as persistent and is associated with the top-level command, it will be inherited by all sub-commands.

Generic Functions Operating on Options

If the existing options provided by clingon are not enough for you, and you need something a bit more specific for your use case, then you can always implement a new option kind.

The following generic functions operate on options and are exported by the clingon system.

• CLINGON:INITILIAZE-OPTION
• CLINGON:FINALIZE-OPTION
• CLINGON:DERIVE-OPTION-VALUE
• CLINGON:OPTION-USAGE-DETAILS
• CLINGON:OPTION-DESCRIPTION-DETAILS
• CLINGON:MAKE-OPTION

New option kinds should inherit from the CLINGON:OPTION class, which implements all of the above generic functions. If you need to customize the behaviour of your new option, you can still override the default implementations.

CLINGON:INITIALIZE-OPTION

The CLINGON:INITIALIZE-OPTION as the name suggests is being used to initialize an option.

The default implementation of this generic function supports initialization from environment variables, but implementors can choose to support other initialization methods, e.g. be able to initialize an option from a key/value store like Redis, Consul or etcd for example.

CLINGON:FINALIZE-OPTION

The CLINGON:FINALIZE-OPTION generic function is called after all command-line arguments have been processed and values for them have been derived already.

CLINGON:FINALIZE-OPTION is meant to finalize the option’s value, e.g. transform it to another object, if needed.

For example the :BOOLEAN option kind transforms user-provided input like true, false, 1 and 0 into their respective Lisp counterparts like T and NIL.

Another example where you might want to customize the behaviour of CLINGON:FINALIZE-OPTION is to convert a string option provided on the command-line, which represents a database connection string into an actual session object for the database.

The default implementation of this generic function simply returns the already set value, e.g. calls #'IDENTITY on the last derived value.

CLINGON:DERIVE-OPTION-VALUE

The CLINGON:DERIVE-OPTION-VALUE is called whenever an option is provided on the command-line.

If that option accepts an argument, it will be passed the respective value from the command-line, otherwise it will be called with a NIL argument.

Responsibility of the option is to derive a value from the given input and return it to the caller. The returned value will be set by the parser and later on it will be used to produce a final value, by calling the CLINGON:FINALIZE-OPTION generic function.

Different kinds of options implement this one different – for example the :LIST option kind accumulates each given argument, while others ignore any previously derived values and return the last provided argument.

The :ENUM option kind for example will derive a value from a pre-defined list of allowed values.

If an option fails to derive a value (e.g. invalid value has been provided) the implementation of this generic function should signal a CLINGON:OPTION-DERIVE-ERROR condition, so that clingon can provide appropriate restarts.

CLINGON:OPTION-USAGE-DETAILS

This generic function is used to provide a pretty-printed usage format for the given option. It will be used when printing usage information on the command-line for the respective commands.

CLINGON:OPTION-DESCRIPTION-DETAILS

This generic function is meant to enrich the description of the option by providing as much details as possible for the given option, e.g. listing the available values that an option can accept.

CLINGON:MAKE-OPTION

The CLINGON:MAKE-OPTION generic function is the primary way for creating new options. Implementors of new option kinds should simply provide an implementation of this generic function, along with the respective option kind.

Additional option kinds may be implemented as separate sub-systems, but still follow the same principle by providing a single and consistent interface for option creation.

Developing New Options

This section contains short guides explaining how to develop new options for clingon.

Developing an Email Option

The option which we’ll develop in this section will be used for specifying email addresses.

Start up your Lisp REPL session and do let’s some work. Load the :clingon and :cl-ppcre systems, since we will need them.

CL-USER> (ql:quickload :clingon)
CL-USER> (ql:quickload :cl-ppcre)

We will first create a new package for our extension and import the symbols we will need from the :clingon and :cl-ppcre systems.

(defpackage :clingon.extensions/option-email
  (:use :cl)
  (:import-from
   :cl-ppcre
   :scan)
  (:import-from
   :clingon
   :option
   :initialize-option
   :derive-option-value
   :make-option
   :option-value
   :option-derive-error)
  (:export
   :option-email))
(in-package :clingon.extensions/option-email)

Then lets define the class, which will represent an email address option.

(defclass option-email (option)
  ((pattern
    :initarg :pattern
    :initform "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"
    :reader option-email-pattern
    :documentation "Pattern used to match for valid email addresses"))
  (:default-initargs
   :parameter "EMAIL")
  (:documentation "An option used to represent an email address"))

Now we will implement CLINGON:INITIALIZE-OPTION for our new option. We will keep the default initialization logic as-is, but also add an additional step to validate the email address, if we have any initial value at all.

(defmethod initialize-option ((option option-email) &key)
  "Initializes our new email address option"
  ;; Make sure to invoke our parent initialization method first, so
  ;; various things like setting up initial value from environment
  ;; variables can still be applied.
  (call-next-method)

  ;; If we don't have any value set, there's nothing else to
  ;; initialize further here.
  (unless (option-value option)
    (return-from initialize-option))

  ;; If we get to this point, that means we've got some initial value,
  ;; which is either set as a default, or via environment
  ;; variables. Next thing we need to do is make sure we've got a good
  ;; initial value, so let's derive a value from it.
  (let ((current (option-value option)))
    (setf (option-value option)
          (derive-option-value option current))))

Next we will implement CLINGON:DERIVE-OPTION-VALUE for our new option kind.

(defmethod derive-option-value ((option option-email) arg &key)
  "Derives a new value based on the given argument.
   If the given ARG represents a valid email address according to the
   pattern we know of we consider this as a valid email address."
  (unless (scan (option-email-pattern option) arg)
    (error 'option-derive-error :reason (format nil "~A is not a valid email address" arg)))
  arg)

Finally, lets register our new option as a valid kind by implemeting the CLINGON:MAKE-OPTION generic function.

(defmethod make-option ((kind (eql :email)) &rest rest)
  (apply #'make-instance 'option-email rest))

We can test things out now. Go back to your REPL and try these expressions out. First we make a new instance of our new option.

(defparameter *opt*
  (make-option :email :short-name #\e :description "email opt" :key :email))

And now, lets validate a couple of good email addresses.

EXTENSIONS/OPTION-EMAIL> (derive-option-value *opt* "[email protected]")
"[email protected]"
EXTENSIONS/OPTION-EMAIL> (derive-option-value *opt* "[email protected]")
"[email protected]"

If we try deriving a value from a bad email address we will have a condition of type CLINGON:OPTION-DERIVE-ERROR signalled.

EXTENSIONS/OPTION-EMAIL> (derive-option-value opt "bad-email-address-here")
; Debugger entered on #<OPTION-DERIVE-ERROR {1002946463}>
...
bad-email-address-here is not a valid email address
   [Condition of type OPTION-DERIVE-ERROR]

Good, we can catch invalid email addresses as well. Whenever an option fails to derive a new value from a given argument, and we signal CLINGON:OPTION-DERIVE-ERROR condition we can recover by providing new values or discarding them completely, thanks to the Common Lisp Condition System.

Last thing to do is actually package this up as an extension system and register it in Quicklisp. That way everyone else can benefit from the newly developed option.

Shell Completions

clingon provides support for Bash and Zsh shell completions.

Bash Completions

In order to enable the Bash completions for your clingon app, follow these instructions.

APP=app-name source extras/completions.bash

Make sure to set APP to your correct application name.

The completions.bash script will dynamically provide completions by invoking the clingon app with the --bash-completions flag. This builtin flag when provided on the command-line will return completions for the sub-commands and the available flags.

Zsh Completions

When developing your CLI app with clingon you can provide an additional command, which will take care of generating the Zsh completion script for your users.

The following code can be used in your app and added as a sub-command to your top-level command.

(defun zsh-completion/command ()
  "Returns a command for generating the Zsh completion script"
  (clingon:make-command
   :name "zsh-completion"
   :description "generate the Zsh completion script"
   :usage ""
   :handler (lambda (cmd)
              ;; Use the parent command when generating the completions,
              ;; so that we can traverse all sub-commands in the tree.
              (let ((parent (clingon:command-parent cmd)))
                (clingon:print-documentation :zsh-completions parent t)))))

You can also check out the clingon-demo app for a fully working CLI app with Zsh completions support.

Buildapp

The demo clingon apps from this repo are usually built using ASDF with :build-operation set to program-op and the respective :entry-point and :build-pathname specified in the system definition. See the included clingon.demo.asd and clingon.intro.asd systems for examples.

You can also use buildapp for building the clingon apps.

This command will build the clingon-demo CLI app using buildapp.

$ buildapp \
  --output clingon-demo \
  --asdf-tree ~/quicklisp/dists/quicklisp/software/ \
  --load-system clingon.demo \
  --entry main \
  --eval '(defun main (argv) (let ((app (clingon.demo::top-level/command))) (clingon:run app (rest argv))))'

Another approach to building apps using buildapp is to create a main entrypoint in your application, similarly to the way you create one for use with ASDF and :entry-point. This function can be used as an entrypoint for buildapp apps.

(defun main (argv)
  "The main entrypoint for buildapp apps"
  (let ((app (top-level/command)))
    (clingon:run app (rest argv))))

Then build your app with this command.

$ buildapp \
  --output my-app-name \
  --asdf-tree ~/quicklisp/dists/quicklisp/software/ \
  --load-system my-system-name \
  --entry my-system-name:main

Ideas For Future Improvements

Additional Documentation Generators

As of now clingon supports generating documentation only in Markdown format.

Would be nice to have additional documentation generators, e.g. man pages, HTML, etc.

Performance Notes

clingon has been developed and tested on a GNU/Linux system using SBCL.

Performance of the resulting binaries with SBCL seem to be good, although I have noticed better performance when the binaries have been produced with Clozure CL. And by better I mean better in terms of binary size and speed (startup + run time).

Although you can enable compression on the image when using SBCL you have to pay the extra price for the startup time.

Here are some additional details. Build the clingon-demo app with SBCL.

$ LISP=sbcl make demo
sbcl --eval '(ql:quickload :clingon.demo)' \
        --eval '(asdf:make :clingon.demo)' \
                --eval '(quit)'
This is SBCL 2.1.7, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
To load "clingon.demo":
  Load 1 ASDF system:
    clingon.demo
; Loading "clingon.demo"
[package clingon.utils]...........................
[package clingon.conditions]......................
[package clingon.options].........................
[package clingon.command].........................
[package clingon].................................
[package clingon.demo]
[undoing binding stack and other enclosing state... done]
[performing final GC... done]
[defragmenting immobile space... (fin,inst,fdefn,code,sym)=1118+969+19070+19610+26536... done]
[saving current Lisp image into /home/dnaeon/Projects/lisp/clingon/clingon-demo:
writing 0 bytes from the read-only space at 0x50000000
writing 736 bytes from the static space at 0x50100000
writing 31391744 bytes from the dynamic space at 0x1000000000
writing 2072576 bytes from the immobile space at 0x50200000
writing 12341248 bytes from the immobile space at 0x52a00000
done]

Now, build it using Clozure CL.

$ LISP=ccl make demo
ccl --eval '(ql:quickload :clingon.demo)' \
        --eval '(asdf:make :clingon.demo)' \
                --eval '(quit)'
To load "clingon.demo":
  Load 1 ASDF system:
    clingon.demo
; Loading "clingon.demo"
[package clingon.utils]...........................
[package clingon.conditions]......................
[package clingon.options].........................
[package clingon.command].........................
[package clingon].................................
[package clingon.demo].

In terms of file size the binaries produced by Clozure CL are smaller.

$ ls -lh clingon-demo*
-rwxr-xr-x 1 dnaeon dnaeon 33M Aug 20 12:56 clingon-demo.ccl
-rwxr-xr-x 1 dnaeon dnaeon 45M Aug 20 12:55 clingon-demo.sbcl

Generating the Markdown documentation for the demo app when using the SBCL executable looks like this.

$ time ./clingon-demo.sbcl print-doc > /dev/null

real    0m0.098s
user    0m0.071s
sys     0m0.027s

And when doing the same thing with the executable produced by Clozure CL we see these results.

$ time ./clingon-demo.ccl print-doc > /dev/null

real    0m0.017s
user    0m0.010s
sys     0m0.007s

Tests

The clingon tests are provided as part of the :clingon.test system.

In order to run the tests you can evaluate the following expressions.

CL-USER> (ql:quickload :clingon.test)
CL-USER> (asdf:test-system :clingon.test)

Or you can run the tests using the run-tests.sh script instead, e.g.

LISP=sbcl ./run-tests.sh

Here’s how to run the tests against SBCL, CCL and ECL for example.

for lisp in sbcl ccl ecl; do
    echo "Running tests using ${lisp} ..."
    LISP=${lisp} make test > ${lisp}-tests.out
done

Docker Images

A few Docker images are available.

Build and run the tests in a container.

docker build -t clingon.test:latest -f Dockerfile.sbcl .
docker run --rm clingon.test:latest

Build and run the clingon-intro application.

docker build -t clingon.intro:latest -f Dockerfile.intro .
docker run --rm clingon.intro:latest

Build and run the clingon.demo application.

docker build -t clingon.demo:latest -f Dockerfile.demo .
docker run --rm clingon.demo:latest

Contributing

clingon is hosted on Github. Please contribute by reporting issues, suggesting features or by sending patches using pull requests.

License

This project is Open Source and licensed under the BSD License.

Authors

• Marin Atanasov Nikolov <[email protected]>