CI

CI/CD toolkit as an Elixir library.

Status

Early alpha. Not tested in any serious production. Expect bugs and breaking changes.

Basic example

Job.run(
  Job.Pipeline.sequence([
    mix("compile --warnings-as-errors"),
    Job.Pipeline.parallel([
      mix("dialyzer"),
      mix("test"),
      mix("format --check-formatted"),
      mix("docs", env: [mix_env: "dev"])
    ])
  ]),
  timeout: :timer.minutes(10)
)
|> report_errors()

defp mix(arg, opts \\ []),
  do: cmd("mix #{arg}", Config.Reader.merge([env: [mix_env: "test"]], opts))

defp cmd(cmd, opts) do
  handler = &IO.write(message(&1, cmd))
  cmd_opts = [handler: handler] ++ Keyword.merge([pty: true], opts)
  OsCmd.action(cmd, cmd_opts)
end

defp message(:starting, cmd), do: "starting #{cmd}\n"
defp message({:output, output}, _cmd), do: output
defp message({:stopped, status}, cmd), do: "#{cmd} stopped with reason #{inspect(status)}"

This is the highest level API, which should fit simpler scenarios. For more involved needs, you can replace Job.Pipeline with imperative actions powered by Job. The example above can be also written as:

Job.run(fn ->
  with {:ok, _output} <- Job.run_action(mix("compile --warnings-as-errors")) do
    [
      mix("dialyzer"),
      mix("test"),
      mix("format --check-formatted"),
      mix("docs", env: [mix_env: "dev"])
    ]
    |> Enum.map(&Job.start_action/1)
    |> Enum.map(&Job.await/1)
    |> interpret_results()
  end
end)

Quick start

1. Make sure the prerequisites are installed:

   • Erlang >= 23
   • Elixir >= 1.11
   • go >= 1.15

2. Add CI as a dependency inside your mix.exs:

   # mix.exs
   
   defp deps do
     [
       {:ci, "~> 0.1.0"},
       # ...
     ]
   end

3. Invoke mix ci.init

4. Invoke mix my_app.ci, replacing my_app with the name of your OTP app.

The generated CI mix task will first compile the project, and then run mix test and mix format --check-formatted in parallel. Note that this task may fail if your project requires additional pre-test tasks, such as Ecto repo setup.

You can optionally write a test that verifies your CI flow. See here for a simple example.

To run these checks on some CI platform (Travis, Circle, GitHub Actions, ...), you need to make sure prerequisites are installed, and then invoke mix deps.get, followed by mix my_app.ci. You can see how this project is configured to test itself on GitHub Actions here.

Alternatively, you can consider creating a separate mix project, which only contains the CI mix task. The benefit of this approach is that you can then move all the steps of the tested project inside the mix task. This can be useful if you e.g. need to perform some preparations before fetching deps (e.g. setup ssh credentials, etc).

Explanation

CI is a collection of small standalone abstractions that can be useful beyond the CI domain:

• OsCmd - Managed execution of OS commands
• Job - Managed execution of potentially failing actions
• Job.Pipeline - a high-level interface for running sequential and parallel pipelines inside a job

Except for the ci.init mix task, no special code exists in the Ci namespace. The generated client CI code merely combines the abstractions listed above to implement the desired CI flow in Elixir. Using a first-class language instead of an ad-hoc proprietary yaml-based DSL leads to the following benefits:

1. Easy to learn

   If the project is developed in Elixir, everyone on the team is already mostly equipped with the necessary knowledge. Obviously you need to study the docs of the provided abstractions, so you can use them properly, but at least this doesn't require learning a completely different syntax, and the knowledge you gather may serve you beyond the CI domain.

2. Flexible

   Write complex loops and branching logic. Declare variables and organize your code in functions for improved readability and/or reusability. Reap all the benefits of BEAM (support for concurrency and fault-tolerance), and its ecosystem.

   Note that despite being imperative first, CI makes it possible to use a full-declarative proprietary DSL. Converting e.g. a yaml or a json into a series of Job.Pipeline actions should be straightforward, and is left as an exercise for the reader :-)

   Being organized as toolkit, rather than a framework, this library leaves a lot of rooms for variations, without requiring a large amount of knobs. You can use only those abstractions that suit your purposes, combining them in whichever way you like. Since most of the code is organized in a porcelain/plumbing way, you can fall back one layer deeper in case you need to make different choices.

3. Local-first

   Easily run the entire flow locally. Test and troubleshoot the problems without pushing commits to trigger an external build.

4. Testable

   Write tests which exercise some critical parts of your CI/CD flow (e.g. a mail is sent on error, or system is deployed after all the checks have passed).

5. Rich tooling

   Use all the tools from the BEAM ecosystem, such as IDE support, debuggers, profilers, ...

Roadmap

• Improve Windows support (help needed)
• Docker support (managing docker images, managed execution of docker containers)
• VCS wrappers (git, ...)
• SCM platform clients (e.g. reacting to GitHub events)

License

MIT