mix_deploy

This module generates scripts which help deploy an Erlang release, handling tasks such as creating initial directory structure, unpacking release files, managing configuration, and starting/stopping. It supports deployment to the local machine, bare-metal servers, or cloud servers using e.g. AWS CodeDeploy.

It supports releases created with Elixir 1.9+ mix release or Distillery.

It assumes that mix_systemd is used to generate a systemd unit file for the application, and shares conventions with it about naming files. See mix_systemd for examples.

Here is a complete example app.

Installation

Add mix_deploy to the list of dependencies in mix.exs:

def deps do
  [
    {:mix_deploy, "~> 0.7"},
  ]
end

Example

A straightforward way to deploy an app is on a virtual private server at e.g. Digital Ocean, building and deploying on the same machine. Check out the code on the server, run mix compile to build, run mix test, then run mix release to generate a release. You then use the scripts generated by mix_release to set up the runtime environment, deploy the release to the target dir, and run it supervised by systemd.

Configure the app

Follow the Phoenix config process for deployment and releases.

The app should read runtime configuration such as the database connection from environment variables in config/runtime.exs. Generate a production secret with mix phx.gen.secret.

Create a file with these environment vars and put it in config/environment, e.g.:

DATABASE_URL="ecto://foo:Sekr!t@localhost/foo"
SECRET_KEY_BASE="VXR6/fViPssuoAyqmr0SvAYBIaMrtiZLaQCn1TfB5NXaOzssHxtegfF+yM+/Senv"

Add the config/environment file to .gitignore so that any secrets do not get checked into git.

Configure mix_deploy and mix_systemd

Configure mix_deploy and mix_systemd in config/prod.exs.

mix_systemd generates a systemd unit file which loads the configuration for the app. On startup, it creates the specified directories for the app in the standard locations.

config :mix_systemd,
  env_files: [
    # Read environment vars from file /srv/foo/etc/environment if it exists
    ["-", :deploy_dir, "/etc/environment"],
    # Read environment vars from file /etc/foo/environment if it exists
    ["-", :configuration_dir, "/environment"]
  ],
  # Set individual env vars
  env_vars: [
    "PHX_SERVER=true"
    "PORT=8080",
  ],
  # Create standard config dirs
  dirs: [
    # /var/cache/foo
    :cache,
    # /etc/foo
    :configuration,
    # /var/log/foo
    :logs,
    # /run/foo
    :runtime,
    # /var/lib/foo
    :state,
    # /var/tmp/foo
    :tmp
  ],
  # Run app under this OS user, default is the app name
  app_user: "app",
  app_group: "app"

mix_deploy generates scripts to initialize the system and deploy it.

config :mix_deploy,
  app_user: "app",
  app_group: "app"
  # Copy config/environment to /etc/foo/environment
  copy_files: [
    %{
      src: "config/environment",
      dst: [:configuration_dir, "/environment"],
      user: "$DEPLOY_USER",
      group: "$APP_GROUP",
      mode: "640"
    },
  ],
  # Generate these scripts in bin
  templates: [
    "init-local",
    "create-users",
    "create-dirs",
    "copy-files",
    "enable",
    "release",
    "restart",
    "rollback",
    "start",
    "stop",
  ]

Initialize mix_systemd and mix_deploy and generate files

mix_systemd and mix_deploy generate output files from templates. Run the following to copy the templates into your project. The templating process most common needs via configuration, but you can also check them into your project and make local modifications to handle special needs.

mix systemd.init
mix deploy.init

Generate output files:

# Create systemd unit file for app under _build/prod/systemd
MIX_ENV=prod mix systemd.generate

# Create deploy scripts project `bin` dir
MIX_ENV=prod mix deploy.generate
chmod +x bin/*

Set up the system

Run the scripts to set up the operating system for the deployment. This creates the app OS user, directory structure under /srv/foo, and the systemd unit file which supervises the app.

deploy-init-local is a convenience script which runs other scripts to set up the system:

sudo bin/deploy-init-local

It does the following:

# Create users to run the app
bin/deploy-create-users

# Create deploy dirs under /srv/foo
bin/deploy-create-dirs

# Copy scripts used at runtime by the systemd unit
cp bin/* /srv/foo/bin

# Copy files and enable systemd unit
bin/deploy-copy-files
bin/deploy-enable

Build the Elixir release

Create the Elixir (Erlang) release. This is a tar file containing the app, the libraries it depends on, and the scripts to manage it.

MIX_ENV=prod mix release

Deploy the release to the local machine:

# Extract release to target directory and make it current
sudo bin/deploy-release

# Restart the systemd unit
sudo bin/deploy-restart

You can roll back the release with the following:

bin/deploy-rollback
sudo bin/deploy-restart

Add an alias to mix.exs, and you can do the deploy by running mix deploy.

def project do
  [
    preferred_cli_env: [
      deploy: :prod
    ]
  ]
end

defp aliases do
  [
    deploy: [
      "release --overwrite",
      "cmd sudo bin/deploy-release",
      "cmd sudo bin/deploy-restart"
    ]
  ]
end

Try it out

Your app should now be running:

curl -v http://localhost:8080/

If it is not, have a look at the logs.

systemctl status foo
journalctl -u foo

If you want it to run on port 80, you can redirect 80 to 8080 in the firewall.

Usage

First, use the deploy.init task to template files from the library to the rel/templates/deploy directory in your project.

mix deploy.init

Next, generate the scripts based on your project's config:

MIX_ENV=prod mix deploy.generate
chmod +x bin/*

By default, mix deploy.generate creates scripts under a bin directory at the top level of your project. If you want to keep them separate, e.g. to create different files based on the environment, set bin_dir to [:output_dir, "bin"] and it will generate files under e.g. _build/prod/deploy.

Configuration

The library tries to choose smart defaults. It reads the app name from mix.exs and calculates default values for its configuration parameters.

If your app is named foo_bar, it will create a service named foo-bar, deployed to /srv/foo-bar, running under the user foo-bar.

The library doesn't generate any output scripts by default, you need to enable them with the templates parameter. It can create the following scripts:

Systemd scripts

These are wrappers on e.g. /bin/systemctl restart foo. They are useful for e.g. CodeDeploy hook scripts where we have to run a script without parameters.

• deploy-start: Start services
• deploy-stop: Stop services
• deploy-restart: Restart services
• deploy-enable: Enable systemd units

System setup scripts

These scripts set up the target system for the application. They are useful for local and automated deploy.

• deploy-create-users: Create OS accounts for app and deploy users
• deploy-create-dirs: Create dirs, including the release dir /srv/foo and standard dirs like /etc/foo if needed.

Local deploy scripts

These scripts deploy the app to the same server as it was built on:

• deploy-copy-files: Copy files from _build to target /srv/foo, or to a staging directory for packaging
• deploy-release: Deploy release, extracting to a timestamped dir under /srv/foo/releases, then making a symlink from /srv/foo/current
• deploy-rollback: Rollback release, resetting the symlink to point to the previous release

The library also has mix tasks to deploy and roll back releases:

mix deploy.local
mix deploy.local.rollback

CodeDeploy deploy scripts

These scripts run on the target machine as lifecycle hooks.

• deploy-clean-target: Delete files under target dir in preparation for deploying update
• deploy-extract-release: Extract release from tar
• deploy-set-perms: Set target file permissions so that they can be used by the app user

Build server scripts

These scripts run on the build server.

• deploy-stage-files: Copy output files to staging directory, default files

Release command scripts

These scripts set up the environment and then run release commands. They make the config match the environment vars set at runtime in the systemd unit. With Elixir 1.9+ you can source /srv/foo/bin/set-env in rel/env.sh.eex. The other scripts are mainly useful with Distillery.

• set-env: Set up environment
• deploy-migrate: Migrate database on target system by running a custom command.
• deploy-remote-console: Launch remote console for the app

Runtime environment scripts

These scripts are called by the systemd unit to set get the application config at runtime prior to starting the app. They are more most useful with Distillery.

Elixir 1.9+ mix releases support runtime configuration via config/runtime.exs and rel/env.sh.eex. It is more secure, however, to separate the process of getting configuration from the app itself using ExecStartPre]). See mix_systemd for examples.

• deploy-sync-config-s3: Sync config files from S3 bucket to app configuration_dir
• deploy-runtime-environment-file: Create #{runtime_dir}/environment file on target from cloud-init metadata
• deploy-runtime-environment-wrap: Get runtime environment from cloud-init metadata, set environment vars, then launch main script.
• deploy-set-cookie-ssm: Get Erlang VM cookie from AWS SSM Parameter Store and write to file.

The most useful of these is deploy-sync-config-s3, the rest are code you might copy into rel/env.sh.eex.

Dependencies

The generated scripts are mostly straight bash, with minimal dependencies.

• deploy-sync-config-s3 uses the AWS CLI to copy files from an S3 bucket.
• deploy-runtime-environment-file and deploy-runtime-environment-wrap use jq to parse the cloud-init JSON file.
• deploy-set-cookie-ssm uses the AWS CLI and jq to interact with Systems Manager Parameter Store.

To install jq on Ubuntu:

apt-get install jq

To install the AWS CLI from the OS package manager on Ubuntu:

apt-get install awscli

Scenarios

CodeDeploy

The library can generate lifecycle hook scripts for use with a deployment system such as AWS CodeDeploy.

config :mix_deploy,
  app_user: "app",
  app_group: "app",
  templates: [
    "stop",
    "create-users",
    "create-dirs",
    "clean-target",
    "extract-release",
    "set-perms",
    "migrate",
    "enable",
    "start",
    "restart",
  ],
    ...

Here is an example appspec.yml file:

version: 0.0
os: linux
files:
  - source: bin
    destination: /srv/foo/bin
  - source: systemd
    destination: /lib/systemd/system
  - source: etc
    destination: /srv/foo/etc
hooks:
  ApplicationStop:
    - location: bin/deploy-stop
      timeout: 300
  BeforeInstall:
    - location: bin/deploy-create-users
    - location: bin/deploy-create-dirs
    - location: bin/deploy-clean-target
  AfterInstall:
    - location: bin/deploy-extract-release
    - location: bin/deploy-set-perms
    - location: bin/deploy-enable
  ApplicationStart:
    - location: bin/deploy-migrate
      runas: app
      timeout: 300
    - location: bin/deploy-start
      timeout: 3600
  # ValidateService:
    - location: bin/validate-service
      timeout: 300

Staging

By default, the scripts deploy the scripts as the same OS user that runs the mix deploy.generate command, and run the app under an OS user with the same name as the app.

Many scripts allow you to override environment variables at execution time. For example, you can override the user accounts which own the files by setting the environment vars APP_USER, APP_GROUP, and DEPLOY_USER.

Similarly, set DESTDIR and the copy script will add a prefix when copying files. This lets you copy files to a staging directory, tar it up, then extract it on a target machine, e.g.:

mkdir -p ~/tmp/deploy
DESTDIR=~/tmp/deploy bin/deploy-create-dirs
DESTDIR=~/tmp/deploy bin/deploy-copy-files

Configuration options

The following sections describe common configuration options. See lib/mix/tasks/deploy.ex for details of more obscure options.

If you need to make changes not supported by the config options, then you can check the templates in rel/templates/deploy into source control and make your own changes. Contributions are welcome!

The list of templates to generate is in the templates config var. You can modify this list to remove scripts, and they won't be generated. You can also add your own scripts and they will be run as templates with the config vars defined.

Basics

app_name: Elixir application name, an atom, from the app field in the mix.exs project.

version: version field in mix.exs project.

module_name: Elixir camel case module name version of app_name, e.g. FooBar.

release_name: Name of release, default app_name.

ext_name: External name, used for files and directories, default app_name with underscores converted to "-", e.g. foo-bar.

service_name: Name of the systemd service, default ext_name.

release_system: :mix | :distillery, default :mix

Identifies the system used to generate the releases, Mix or Distillery.

Users

deploy_user: OS user account that is used to deploy the app, e.g. own the files and restart it. For security, this is separate from app_user, keeping the runtime user from being able to modify the source files. Defaults to the user running the script, supporting local deploy. For remote deploy, set this to a user like deploy or same as the app user.

deploy_group: OS group account, default deploy_user.

app_user: OS user account that the app should run under. Default deploy_user.

app_group: OS group account, default deploy_group.

Directories

base_dir: Base directory for app files on target, default /srv.

deploy_dir: Directory for app files on target, default #{base_dir}/#{ext_name}.

We use the standard app directories, for modern Linux systems. App files are under /srv, configuration under /etc, transient files under /run, data under /var/lib.

Directories are named based on the app name, e.g. /etc/#{ext_name}. The dirs variable specifies which directories the app uses. By default, it doesn't set up anything. To enable them, configure dirs, e.g.:

dirs: [
  # :runtime,       # App runtime files which may be deleted between runs, /run/#{ext_name}
  # :configuration, # App configuration, e.g. db passwords, /etc/#{ext_name}
  # :state,         # App data or state persisted between runs, /var/lib/#{ext_name}
  # :cache,         # App cache files which can be deleted, /var/cache/#{ext_name}
  # :logs,          # App external log files, not via journald, /var/log/#{ext_name}
  # :tmp,           # App temp files, /var/tmp/#{ext_name}
],

Recent versions of systemd (since 235) will create these directories at start time based on the settings in the unit file. For earlier systemd versions, deploy-create-dirs will create them.

For security, we set permissions to 750, more restrictive than the systemd defaults of 755. You can configure them with variables like configuration_directory_mode. See the defaults in lib/mix/tasks/deploy.ex.

systemd_version: Sets the systemd version on the target system, default 235. This determines which systemd features the library will enable. If you are targeting an older OS release, you may need to change it. Here are the systemd versions in common OS releases:

• CentOS 7: 219
• Ubuntu 16.04: 229
• Ubuntu 18.04: 237

Additional directories

The library uses a directory structure under deploy_dir which supports multiple releases, similar to Capistrano.

• scripts_dir: deployment scripts which e.g. start and stop the unit, default bin.
• current_dir: where the current Erlang release is unpacked or referenced by symlink, default current.
• releases_dir: where versioned releases are unpacked, default releases.
• flags_dir: dir for flag files to trigger restart, e.g. when restart_method is :systemd_flag, default flags.

When using multiple releases and symlinks, the deployment process works as follows:

1. Create a new directory for the release with a timestamp like /srv/foo/releases/20181114T072116.

2. Upload the new release tarball to the server and unpack it to the releases dir

3. Make a symlink from /srv/#{ext_name}/current to the new release dir.

4. Restart the app.

If you are only keeping a single version, then deploy it to the directory /srv/#{ext_name}/current.

Variable expansion

The following variables support variable expansion:

expand_keys: [
  :env_files,
  :env_vars,
  :runtime_environment_service_script,
  :conform_conf_path,
  :pid_file,
  :root_directory,
  :bin_dir,
]

You can specify values as a list of terms, and it will look up atoms as keys in the config. This lets you reference e.g. the deploy dir or configuration dir without having to specify the full path, e.g. ["!", :deploy_dir, "/bin/myscript"] gets converted to "!/srv/foo/bin/myscript".

Environment vars

Config vars set a few common env vars:

• mix_env: default Mix.env(), sets MIX_ENV
• env_lang: default en_US.utf8, used to set LANG

In addition, you can set env_vars and env_files the same way as for mix_systemd. The set-env script will then set these variables the same way as they are in the systemd unit, allowing you to run release commands with the same config, e.g. database migrations or console. It also sets:

• RUNTIME_DIR: runtime_dir, if :runtime in dirs
• CONFIGURATION_DIR: configuration_dir, if :configuration in dirs
• LOGS_DIR: logs_dir, if :logs in dirs
• CACHE_DIR: cache_dir, if :cache in dirs
• STATE_DIR: state_dir, if :state in dirs
• TMP_DIR: tmp_dir, if :tmp in dirs

You can set additional vars using env_vars, e.g.:

env_vars: [
  "PORT=8080",
]

You can also reference the value of other parameters by name, e.g.:

env_vars: [
  ["RELEASE_TMP=", :runtime_dir],
]

You can read environment vars from files with env_files, e.g.:

env_files: [
  ["-", :deploy_dir, "/etc/environment"],
  ["-", :configuration_dir, "environment"],
  ["-", :runtime_dir, "environment"],
],

The "-" at the beginning makes the file optional, the system will start without them. Later values override earlier values, so you can set defaults in the release which get overridden in the deployment or runtime environment.

With Distillery, you can generate a file under the release with an overlay in rel/config.exs, e.g.:

environment :prod do
  set overlays: [
    {:mkdir, "etc"},
    {:copy, "rel/etc/environment", "etc/environment"},
    # {:template, "rel/etc/environment", "etc/environment"}
  ]
end

That results in a file that would be read by:

env_files: [
  ["-", :current_dir, "/etc/environment"],
],

Starting and restarting

The following variables set systemd variables:

service_type: :simple | :exec | :notify | :forking. systemd Type, default :simple.

Modern applications don't fork, they run in the foreground and rely on the supervisor to manage them as a daemon. This is done by setting service_type to :simple or :exec. Note that in simple mode, systemd doesn't actually check if the app started successfully, it just continues starting other units. If something depends on your app being up, :exec may be better.

Set service_type to :forking, and the library sets pid_file to #{runtime_directory}/#{app_name}.pid and sets the PIDFILE env var to tell the boot scripts where it is.

The Erlang VM runs pretty well in foreground mode, but traditionally runs as as a standard Unix-style daemon, so forking might be better. Systemd expects foregrounded apps to die when their pipe closes. See https://elixirforum.com/t/systemd-cant-shutdown-my-foreground-app-cleanly/14581/2

restart_method: :systemctl | :systemd_flag | :touch, default :systemctl

The normal situation is that the app will be restarted using e.g. systemctl restart foo.

With :systemd_flag, an additional systemd unit file watches for changes to a flag file and restarts the main unit. This allows updates to be pushed to the target machine by an unprivileged user account which does not have permissions to restart processes. Touch the file #{flags_dir}/restart.flag and systemd will restart the unit. See mix_systemd for details.

With :touch, the app itself watches the file #{flags_dir}/restart.flag. If it changes, the app shuts itself down, relying on systemd to notice and restart it.

sudo_deploy: Creates /etc/sudoers.d/#{ext_name} file which allows the deploy user to start/stop/restart the app using sudo. Default false. Note that when you must call systemctl with the full path, e.g. sudo /bin/systemctl restart foo for this to work.

sudo_app: Creates /etc/sudoers.d/#{ext_name} file allowing the app user user to start/stop/restart the app using sudo. Default false.

Configuration examples

Here is a complete example of configuring an app from a config file which it pulls from S3 on startup.

We set up an ExecStartPre command in the systemd unit file which runs deploy-sync-config-s3 before starting the app. It runs the AWS cli command:

aws s3 sync "s3://${CONFIG_S3_BUCKET}/${CONFIG_S3_PREFIX}" "${CONFIG_DIR}/"

CONFIG_S3_BUCKET is the source bucket, and CONFIG_S3_PREFIX is an optional path in the bucket. CONFIG_DIR is the app configuration dir on the target system, /etc/foo.

We need to bootstrap the config process, so we use a different environment file from the main config.

mkdir -p rel/etc
echo "CONFIG_S3_BUCKET=cogini-foo-dev-app-config" >> rel/etc/environment

Set exec_start_pre in the mix_systemd config:

config :mix_systemd,
  app_user: "app",
  app_group: "app",
  # systemd runs this before starting the app as root
  exec_start_pre: [
    ["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
  ],
  dirs: [
    # Create /etc/foo
    :configuration,
    # Create /run/foo
    :runtime,
  ],
  # systemd should not clean up /run/foo
  runtime_directory_preserve: "yes",
  # Load env from /srv/foo/etc/environment and /etc/foo/environment
  env_files: [
    ["-", :deploy_dir, "/etc/environment"],
    ["-", :configuration_dir, "/environment"],
  ],
  # deploy-copy-files will copy the env file to /srv/foo/etc
  # more likely it is done by e.g. appspec.yml
  copy_files: [
    %{
      src: "rel/etc/environment",
      dst: [:deploy_dir, "/etc"],
      user: "$DEPLOY_USER",
      group: "$APP_GROUP",
      mode: "640"
    },
  ],
  env_vars: [
    # Temp files are in /run/foo
    ["RELEASE_TMP=", :runtime_dir],
  ]

config :mix_deploy,
  app_user: "app",
  app_group: "app"
  templates: [
    "init-local",
    "create-users",
    "create-dirs",
    "copy-files",
    "enable",
    "release",
    "restart",
    "rollback",
    "start",
    "stop",

    "sync-config-s3",
  ],
  dirs: [
    :configuration,
    :runtime,
  ],
  # Set env config in e.g. deploy-set-env to match above.
  env_files: [
    ["-", :deploy_dir, "/etc/environment"],
    ["-", :configuration_dir, "/environment"],
  ]
  env_vars: [
    ["RELEASE_TMP=", :runtime_dir],
  ]

For security, the app only has read-only access to its config files, and /etc/foo has ownership deploy:foo and mode 750. We prefix the command with "!" so it runs with elevated permissions, not as the foo user.

We need to set the CONFIG_S3_BUCKET variable in the environment so that deploy-sync-config-s3 can use it. We can set it in env_vars or put it in the file /etc/foo/environment.

• /srv/foo/etc/environment settings are configured at deploy time.
• /etc/foo/environment settings might come from an S3 bucket.
• /run/foo/environment settings might be generated dynamically, e.g. getting the IP address.

For example, post_build commands in the CodeBuild CI buildspec.yml file can generate a config file files/etc/environment:

post_build:
  commands:
    - mkdir -p files/etc
    - echo "CONFIG_S3_BUCKET=$BUCKET_CONFIG" >> files/etc/environment

Then the CodeDeploy appspec.yml copies it to the target system under /srv/foo/etc:

files:
  - source: bin
    destination: /srv/foo/bin
  - source: systemd
    destination: /lib/systemd/system
  - source: etc
    destination: /srv/foo/etc

See mix_systemd for more examples.

I am jakemorrison on on the Elixir Slack and Discord, reachfh on Freenode #elixir-lang IRC channel. Happy to chat or help with your projects.

Copyright and License

Copyright (c) 2019 Jake Morrison

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.