Dockerfile Image Updater
This tool provides a mechanism to make security updates to docker images at scale. The tool searches github for declared docker images and sends pull requests to projects that are not using the desired version of the requested docker image.
Docker builds images using a declared Dockerfile. Within the Dockerfile,
there is a FROM declaration that specifies the base image and a tag that
will be used as the starting layers for the new image. If the base image that
FROM depends on is rebuilt, the Docker images that depend on it will never
be updated with the newer layers. This becomes a major problem if the reason
the base image was updated was to fix a security vulnerability. All Docker
images are often based on operating system libraries and these get patched
for security updates quite frequently. This tool, the Dockerfile Image Updater
was created to automatically make sure that child images are updated when the
images they depend on get updated.
Table of contents
User Guide
What it does
The tool has three modes
-
all- Reads store that declares the docker images and versions that you intend others to use.Example:
export git_api_url=https://api.github.com export git_api_token=my_github_token docker run --rm -e git_api_token -e git_api_url \ salesforce/dockerfile-image-update all image-to-tag-store -
parent- Searches github for images that use a specified image name and sends pull requests if the image tag doesn't match intended tag. The intended image with tag is passed in the command line parameters. The intended image-to-tag mapping is persisted in a store in a specified git repository under the token owner.Example:
export git_api_url=https://api.github.com export git_api_token=my_github_token docker run --rm -e git_api_token -e git_api_url \ salesforce/dockerfile-image-update parent my_org/my_image v1.0.1 \ image-to-tag-store -
child- Given a specific git repo, sends a pull request to update the image to a given version. You can optionally persist the image version combination in the image-to-tag store.Example:
export git_api_url=https://api.github.com export git_api_token=my_github_token docker run --rm -e git_api_token -e git_api_url \ salesforce/dockerfile-image-update child my_gh_org/my_gh_repo \ my_image_name v1.0.1
Prerequisites
In environment variables, please provide:
git_api_token: This is your GitHub token to your account. Set these privileges by: going to your GitHub account --> settings --> Personal access tokens --> checkrepoanddelete_repo.git_api_url: This is the Endpoint URL of the GitHub API. In general GitHub, this ishttps://api.github.com/; for Enterprise, this should behttps://hostname/api/v3. (this variable is optional; you can provide it through the command line.)
Precautions
- This tool may create a LOT of forks in your account. All pull requests created are through a fork on your own account.
- We currently do not operate on forked repositories due to limitations in forking a fork on GitHub. We should invest some time in doing this right. See issue #21
- Submodules are separate repositories and get their own pull requests.
How to use it
Our recommendation is to run it as a docker container:
export git_api_url=https://api.github.com
export git_api_token=my_github_token
docker run --rm -e git_api_token -e git_api_url \
salesforce/dockerfile-image-update <COMMAND> <PARAMETERS>
usage: dockerfile-image-update [-h] [-l GHAPISEARCHLIMIT] [-o ORG] [-b BRANCH] [-g GHAPI] [-f] [-m M] [-c C] [-e EXCLUDES] [-B B] [-s {true,false}] [-x X] COMMAND ...
Image Updates through Pull Request Automator
named arguments:
-h, --help show this help message and exit
-l GHAPISEARCHLIMIT, --ghapisearchlimit GHAPISEARCHLIMIT
limit the search results for github api (default: 1000)
-o ORG, --org ORG search within specific organization (default: all of github)
-b BRANCH, --branch BRANCH
make pull requests for given branch name (default: main)
-g GHAPI, --ghapi GHAPI
link to github api; overrides environment variable
-f, --auto-merge NOT IMPLEMENTED / set to automatically merge pull requests if available
-m M message to provide for pull requests
-c C additional commit message for the commits in pull requests
-e EXCLUDES, --excludes EXCLUDES
regex of repository names to exclude from pull request generation
-B B additional body text to include in pull requests
-s {true,false}, --skipprcreation {true,false}
Only update image tag store. Skip creating PRs
-x X comment snippet mentioned in line just before FROM instruction for ignoring a child image. Defaults to 'no-dfiu'
-r, --rate_limit_pr_creations
Enable rateLimiting for throttling the number of PRs DFIU will cut over a period of time.
The argument value should be in format "<positive_integer>-<ISO-8601_formatted_time>". For example "--rate_limit_pr_creations 60-PT1H" to create 60 PRs per hour.
Default is not set, this means no ratelimiting is imposed.
subcommands:
Specify which feature to perform
COMMAND FEATURE
parent updates all repositories' Dockerfiles with given base image
all updates all repositories' Dockerfiles
child updates one specific repository with given tag
The all command
Specify an image-to-tag store (a repository name on GitHub that contains a file called store.json); looks through the JSON file and checks/updates all the base images in GitHub to the tag in the store.
usage: dockerfile-image-update all [-h] <IMG_TAG_STORE>
positional arguments:
<IMG_TAG_STORE> REQUIRED
optional arguments:
-h, --help show this help message and exit
The child command
Forcefully updates a repository's Dockerfile(s) to given tag. If specified a store, it will also forcefully update the store.
usage: dockerfile-image-update child [-h] [-s <IMG_TAG_STORE>] <GIT_REPO> <IMG> <FORCE_TAG>
positional arguments:
<GIT_REPO> REQUIRED
<IMG> REQUIRED
<FORCE_TAG> REQUIRED
optional arguments:
-h, --help show this help message and exit
-s <IMG_TAG_STORE> OPTIONAL
The parent command
Given an image, tag, and store, it will create pull requests for any Dockerfiles that has the image as a base image and an outdated tag. It also updates the store.
usage: dockerfile-image-update parent [-h] <IMG> <TAG> <IMG_TAG_STORE>
positional arguments:
<IMG> REQUIRED
<TAG> REQUIRED
<IMG_TAG_STORE> REQUIRED
optional arguments:
-h, --help show this help message and exit
Skip An Image
In case you want the tool to skip updating a particular image tag then add a comment no-dfiu after the FROM declaration in the Dockerfile. The tool will process the comment following FROM declaration and if no-dfiu is mentioned, pull request for that image tag will be ignored. You can use an alternate comment string by passing an additional command line parameter -x IGNORE_IMAGE_STRING. In that case string mentioned with the parameter, will be used for skipping PR creation.
Example:
FROM imagename:imagetag # no-dfiu
PR throttling
In case you want to throttle the number of PRs cut by DFIU over a period of time, set --rate_limit_pr_creations with appropriate value.
Default case:
By default, this feature is disabled. This will be enabled when argument --rate_limit_pr_creations will be passed
with appropriate value.
example: dockerfile-image-update all image-tag-store-repo-falcon //throttling will be disabled by default
Configuring the rate limit:
Below are some examples that will throttle the number of PRs cut based on values passed to the
argument --rate_limit_pr_creations
The argument value should be in format <positive_integer>-<ISO-8601_formatted_time>.
For example --rate_limit_pr_creations 60-PT1H would mean the tool will cut 60 PRs every hour and the rate of adding
a new PR will be (PT1H/60) i.e. one minute.
This will distribute the load uniformly and avoid sudden spikes, The process will go in waiting state until next PR
could be sent.
Below are some more examples:
Usage:
dockerfile-image-update --rate_limit_pr_creations 60-PT1H all image-tag-store-repo-falcon //DFIU can send up to 60 PRs per hour.
dockerfile-image-update --rate_limit_pr_creations 500-PT1H all image-tag-store-repo-falcon //DFIU can send up to 500 PRs per hour.
dockerfile-image-update --rate_limit_pr_creations 86400-PT24H all image-tag-store-repo-falcon //DFIU can send up to 1 PRs per second.
dockerfile-image-update --rate_limit_pr_creations 1-PT1S all image-tag-store-repo-falcon //Same as above. DFIU can send up to 1 PRs per second.
dockerfile-image-update --rate_limit_pr_creations 5000 all image-tag-store-repo-falcon //rate limiting will be disabled because argument is not in correct format.
Developer Guide
Building
git clone https://github.com/salesforce/dockerfile-image-update.git
cd dockerfile-image-update
mvn clean install
Running locally
cd dockerfile-image-update/target
java -jar dockerfile-image-update-1.0-SNAPSHOT.jar <COMMAND> <PARAMETERS>
Creating a new feature
Under dockerfile-image-update/src/main/java/com/salesforce/dva/dockerfileimageupdate/subcommands/impl,
create a new class YOUR_FEATURE.java.
Make sure it implements ExecutableWithNamespace and has the SubCommand
annotation with a help, requiredParams, and optionalParams.
Then, under the execute method, code what you want this tool to do.
Running unit tests
Run unit tests by running mvn test.
Running integration tests
Before you run the integration tests (locally):
-
Make sure that you have access to the github orgs specified in TestCommon.ORGS. You likely will need to change it to three orgs where you have permissions to create repositories.
-
Make sure you have
git_api_url=https://api.github.comin/dockerfile-image-update-itest/itest.env, or set it to your internal GitHub Enterprise. -
Make sure you have a secret file which contains the
git_api_token. The token needs to havedelete_repo, repopermissions. You can generate your token by going to personal access tokens in GitHub. Once you have your token place it in a file:echo git_api_token=[copy personal access token here] > ${HOME}/.dfiu-itest-token -
Export the following environment variable to point to the file:
export user_itest_secrets_file_secret=${HOME}/.dfiu-itest-token -
Run integration tests by running
make integration-test
Release Process
We currently use GitHub Actions and Releases. In order to collect dependency updates from dependabot and any other minor changes, we've switched to a process to manually trigger the release process. For now, that looks like the following:
1. Versioned Git Tag
- Decide what version you desire to have. If you want to bump the major or minor
version then you need to bump the
MVN_SNAPSHOT_VERSIONin the Makefile and in the Dockerfile before proceeding to the next steps. For exampleMVN_SNAPSHOT_VERSION=1.0-SNAPSHOTtoMVN_SNAPSHOT_VERSION=2.0-SNAPSHOT. - After PRs have been merged to the primary branch, go to the Actions tab
and trigger the
Release new versionWorkflow. This will build, integration test, deploy the latest version to Docker Hub and Maven Central, and tag that commit hash with the next semantic version.
2. Cut Release with Release Notes
- PRs continually get updated with labels by Pull Request Labeler and that helps set us up for nice release notes by Release Drafter.
- Once that release has been tagged you can go to the draft release which
is continually updated by Release Drafter
and select the latest tag to associate with that release. Change the
version to reflect the same version as the tag (
1.0.${NEW_VERSION}). Take a look at the release notes to make sure that PRs are categorized correctly. The categorization is based on the labels of the PRs. You can either fix the labels on the PRs, which will trigger the release drafter action, or simply modify the release notes before publishing. Ideally we'll automate this to run at the end of the triggered workflow with something like svu.
Checking Code Climate Locally
If you'd like to check Code Climate results locally you can run the following:
docker run --interactive --tty --rm \
--env CODECLIMATE_CODE="$(pwd)" \
--volume "$(pwd)":/code \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume /tmp/cc:/tmp/cc \
codeclimate/codeclimate analyze README.md