• Stars
    star
    321
  • Rank 130,752 (Top 3 %)
  • Language
    TypeScript
  • License
    Apache License 2.0
  • Created over 3 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

CDK Pipelines for GitHub Workflows

cdk-constructs: Experimental

View on Construct Hub

The APIs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

A construct library for painless Continuous Delivery of CDK applications, deployed via GitHub Workflows.

The CDK already has a CI/CD solution, CDK Pipelines, which creates an AWS CodePipeline that deploys CDK applications. This module serves the same surface area, except that it is implemented with GitHub Workflows.

Table of Contents

Usage

Assuming you have a Stage called MyStage that includes CDK stacks for your app and you want to deploy it to two AWS environments (BETA_ENV and PROD_ENV):

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  awsCreds: AwsCredentials.fromOpenIdConnect({
    gitHubActionRoleArn: 'arn:aws:iam::<account-id>:role/GitHubActionRole',
  }),
});

// Build the stages
const betaStage = new MyStage(app, 'Beta', { env: BETA_ENV });
const prodStage = new MyStage(app, 'Prod', { env: PROD_ENV });

// Add the stages for sequential build - earlier stages failing will stop later ones:
pipeline.addStage(betaStage);
pipeline.addStage(prodStage);

// OR add the stages for parallel building of multiple stages with a Wave:
const wave = pipeline.addWave('Wave');
wave.addStage(betaStage);
wave.addStage(prodStage);

app.synth();

When you run cdk synth, a deploy.yml workflow will be created under .github/workflows in your repo. This workflow will deploy your application based on the definition of the pipeline. In the example above, it will deploy the two stages in sequence, and within each stage, it will deploy all the stacks according to their dependency order and maximum parallelism. If your app uses assets, assets will be published to the relevant destination environment.

The Pipeline class from cdk-pipelines-github is derived from the base CDK Pipelines class, so most features should be supported out of the box. See the CDK Pipelines documentation for more details.

To express GitHub-specifc details, such as those outlined in Additional Features, you have a few options:

  • Use a GitHubStage instead of Stage (or make a GitHubStage subclass instead of a Stage subclass) - this adds the GitHubCommonProps to the Stage properties
    • With this you can use pipeline.addStage(myGitHubStage) or wave.addStage(myGitHubStage) and the properties of the stage will be used
  • Using a Stage (or subclass thereof) or a GitHubStage (or subclass thereof) you can call pipeline.addStageWithGitHubOptions(stage, stageOptions) or wave.addStageWithGitHubOptions(stage, stageOptions)
    • In this case you're providing the same options along with the stage instead of embedded in the stage.
    • Note that properties of a GitHubStage added with addStageWithGitHubOptions() will override the options provided to addStageWithGitHubOptions()

NOTES:

Initial Setup

Assuming you have your CDK app checked out on your local machine, here are the suggested steps to develop your GitHub Workflow.

  • Set up AWS Credentials your local environment. It is highly recommended to authenticate via an OpenId Connect IAM Role. You can set one up using the GithubActionRole class provided in this module. For more information (and alternatives), see AWS Credentials.

  • When you've updated your pipeline and are ready to deploy, run cdk synth. This creates a workflow file in .github/workflows/deploy.yml.

  • When you are ready to test your pipeline, commit your code changes as well as the deploy.yml file to GitHub. GitHub will automatically try to run the workflow found under .github/workflows/deploy.yml.

  • You will be able to see the result of the run on the Actions tab in your repository:

    Screen Shot 2021-08-22 at 12 06 05

For an in-depth run-through on creating your own GitHub Workflow, see the Tutorial section.

AWS Credentials

There are two ways to supply AWS credentials to the workflow:

  • GitHub Action IAM Role (recommended).
  • Long-lived AWS Credentials stored in GitHub Secrets.

The GitHub Action IAM Role authenticates via the GitHub OpenID Connect provider and is recommended, but it requires preparing your AWS account beforehand. This approach allows your Workflow to exchange short-lived tokens directly from AWS. With OIDC, benefits include:

  • No cloud secrets.
  • Authentication and authorization management.
  • Rotating credentials.

You can read more here.

GitHub Action Role

Authenticating via OpenId Connect means you do not need to store long-lived credentials as GitHub Secrets. With OIDC, you provide a pre-provisioned IAM role with optional role session name to your GitHub Workflow via the awsCreds.fromOpenIdConnect API:

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  awsCreds: AwsCredentials.fromOpenIdConnect({
    gitHubActionRoleArn: 'arn:aws:iam::<account-id>:role/GitHubActionRole',
    roleSessionName: 'optional-role-session-name',
  }),
});

There are two ways to create this IAM role:

  • Use the GitHubActionRole construct (recommended and described below).
  • Manually set up the role (Guide).

GitHubActionRole Construct

Because this construct involves creating an IAM role in your account, it must be created separate to your GitHub Workflow and deployed via a normal cdk deploy with your local AWS credentials. Upon successful deployment, the arn of your newly created IAM role will be exposed as a CfnOutput.

To utilize this construct, create a separate CDK stack with the following code and cdk deploy:

class MyGitHubActionRole extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const provider = new GitHubActionRole(this, 'github-action-role', {
      repos: ['myUser/myRepo'],
    });
  }
}

const app = new App();
new MyGitHubActionRole(app, 'MyGitHubActionRole');
app.synth();

Note: If you have previously created the GitHub identity provider with url https://token.actions.githubusercontent.com, the above example will fail because you can only have one such provider defined per account. In this case, you must provide the already created provider into your GithubActionRole construct via the provider property.

Make sure the audience for the provider is sts.amazonaws.com in this case.

class MyGitHubActionRole extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const provider = new GitHubActionRole(this, 'github-action-role', {
      repos: ['myUser/myRepo'],
      provider: GitHubActionRole.existingGitHubActionsProvider(this),
    });
  }
}

GitHub Secrets

Authenticating via this approach means that you will be manually creating AWS credentials and duplicating them in GitHub secrets. The workflow expects the GitHub repository to include secrets with AWS credentials under AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. You can override these defaults by supplying the awsCreds.fromGitHubSecrets API to the workflow:

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  awsCreds: AwsCredentials.fromGitHubSecrets({
    accessKeyId: 'MY_ID', // GitHub will look for the access key id under the secret `MY_ID`
    secretAccessKey: 'MY_KEY', // GitHub will look for the secret access key under the secret `MY_KEY`
  }),
});

Runners with Preconfigured Credentials

If your runners provide credentials themselves, you can configure awsCreds to skip passing credentials:

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  awsCreds: AwsCredentials.runnerHasPreconfiguredCreds(), // NO credentials will be provided.
});

Using Docker in the Pipeline

You can use Docker in GitHub Workflows in a similar fashion to CDK Pipelines. For a full discussion on how to use Docker in CDK Pipelines, see Using Docker in the Pipeline.

Just like CDK Pipelines, you may need to authenticate to Docker registries to avoid being throttled.

Authenticating to Docker registries

You can specify credentials to use for authenticating to Docker registries as part of the Workflow definition. This can be useful if any Docker image assets β€” in the pipeline or any of the application stages β€” require authentication, either due to being in a different environment (e.g., ECR repo) or to avoid throttling (e.g., DockerHub).

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  dockerCredentials: [
    // Authenticate to ECR
    DockerCredential.ecr('<account-id>.dkr.ecr.<aws-region>.amazonaws.com'),

    // Authenticate to DockerHub
    DockerCredential.dockerHub({
      // These properties are defaults; feel free to omit
      usernameKey: 'DOCKERHUB_USERNAME',
      personalAccessTokenKey: 'DOCKERHUB_TOKEN',
    }),

    // Authenticate to Custom Registries
    DockerCredential.customRegistry('custom-registry', {
      usernameKey: 'CUSTOM_USERNAME',
      passwordKey: 'CUSTOM_PASSWORD',
    }),
  ],
});

Runner Types

You can choose to run the workflow in either a GitHub hosted or self-hosted runner.

GitHub Hosted Runner

The default is Runner.UBUNTU_LATEST. You can override this as shown below:

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  runner: Runner.WINDOWS_LATEST,
});

Self Hosted Runner

The following example shows how to configure the workflow to run on a self-hosted runner. Note that you do not need to pass in self-hosted explicitly as a label.

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  runner: Runner.selfHosted(['label1', 'label2']),
});

Escape Hatches

You can override the deploy.yml workflow file post-synthesis however you like.

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
});

const deployWorkflow = pipeline.workflowFile;
// add `on: workflow_call: {}` to deploy.yml
deployWorkflow.patch(JsonPatch.add('/on/workflow_call', {}));
// remove `on: workflow_dispatch` from deploy.yml
deployWorkflow.patch(JsonPatch.remove('/on/workflow_dispatch'));

Additional Features

Below is a compilation of additional features available for GitHub Workflows.

GitHub Action Step

If you want to call a GitHub Action in a step, you can utilize the GitHubActionStep. GitHubActionStep extends Step and can be used anywhere a Step type is allowed.

The jobSteps array is placed into the pipeline job at the relevant jobs.<job_id>.steps as documented here.

In this example,

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
});

// "Beta" stage with a pre-check that uses code from the repo and an action
const stage = new MyStage(app, 'Beta', { env: BETA_ENV });
pipeline.addStage(stage, {
  pre: [new GitHubActionStep('PreBetaDeployAction', {
    jobSteps: [
      {
        name: 'Checkout',
        uses: 'actions/checkout@v3',
      },
      {
        name: 'pre beta-deploy action',
        uses: '[email protected]',
      },
      {
        name: 'pre beta-deploy check',
        run: 'npm run preDeployCheck',
      },
    ],
  })],
});

app.synth();

Configure GitHub Environment

You can run your GitHub Workflow in select GitHub Environments. Via the GitHub UI, you can configure environments with protection rules and secrets, and reference those environments in your CDK app. A workflow that references an environment must follow any protection rules for the environment before running or accessing the environment's secrets.

Assuming (just like in the main example) you have a Stage called MyStage that includes CDK stacks for your app and you want to deploy it to two AWS environments (BETA_ENV and PROD_ENV) as well as GitHub Environments beta and prod:

import { ShellStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const pipeline = new GitHubWorkflow(app, 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: [
      'yarn install',
      'yarn build',
    ],
  }),
  awsCreds: AwsCredentials.fromOpenIdConnect({
    gitHubActionRoleArn: 'arn:aws:iam::<account-id>:role/GitHubActionRole',
  }),
});

pipeline.addStageWithGitHubOptions(new Stage(this, 'Beta', {
  env: BETA_ENV,
}), {
  gitHubEnvironment: { name: 'beta' },
});
pipeline.addStageWithGitHubOptions(new MyStage(this, 'Prod', {
  env: PROD_ENV,
}), {
  gitHubEnvironment: { name: 'prod' },
});

app.synth();

Waves for Parallel Builds

You can add a Wave to a pipeline, where each stage of a wave will build in parallel.

Note: The pipeline.addWave() call will return a Wave object that is actually a GitHubWave object, but due to JSII rules the return type of addWave() cannot be changed. If you need to use wave.addStageWithGitHubOptions() then you should call pipeline.addGitHubWave() instead, or you can use GitHubStages to carry the GitHub properties.

When deploying to multiple accounts or otherwise deploying mostly-unrelated stacks, using waves can be a huge win.

Here's a relatively large (but real) example, without a wave:

without-waves-light-mode

You can see how dependencies get chained unnecessarily, where the cUrl step should be the final step (a test) for an account:

without-waves-deps-light-mode

Here's the exact same stages deploying the same stacks to the same accounts, but with a wave:

with-waves

And the dependency chains are reduced to only what is actually needed, with the cUrl calls as the final stage for each account:

deps

For additional information and a code example see here.

Manual Approval Step

One use case for using GitHub Environments with your CDK Pipeline is to create a manual approval step for specific environments via Environment protection rules. From the GitHub UI, you can specify up to 5 required reviewers that must approve before the deployment can proceed:

require-reviewers

For more information and a tutorial for how to set this up, see this discussion.

Pipeline YAML Comments

An "AUTOMATICALLY GENERATED FILE..." comment will by default be added to the top of the pipeline YAML. This can be overriden as desired to add additional context to the pipeline YAML.

declare const pipeline: GitHubWorkflow;

pipeline.workflowFile.commentAtTop = `AUTOGENERATED FILE, DO NOT EDIT DIRECTLY!

Deployed stacks from this pipeline:
${STACK_NAMES.map((s)=>`- ${s}\n`)}`;

This will generate the normal deploy.yml file, but with the additional comments:

# AUTOGENERATED FILE, DO NOT EDIT DIRECTLY!

# Deployed stacks from this pipeline:
# - APIStack
# - AuroraStack

name: deploy
on:
  push:
    branches:
< the rest of the pipeline YAML contents>

Tutorial

You can find an example usage in test/example-app.ts which includes a simple CDK app and a pipeline.

You can find a repository that uses this example here: eladb/test-app-cdkpipeline.

To run the example, clone this repository and install dependencies:

cd ~/projects # or some other playground space
git clone https://github.com/cdklabs/cdk-pipelines-github
cd cdk-pipelines-github
yarn

Now, create a new GitHub repository and clone it as well:

cd ~/projects
git clone https://github.com/myaccount/my-test-repository

You'll need to set up AWS credentials in your environment. Note that this tutorial uses long-lived GitHub secrets as credentials for simplicity, but it is recommended to set up a GitHub OIDC role instead.

export AWS_ACCESS_KEY_ID=xxxx
export AWS_SECRET_ACCESS_KEY=xxxxx

Bootstrap your environments:

export CDK_NEW_BOOTSTRAP=1
npx cdk bootstrap aws://ACCOUNTID/us-east-1
npx cdk bootstrap aws://ACCOUNTID/eu-west-2

Now, run the manual-test.sh script when your working directory is the new repository:

cd ~/projects/my-test-repository
~/projects/cdk-piplines/github/test/manual-test.sh

This will produce a cdk.out directory and a .github/workflows/deploy.yml file.

Commit and push these files to your repo and you should see the deployment workflow in action. Make sure your GitHub repository has AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY secrets that can access the same account that you synthesized against.

In this tutorial, you are supposed to commit cdk.out (i.e. the code is pre-synthed). Do not do this in your app; you should always synth during the synth step of the GitHub workflow. In the example app this is achieved through the preSynthed: true option. It is for example purposes only and is not something you should do in your app.

import { ShellStep } from 'aws-cdk-lib/pipelines';
const pipeline = new GitHubWorkflow(new App(), 'Pipeline', {
  synth: new ShellStep('Build', {
    commands: ['echo "nothing to do (cdk.out is committed)"'],
  }),
  // only the example app should do this. your app should synth in the synth step.
  preSynthed: true,
});

Not supported yet

Most features that exist in CDK Pipelines are supported. However, as the CDK Pipelines feature are expands, the feature set for GitHub Workflows may lag behind. If you see a feature that you feel should be supported by GitHub Workflows, please open a GitHub issue to track it.

Contributing

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

More Repositories

1

cdk-nag

Check CDK applications for best practices using a combination of available rule packs
TypeScript
782
star
2

cdk-watchful

Watching what's up with your CDK apps since 2019
TypeScript
531
star
3

cdk-monitoring-constructs

Easy-to-use CDK constructs for monitoring your AWS infrastructure
TypeScript
456
star
4

aws-delivlib

setup and manage continuous delivery pipelines for code libraries in multiple languages
TypeScript
372
star
5

construct-hub

AWS CDK construct library that can be used to deploy instances of the Construct Hub in any AWS Account.
TypeScript
200
star
6

cdk-ecr-deployment

A CDK construct to deploy docker image to Amazon ECR
Go
152
star
7

cdk-dynamo-table-viewer

A CDK construct which exposes an endpoint with the contents of a DynamoDB table
TypeScript
124
star
8

cdk-stacksets

TypeScript
80
star
9

cdk-docker-image-deployment

TypeScript
75
star
10

cdk-triggers

TypeScript
73
star
11

cdk-validator-cfnguard

TypeScript
67
star
12

cdk-cloudformation

TypeScript
60
star
13

decdk

Define AWS CDK applications declaratively
TypeScript
60
star
14

cdk-ecs-service-extensions

TypeScript
59
star
15

cdk-from-cfn

A cloudformation json -> cdk typescript transpiler.
Rust
58
star
16

cdk-import

Generates CDK level 1 constructs for public CloudFormation Registry types and modules
TypeScript
52
star
17

cdk-app-cli

The operator CLI for CDK apps.
TypeScript
52
star
18

publib

A unified toolchain for publishing libraries to popular package managers (formally jsii-release)
TypeScript
52
star
19

cdk-drift-monitor

Monitors for CloudFormation stack drifts.
TypeScript
48
star
20

jsii-docgen

Generates reference documentation for jsii modules
TypeScript
48
star
21

aws-cdk-testing-examples

Java
39
star
22

cdk-ecs-codedeploy

TypeScript
34
star
23

cdk-enterprise-iac

Utilities for using CDK within enterprise constraints
TypeScript
33
star
24

cdk-tweet-queue

SQS queue fed with a stream of tweets from a Twitter search
TypeScript
29
star
25

cdk-ssm-documents

TypeScript
27
star
26

awscdk-service-spec

TypeScript
25
star
27

awscdk-appsync-utils

TypeScript
24
star
28

aws-secrets-github-sync

Sync GitHub repository secrets from AWS Secrets Manager
TypeScript
22
star
29

cdk-amazon-chime-resources

TypeScript
22
star
30

construct-hub-webapp

React web app for Construct Hub
TypeScript
21
star
31

json2jsii

Generates jsii-compatible structs from JSON schemas
TypeScript
20
star
32

node-sscaff

Stupid scaffolding: copies an entire directory with variable substitution
TypeScript
20
star
33

cdk-cicd-wrapper

This repository contains the infrastructure as code to wrap your AWS CDK project with CI/CD around it.
TypeScript
20
star
34

jsii-srcmak

Generate multi-language object-oriented source code from typescript
TypeScript
19
star
35

awscdk-v1-stack-finder

TypeScript
18
star
36

awscdk-asset-kubectl

TypeScript
18
star
37

cdk-skylight

CDK Skylight is a set of Level 3 constructs for the Microsoft products on AWS
TypeScript
17
star
38

markmac

Embed program outputs in markdown
TypeScript
17
star
39

cdk-verified-permissions

Amazon Verified Permissions L2 CDK Constructs
TypeScript
16
star
40

awscdk-change-analyzer

A toolkit that analyzes the differences between two cloud infrastructures states.
HTML
13
star
41

cdk-ethereum-node

CDK construct to deploy an Ethereum node running on Amazon Managed Blockchain
TypeScript
11
star
42

aws-lambda-rust

TypeScript
10
star
43

cdk-appflow

This repository contains L2 AWS CDK constructs for Amazon AppFlow
TypeScript
9
star
44

aws-cdk-notices

Static website for the distribution of notices to AWS CDK users.
TypeScript
8
star
45

cdk-hyperledger-fabric-network

CDK construct to deploy a Hyperledger Fabric network running on Amazon Managed Blockchain
TypeScript
7
star
46

cfn-resources

TypeScript
7
star
47

cdk-nag-go

cdk-nag bindings for Go.
Go
7
star
48

awscdk-asset-awscli

TypeScript
7
star
49

cdk-codepipeline-extensions

Go
7
star
50

cdk-aws-sagemaker-role-manager

Create roles and policies for ML Activities and ML Personas
TypeScript
6
star
51

cdklabs-projen-project-types

TypeScript
6
star
52

cdk-golden-signals-dashboard

TypeScript
6
star
53

awscdk-lambda-dotnet

TypeScript
5
star
54

cdk-deployer

A CDK construct library for deploying artifacts via CodeDeploy.
TypeScript
4
star
55

awscdk-dynamodb-global-tables

TypeScript
4
star
56

node-backpack

TypeScript
3
star
57

cfunctions

TypeScript
3
star
58

cdk-dynamo-table-viewer-go

A CDK construct which exposes an endpoint with the contents of a DynamoDB table Topics Resources License
Go
3
star
59

cdk-cdk8s-reinvent-2021

Java
2
star
60

construct-hub-probe

Probe package intended to test construct hub instances
TypeScript
2
star
61

construct-hub-cli

TypeScript
1
star
62

cdk-assets

TypeScript
1
star
63

cloud-assembly-schema

TypeScript
1
star
64

cdk-ecs-codedeploy-go

Go
1
star
65

cdk-generate-synthetic-examples

Find all classes in the JSII assembly that don't yet have any example code associated with them, and generate a synthetic example that shows how to instantiate the type.
TypeScript
1
star
66

cdk-ecs-service-extensions-go

Go
1
star
67

cdk-aws-sagemaker-role-manager-go

Go
1
star