• Stars
    star
    105
  • Rank 321,286 (Top 7 %)
  • Language HCL
  • License
    MIT License
  • Created over 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Terraform module for creating a simple private Terraform registry in AWS with DynamoDB

Terraform Private Registry for AWS

This Terraform module establishes a private registry for Terraform, allowing you to publish your own modules in a location you control independent of Terraform's public registry at registry.terraform.io.

Terraform module addresses can include an optional hostname part which allows them to be downloaded from services other than the public registry:

module "awesomeapp" {
  source = "tf.example.com/awesomecorp/awesomeapp/aws"
}

The module in this repository provides the API endpoints necessary to provide such a module source hostname. Terraform's documented registry HTTP API is implemented via Amazon API Gateway relaying requests to a DynamoDB table that contains a simple index of modules. The module packages themselves can be stored at any non-registry module source address supported by Terraform, including in an S3 bucket with standard AWS authentication.

This module requires Terraform 0.12 or newer.

If using this module in production, be sure to select a specific version via a version constraint argument in your module block to avoid surprising changes during upgrades. This module uses semantic versioning, and so new minor releases may introduce additional features that may lead to additional cost.

For the moment this module remains EXPERIMENTAL. Once we've heard good feedback about how it behaves in real-world situations it will be blessed with a non-experimental version number (1.0 or greater).

Simple Usage

When called with no arguments, this module will create a registry API with no access control backed by a DynamoDB table called TerraformRegistry-modules:

module "tf_registry" {
  source = "apparentlymart/tf-registry/aws"
}

output "rest_api_id" {
  value = module.tf_registry.rest_api_id
}

output "services" {
  value = module.tf_registry.services
}

Since the registry is just an index for module packages stored elsewhere, it may be acceptable in some environments to allow unauthenticated access to the registry API while protecting access to the packages themselves, and so the above may be sufficient to get started.

After the initial creation of the registry, the DynamoDB table will be empty. Use any normal strategy for population of the registry, such as the DynamoDB management console, the AWS CLI, or a custom program using an AWS SDK. For the sake of example here, we'll create an alias for the HashiCorp Consul module in the public registry using the AWS CLI:

aws dynamodb put-item \
  --table-name TerraformRegistry-modules \
  --item '{
      "Id": {"S":"hashicorp/consul/aws"},
      "Version": {"S":"0.4.4"},
      "Source": {"S":"https://api.github.com/repos/hashicorp/terraform-aws-consul/tarball/v0.4.4//*?archive=tar.gz"}
  }'

The {"S":...} objects here are the DynamoDB convention for indicating a string value. The Id and Version attributes together form the primary key for the table, and Source specifies a module source address where the module package can be downloaded. In this case, we indicate a .tar.gz archive of a tag from a repository on GitHub.

The terraform apply log should include a value for the services output indicated in the configuration above, which will look something like this:

Outputs:

rest_api_id = b9h60hion6
services = {
  "modules.v1" = "https://b9h60hion6.execute-api.us-west-2.amazonaws.com/live/modules.v1/"
}

This map value is a service discovery document for Terraform's service discovery protocol. For normal use it would be necessary to publish a JSON version of this document at /.well-known/terraform.json on an HTTPS server running at the hostname that will be used to install modules, but for initial testing we can use an override configuration in the Terraform CLI config file (not your infrastructure configuration in .tf files):

host "tf.example.com" {
  services = {
    "modules.v1" = "https://b9h60hion6.execute-api.us-west-2.amazonaws.com/live/modules.v1/"
  }
}

With this block in place, Terraform will use this hard-coded map instead of trying to request a discovery document over the network. From another separate Terraform configuration we should then be able to request the Consul module via this private registry:

module "consul" {
  source = "tf.example.com/hashicorp/consul/aws"

  # ...
}

The module installer in terraform init should then be able to download the module by first requesting its package location from our private registry.

The remaining sections of this README will cover some other options and configuration details.

Customizing AWS Object Names

By default, this module creates various objects across a number of different AWS services using names starting with TerraformRegistry. You can customize this prefix by setting the name_prefix argument:

module "tf_registry" {
  source = "apparentlymart/tf-registry/aws"

  name_prefix = "AnotherTerraformRegistry"
}

Changing this name after the module is initially created requires re-creating all remote objects, including the underlying DynamoDB table. That means any data in that table would be lost and must be restored from backup.

Publishing Module Packages in Amazon S3

If all of the users or compute instances where you run the Terraform CLI have access to suitable AWS credentials, it may be convenient to publish your private modules as archives in a protected Amazon S3 bucket.

To do this, create a zip file containing the module source code with the main (top-level) module at the root of the file. Place this file in an S3 bucket with suitable permissions. The naming scheme for objects in this bucket is up to you, but we'd suggest using a systematic scheme like namespace/module/provider/module_provider_version.zip, giving (for example) hashicorp/consul/aws/consul_aws_v0.4.4.zip.

When you record these in the registry's DynamoDB table, use the s3:: prefix followed by an S3 URL to instruct Terraform to use the S3 authentication protocol when credentials are available:

aws dynamodb put-item \
  --table-name TerraformRegistry-modules \
  --item '{
    "Id": {"S":"hashicorp/consul/aws"},
    "Version": {"S":"0.4.4"},
    "Source": {"S":"s3::https://s3-us-west-2.amazonaws.com/hashicorp/consul/aws/consul_aws_v0.4.4.zip"}
  }'

For more information on how Terraform retrieves modules from S3 buckets, and in particular where it will look to obtain AWS credentials for the request, see the S3 Bucket source documentation.

The module registry protocol itself cannot support AWS-style authentication, so you must either allow unauthenticated requests to the registry endpoints (which will disclose only metadata about the modules) or configure the registry to use a Lambda authorizer, as described in a later section.

Re-deploying the API after Changes

Most customizations in the following sections cause changes to the API Gateway configuration. Due to the design of API Gateway, such changes must be explicitly re-deployed after Terraform has finished applying them.

To do this, look for the rest_api_id output value in the terraform apply output and insert as the --rest-api-id value in the following AWS CLI command line:

aws apigateway create-deployment \
    --rest-api-id b9h60hion6 \
    --stage-name live

This will be necessary after any terraform apply whose plan includes changes to resources with types starting with aws_api_gateway_.

Publishing the Discovery Document

If you already have an HTTPS server running on a suitable hostname then you can make your private registry accessible via that hostname by publishing a JSON version of the discovery map at /.well-known/terraform.json on that server:

{
  "modules.v1": "https://b9h60hion6.execute-api.us-west-2.amazonaws.com/live/modules.v1/"
}

For example, to make the above example hostname tf.example.com work without local overrides, the discovery document would need to be published at https://tf.example.com/.well-known/terraform.json.

Automatic Discovery Document

If you'd rather keep the whole registry deployment self-contained, this tf-registry module can optionally publish itself at a hostname of your choice and host its own JSON discovery document like the above.

For this to work you will first need to create and verify an AWS Certificate Manager certificate in the same region where this module is being deployed. Such a certificate can be provisioned automatically in Terraform if your domain is hosted in Route53.

With the certificate created, the optional friendly_hostname argument for this module calls for the hostname mapping to be configured:

module "tf_registry" {
  source = "apparentlymart/tf-registry/aws"

  friendly_hostname = {
    host                = aws_acm_certificate.cert.domain_name
    acm_certificate_arn = aws_acm_certificate_validation.cert.certificate_arn
  }
}

When friendly_hostname is set, the module will additionally configure API Gateway to serve the registry API and an automatically-generated discovery document at the given hostname and with the given certificate.

To complete this configuration, you'll need to create an entry in your DNS zone to point requests at the registry API. If you're using Route53 then you can create an alias record using aws_route53_record:

resource "aws_route53_record" "tf" {
  zone_id = var.your_zone_id

  name = aws_acm_certificate.cert.domain_name
  type = "A"
  alias {
    name    = module.tf_registry.dns_alias.hostname
    zone_id = module.tf_registry.dns_alias.route53_zone_id
  }
}

After giving time for the changes to propagate, you should be able to request the discovery document at your hostname using curl, such as the following example continuing to use tf.example.com:

$ curl https://tf.example.com/.well-known/terraform.json
{
  "modules.v1":"/modules.v1/"
}

Once this works, Terraform should be able to find modules via that hostname. If you added a host block to the Terraform CLI configuration during the "Simple Usage" steps above, remember to remove it to allow Terraform to do normal discovery over the network.

Access Control

Terraform CLI supports bearer-token authentication credentials when making API requests. Credentials are configured on a per-hostname basis and apply to all services at that hostname. An authentication token for a particular hostname can be configured using a credentials block in the CLI configuration.

This module has no built-in support for authentication, but you can add token authentication by writing an AWS Lambda-based authorizer function that checks submitted API tokens in any way that makes sense for your environment.

Once you have written an authorizer function, you can enable it for your registry using the optional lambda_authorizer argument:

module "tf_registry" {
  source = "apparentlymart/tf-registry/aws"

  lambda_authorizer = {
    type          = "TOKEN"
    function_name = aws_lambda_function.auth.function_name
  }
}

The type attribute can be set to either TOKEN or REQUEST, depending on which calling convention the authorizer function is expecting. If TOKEN is selected, the function recieves the content of the Authorization HTTP header, where Terraform CLI places any configured bearer token.

When writing your authorizer function, remember that the Authorization header value has a Bearer prefix to indicate that Terraform is using bearer token authentication. Your function must check for this and then strip it off before checking whether the rest of the header value is a valid token.

The details of writing an authorizer function are beyond the scope of this readme. For more information, see Introducing custom authorizers in Amazon API Gateway from the AWS Compute Blog.

Advanced Scenarios Using the Submodules

The top-level module in this package is intended as a good set of defaults for a simple registry deployment. If you need more control in your environment, you may prefer to use directly the sub-modules that the top-level module is constructed from, which can be composed together in different ways to make different tradeoffs:

  • modules-store manages the DynamoDB table that stores the module registry index.
  • modules.v1 implements the version 1 HTTP API that Terraform CLI expects, using API Gateway against a given DynamoDB table which is assumed to be one created by the modules-store module.
  • disco adds a /.well-known/terraform.json discovery document to the root of any given API Gateway REST API.

None of these sub-modules create an API Gateway REST API themselves, so you can write your own module that creates and configures a REST API as meets your needs and then use these sub-modules to populate it with the modules API functionality and, if desired, a discovery document.

More Repositories

1

terraform-clean-syntax

A simple tool for Terraform language syntax cleanup, extending terraform fmt
Go
159
star
2

go-cidr

Go library for various manipulations of CIDR netmasks and their associated addresses
Go
113
star
3

angularjs-viewhead

Change the HTML title and head elements on a per-view basis
HTML
99
star
4

livejournal

LiveJournal Server Source Code (stale history from before it went closed-source)
Perl
78
star
5

terraform-provider-testing

An experimental Terraform provider to assist in writing tests for Terraform modules
Go
69
star
6

terraform-simple-registry

Go
64
star
7

terrafy

An experimental little tool to automate importing collections of things into Terraform. Bork impult.
Go
41
star
8

terraform-provider-bash

Terraform utility provider for constructing bash scripts that use data from a Terraform module
Go
34
star
9

terraform-credentials-env

Terraform credentials helper for supplying credentials in environment variables
Go
31
star
10

activity-streams-python

Activity Streams Parser for Python
Python
28
star
11

activity-streams-specs

Specifications for publishing and consuming Activity Streams
28
star
12

node-angularcontext

Use AngularJS in Node applications
JavaScript
24
star
13

go-openvpn-mgmt

[WIP] Go client library for OpenVPN's management protocol
Go
22
star
14

proximobus

An alternative interface to the NextBus Public API
Python
20
star
15

python-nextbus

A Python library for connecting to the NextBus public API
Python
18
star
16

libdanga-socket-anyevent-perl

Danga::Socket reimplementation in terms of AnyEvent
Perl
13
star
17

terraform-sdk

Experimental next-generation Terraform SDK (prototype)
Go
11
star
18

libnet-openid-perl

OpenID libraries for Perl
Perl
10
star
19

verilog-vga-simulator

Verilog VPI VGA Simulator using SDL
C
10
star
20

activity-streams-tester

Google App Engine app which parses Activity Streams and renders what it found
Python
10
star
21

terraform-provider-javascript

A Terraform provider for computing values using JavaScript
Go
10
star
22

68k-computer

A homebrew computer design based on the MC68SEC000 CPU
C
10
star
23

go-versions

Version-wrangling library for Go
Go
9
star
24

python-stl

Python library for parsing and producing STL files, in both ASCII and binary formats. No longer maintained.
Python
9
star
25

python-camlistore

Python interface to Camlistore
Python
8
star
26

padstone

Go
8
star
27

hcltemplate

CLI filter for rendering JSON objects using the HCL template language
Go
7
star
28

terraform-provider

Client library for using Terraform providers without using Terraform itself
Go
7
star
29

perl-anyevent-websocket

WebSocket implementation for AnyEvent
Perl
7
star
30

libhttp-request-multi-perl

HTTP::Request::Multi - Perl implementation of Batch HTTP Requests
7
star
31

camlistore-aws

Packer and Terraform configs to build and run camlistore server in AWS
Shell
6
star
32

activity-streams-perl

Activity Streams Library for Perl
Perl
6
star
33

atom-media-specs

Specifications for extensions to Atom to support media use-cases
6
star
34

go-rundeck-api

Go client for the Rundeck HTTP API
Go
6
star
35

poe-danga-socket

POE Loop implementation that targets Danga::Socket
Perl
5
star
36

android-nextbus

This does something secret
Java
5
star
37

http-batch-specs

Specification for a protocol for batch requests to RESTful APIs
5
star
38

sf-ped-map

OpenStreetMap-based San Francisco Map for Pedestrians
Python
5
star
39

dotcloud-serviceconfig

Simple mechanism to help your dotcloud services find each other
Python
5
star
40

terraform-schema-go

Go library for modelling Terraform schemas
Go
5
star
41

camlistore-git

Toy implementation of git "server" that stores repository data in a camlistore instance.
Python
5
star
42

json-syn-specs

Specifications for the representation of syndication objects in JSON
4
star
43

media-center

Yet Another Media Center?
JavaScript
4
star
44

go-rst

[WIP] ReStructuredText Parser for golang
Go
4
star
45

brightsquare

A stupid little web app for updating both BrightKite and FourSquare at the same time
Perl
4
star
46

go-hcl-overlay

HCL helper for applying overlays to configuration, such as command line arguments
Go
4
star
47

libjson-streaming-writer-perl

JSON::Streaming::Writer - Generate JSON output without assembling data structure in memory
Perl
4
star
48

terraform-filter-vars

A small utility for filtering .tfvars files to include only definitions related to a particular Terraform module
Go
4
star
49

rust-linux

Lightweight safe wrappers around direct Linux system calls
Rust
3
star
50

terraform-aws-serverless-webapp

Terraform module for deploying "serverless" applications to AWS Lambda and Amazon API Gateway
HCL
3
star
51

domain-federation-protocol-specs

Specifications for a Domain Federation Protocol
3
star
52

activitystreams-old-registry-tools

Prototype tools for managing the prototype registry
Perl
3
star
53

raspi-jtag-hat

C
3
star
54

python-promise

Batchable, Coalescable, Concurrent Promises in Python
Python
3
star
55

pish

Python
3
star
56

go-protohcl

[Prototype!] Represent HCL schemas using protobuf message descriptors
Go
3
star
57

quantumkeep

Key-value store with version control
Python
3
star
58

epub-recipes

Makes an ePub book from some recipes parsed from simple text files
Perl
3
star
59

s2

S2 Template System
Perl
3
star
60

terraform-plan-output-grammar

A TextMate-style grammar for the human-oriented Terraform plan diff output
3
star
61

libjson-acrobatic-perl

Perl implementation of Acrobatic JSON
Perl
3
star
62

libwebservice-typepad-perl

Perl library for accessing Six Apart's TypePad JSON API
Perl
3
star
63

atom-crossposting-specs

Specification to allow feed publishers to declare cross-post duplicates that they create
2
star
64

hclcalc

Go
2
star
65

activity-api

Activity API
2
star
66

libutil-future-perl

Smart batch-loader using "Futures"
Perl
2
star
67

libsyntax-highlighting-json-perl

JSON Syntax Highlighting in Perl
2
star
68

libutil-datathing-perl

Perl
2
star
69

libjson-streaming-reader-perl

Parse JSON strings in a streaming fashion
Perl
2
star
70

juniper-vpn-launcher

An API for launching the Juniper VPN ncsvc client
2
star
71

led-alpha-sign-msg-server

A simple HTTP interface to showing and hiding priority messages on an Adaptive Micro Systems LED sign
Perl
2
star
72

libutil-task-perl

Coalescable Tasks
Perl
2
star
73

libhardware-hal-perl

Hardware::HAL - Perl API for freedesktop.org HAL
2
star
74

python-acrobaticjson

Acrobatic JSON implementation for Python
Python
2
star
75

pubsubhubbub-json

PubSubHubbub for JSON
2
star
76

python-canal

Simple graph processing library built on greenlets
Python
2
star
77

nextbus-aggregator

Some gunk for aggregating and optimizing queries to NextBus
Perl
2
star
78

led-matrix-fonts

Some bitmap fonts that are small enough to use on an LED matrix
Python
2
star
79

federationsandbox

An AppEngine app that provides a sandbox and testing ground for DFP implementations
Python
2
star
80

hcl-compat-bridge

WIP bridge from the HCL 2 API to the HCL 1 and HIL parsers/evaluators
Go
2
star
81

reviewhoo

Little tool that suggests who you might ask to review a particular git changeset
Go
2
star
82

terraform-aws-service-roles

A Terraform module for creating AWS IAM service roles
HCL
2
star
83

blogwhack

A shim to whack together multiple blogs served from different services
2
star
84

python-conjurer

A simple utility for mapping SQLAlchemy resultsets to object instances in a declarative way
Python
2
star
85

go-shquot

Go
2
star
86

wordclock

A clock that tells the time in words
KiCad Layout
2
star
87

libutil-task-httprequest-perl

Util::Task implementation for doing HTTP requests
Perl
2
star
88

py2bsc

Translates a subset of Python into BrightScript to enable more sensible Roku/BrightSign programming
Python
2
star
89

genext2fs-tar-index

Generate device/metadata indices for genext2fs
Go
2
star
90

elfbin

Tool (and Rust library) for creating ELF object files with symbols whose data comes from other files
Rust
2
star
91

nextbus-trip

Uses NextBus predictions to give a whole-trip estimate
Perl
2
star
92

python-tfplugin

EXPERIMENTAL+WIP library for implementing Terraform plugins in Python
Python
2
star
93

rust-lpc81x-hal

Rust HAL crate for lpc81x-series microcontrollers
Rust
2
star
94

proximo-web

A HTML5 UI for bus predictions built on ProximoBus
JavaScript
2
star
95

go-ssd1306

Go interface to drive SSD1306 OLED controllers
Go
2
star
96

terraform

Go
2
star
97

camlistore-debian

Debian packaging for Camlistore
Go
2
star
98

go-gpio

Hardware-agnostic GPIO interfaces for Go
Go
2
star
99

ng-anon-module

Create AngularJS modules without polluting the global module registry
JavaScript
1
star
100

apparentlymart.github.com

1
star