Vault on GCE Terraform Module

Modular deployment of Vault on Google Compute Engine.

This module is versioned and released on the Terraform module registry. Look for the tag that corresponds to your version for the correct documentation.

• Vault HA - Vault is configured to run in high availability mode with Google Cloud Storage. Choose a vault_min_num_servers greater than 0 to enable HA mode.

• Production hardened - Vault is deployed according to applicable parts of the production hardening guide.

  • Traffic is encrypted with end-to-end TLS using self-signed certificates which can be generated or supplied (see Managing TLS below).

  • Vault is the main process on the VMs, and Vault runs as an unprivileged user (vault:vault) on the VMs under systemd.

  • Outgoing Vault traffic happens through a restricted NAT gateway through dedicated IPs for logging and monitoring. You can further restrict outbound access with additional firewall rules.

  • The Vault nodes are not publicly accessible. They do have SSH enabled, but require a bastion host on their dedicated network to access. You can disable SSH access entirely by setting ssh_allowed_cidrs to the empty list.

  • Swap is disabled (the default on all GCE VMs), reducing the risk that in-memory data will be paged to disk.

  • Core dumps are disabled.

  The following values do not represent Vault's best practices and you may wish to change their defaults:

  • Vault is publicly accessible from any IP through the load balancer. To limit the list of source IPs that can communicate with Vault nodes, set vault_allowed_cidrs to a list of CIDR blocks.

  • Auditing is not enabled by default, because an initial bootstrap requires you to initialize the Vault. Everything is pre-configured for when you're ready to enable audit logging, but it cannot be enabled before Vault is initialized.

  • Auto-unseal - Vault is automatically unsealed using the built-in Vault 1.0+ auto-unsealing mechanisms for Google Cloud KMS. The Vault servers are not automatically initialized, providing a clear separation.

  • Isolation - The Vault nodes are not exposed publicly. They live in a private subnet with a dedicated NAT gateway.

  • Audit logging - The system is setup to accept Vault audit logs with a single configuration command. Vault audit logs are not enabled by default because you have to initialize the system first.

Usage

1. Add the module definition to your Terraform configurations:

   module "vault" {
     source         = "terraform-google-modules/vault/google"
     project_id     = var.project_id
     region         = var.region
     kms_keyring    = var.kms_keyring
     kms_crypto_key = var.kms_crypto_key
   }

   Make sure you are using version pinning to avoid unexpected changes when the module is updated.

2. Execute Terraform:

   $ terraform apply
   

3. Configure your local Vault binary to communicate with the Vault server:

   $ export VAULT_ADDR="$(terraform output vault_addr)"
   $ export VAULT_CACERT="$(pwd)/ca.crt"
   

4. Wait for Vault to start. Here's a script or you can wait ~2 minutes.

   (while [[ $count -lt 60 && "$(vault status 2>&1)" =~ "connection refused" ]]; do ((count=count+1)) ; echo "$(date) $count: Waiting for Vault to start..." ; sleep 2; done && [[ $count -lt 60 ]])
   [[ $? -ne 0 ]] && echo "ERROR: Error waiting for Vault to start" && exit 1
   

5. Initialize the Vault cluster, generating the initial root token and unseal keys:

   $ vault operator init \
       -recovery-shares 5 \
       -recovery-threshold 3
   

   The Vault servers will automatically unseal using the Google Cloud KMS key created earlier. The recovery shares are to be given to operators to unseal the Vault nodes in case Cloud KMS is unavailable in a disaster recovery. They can also be used to generate a new root token. Distribute these keys to trusted people on your team (like people who will be on-call and responsible for maintaining Vault).

   The output will look like this:

   Recovery Key 1: 2EWrT/YVlYE54EwvKaH3JzOGmq8AVJJkVFQDni8MYC+T
   Recovery Key 2: 6WCNGKN+dU43APJuGEVvIG6bAHA6tsth5ZR8/bJWi60/
   Recovery Key 3: XC1vSb/GfH35zTK4UkAR7okJWaRjnGrP75aQX0xByKfV
   Recovery Key 4: ZSvu2hWWmd4ECEIHj/FShxxCw7Wd2KbkLRsDm30f2tu3
   Recovery Key 5: T4VBvwRv0pkQLeTC/98JJ+Rj/Zn75bLfmAaFLDQihL9Y
   
   Initial Root Token: s.kn11NdBhLig2VJ0botgrwq9u
   

   Save this initial root token and do not clear your history. You will need this token to continue the tutorial.

Managing TLS

If, like many orgs, you manage your own self-signed TLS certificates, you likely will not want them managed by Terraform. Additionally this poses a security risk since the private keys will be stored in plaintext in the terraform.tfstate file. To use your own certificates, set manage_tls = false. Then before you apply this module, you'll need to have your certificates prepared. An example instantiation would look like this:

module "vault" {
  source  = "terraform-google-modules/vault/google"
  ...

  # Manage our own TLS Certs so the private keys don't
  # end up in Terraform state
  manage_tls        = false
  vault_tls_bucket  = google_storage_bucket.vault_tls.name
  vault_tls_kms_key = google_kms_crypto_key.vault_tls.self_link

  # These are the default values shown here for clarity
  vault_ca_cert_filename  = "ca.crt"
  vault_tls_cert_filename = "vault.crt"
  vault_tls_key_filename  = "vault.key.enc"
}

To store your keys, you’ll need to create at minimum the 3 files that are shown above at the root of the TLS bucket specified by vault_tls_bucket, but their filenames and paths can be overridden using the vault_*_filename variables shown above.

• CA Certificate. This file should be the PEM formatted CA certificate that the Vault server certificate is created from
• Vault Server Certificate. This file should correspond to the Vault Private Key also stored on the Vault hosts to terminate the TLS connection.
• Vault Private Key. As you’ll notice, this key has the .enc file extension denoting it is encrypted. When the Vault host spins up, it will fetch all these certificates and the key and on the key it will use the specified TLS KMS Key to Base64 decode and then decrypt the private key before storing it on the filesystem.

Assuming you have these files locally that have been generated by OpenSSL or some other CA, you can store them with the following commands:

gcloud kms encrypt \
  --project=${PROJECT} \
  --key=${KMS_KEY} \
  --plaintext-file=vault.key \
  --ciphertext-file=- | base64 > "vault.key.enc"

for file in vault.key.enc ca.crt vault.crt; do
  gsutil cp $file gs://$TLS_BUCKET/$file
done

Inputs

[
  "cloudkms.googleapis.com",
  "cloudresourcemanager.googleapis.com",
  "compute.googleapis.com",
  "iam.googleapis.com",
  "logging.googleapis.com",
  "monitoring.googleapis.com"
]

[
  "roles/logging.logWriter",
  "roles/monitoring.metricWriter",
  "roles/monitoring.viewer"
]

[
  "roles/storage.legacyBucketReader",
  "roles/storage.objectAdmin"
]

[
  "0.0.0.0/0"
]

list(object({
    action = map(object({
      type          = string,
      storage_class = string
    })),
    condition = map(object({
      age                   = number,
      created_before        = string,
      with_state            = string,
      is_live               = string,
      matches_storage_class = string,
      num_newer_versions    = number
    }))
  }))

object({
    common_name         = string,
    organization        = string,
    organizational_unit = string,
    street_address      = list(string),
    locality            = string,
    province            = string,
    country             = string,
    postal_code         = string,
  })

{
  "common_name": "Example Inc. Root",
  "country": "US",
  "locality": "The Intranet",
  "organization": "Example, Inc",
  "organizational_unit": "Department of Certificate Authority",
  "postal_code": "95559-1227",
  "province": "CA",
  "street_address": [
    "123 Example Street"
  ]
}

[
  "vault.example.net"
]

[
  "127.0.0.1"
]

[
  "0.0.0.0/0"
]

Outputs

Additional permissions

The default installation includes the most minimal set of permissions to run Vault. Certain plugins may require more permissions, which you can grant to the service account using service_account_project_additional_iam_roles:

GCP auth method

The GCP auth method requires the following additional permissions:

roles/iam.serviceAccountKeyAdmin

GCP secrets engine

The GCP secrets engine requires the following additional permissions:

roles/iam.serviceAccountKeyAdmin
roles/iam.serviceAccountAdmin

GCP KMS secrets engine

The GCP secrets engine permissions vary. There are examples in the secrets engine documentation.

Logs

The Vault server logs will automatically appear in Stackdriver under "GCE VM Instance" tagged as "vaultproject.io/server".

The Vault audit logs, once enabled, will appear in Stackdriver under "GCE VM Instance" tagged as "vaultproject.io/audit".

Sandboxes & Terraform Cloud

When running in a sandbox such as Terraform Cloud, you need to disable filesystem access. You can do this by setting the following variables:

# terraform.tfvars
tls_save_ca_to_disk = false

FAQ

• I see unhealthy Vault nodes in my load balancer pool!

  This is the expected behavior. Only the active Vault node is added to the load balancer to prevent redirect loops. If that node loses leadership, its health check will start failing and a standby node will take its place in the load balancer.

• Can I connect to the Vault nodes directly?

  Connecting to the vault nodes directly is not recommended, even if on the same network. Always connect through the load balancer. You can alter the load balancer to be an internal-only load balancer if needed.