• Stars
    star
    504
  • Rank 87,510 (Top 2 %)
  • Language
    Go
  • License
    Mozilla Public Li...
  • Created over 10 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Consul cross-DC KV replication daemon.

Consul Replicate

Build Status

This project provides a convenient way to replicate values from one Consul datacenter to another using the consul-replicate daemon.

The daemon consul-replicate integrates with Consul to perform cross-data-center K/V replication. This makes it possible to manage application configuration from a central data center, with low-latency asynchronous replication to other data centers, thus avoiding the need for smart clients that would need to write to all data centers and queue writes to handle network failures.


The documentation in this README corresponds to the master branch of Consul Replicate. It may contain unreleased features or different APIs than the most recently released version.

Please see the Git tag that corresponds to your version of Consul Replicate for the proper documentation.


Installation

  1. Download a pre-compiled, released version from the Consul Replicate releases page.

  2. Extract the binary using unzip or tar.

  3. Move the binary into $PATH.

To compile from source, please see the instructions in the contributing section.

Usage

For the full list of options:

$ consul-replicate -h

Command Line Flags

The CLI interface supports all options in the configuration file and visa-versa. Here are a few examples of common integrations on the command line.

Replicate all keys under "global" from the nyc1 data center:

$ consul-replicate \
  -prefix "global@nyc1"

Replicate all keys under "global" from the nyc1 data center, renaming the key to "default" in the replicated stores:

$ consul-replicate \
  -prefix "global@nyc1:default"

Replicate all keys under "global" from the nyc1 data center, but do not poll or watch for changes (just do it one time):

$ consul-replicate \
  -prefix "global@nyc1" \
  -once

Replicate all keys under "global" from the nyc1 data center, but exclude the global/private prefix:

$ consul-replicate \
  -prefix "global@nyc1" \
  -exclude "global/private" \
  -once

Configuration File Format

Configuration files are written in the HashiCorp Configuration Language. By proxy, this means the configuration is also JSON compatible.

# This denotes the start of the configuration section for Consul. All values
# contained in this section pertain to Consul.
consul {
  # This block specifies the basic authentication information to pass with the
  # request. For more information on authentication, please see the Consul
  # documentation.
  auth {
    enabled  = true
    username = "test"
    password = "test"
  }

  # This is the address of the Consul agent. By default, this is
  # 127.0.0.1:8500, which is the default bind and port for a local Consul
  # agent. It is not recommended that you communicate directly with a Consul
  # server, and instead communicate with the local Consul agent. There are many
  # reasons for this, most importantly the Consul agent is able to multiplex
  # connections to the Consul server and reduce the number of open HTTP
  # connections. Additionally, it provides a "well-known" IP address for which
  # clients can connect.
  address = "127.0.0.1:8500"

  # This is the ACL token to use when connecting to Consul. If you did not
  # enable ACLs on your Consul cluster, you do not need to set this option.
  #
  # This option is also available via the environment variable CONSUL_TOKEN.
  token = "abcd1234"

  # This controls the retry behavior when an error is returned from Consul.
  # Consul Replicate is highly fault tolerant, meaning it does not exit in the
  # face of failure. Instead, it uses exponential back-off and retry functions
  # to wait for the cluster to become available, as is customary in distributed
  # systems.
  retry {
    # This enabled retries. Retries are enabled by default, so this is
    # redundant.
    enabled = true

    # This specifies the number of attempts to make before giving up. Each
    # attempt adds the exponential backoff sleep time. Setting this to
    # zero will implement an unlimited number of retries.
    attempts = 12

    # This is the base amount of time to sleep between retry attempts. Each
    # retry sleeps for an exponent of 2 longer than this base. For 5 retries,
    # the sleep times would be: 250ms, 500ms, 1s, 2s, then 4s.
    backoff = "250ms"

    # This is the maximum amount of time to sleep between retry attempts.
    # When max_backoff is set to zero, there is no upper limit to the
    # exponential sleep between retry attempts.
    # If max_backoff is set to 10s and backoff is set to 1s, sleep times
    # would be: 1s, 2s, 4s, 8s, 10s, 10s, ...
    max_backoff = "1m"
  }

  # This block configures the SSL options for connecting to the Consul server.
  ssl {
    # This enables SSL. Specifying any option for SSL will also enable it.
    enabled = true

    # This enables SSL peer verification. The default value is "true", which
    # will check the global CA chain to make sure the given certificates are
    # valid. If you are using a self-signed certificate that you have not added
    # to the CA chain, you may want to disable SSL verification. However, please
    # understand this is a potential security vulnerability.
    verify = false

    # This is the path to the certificate to use to authenticate. If just a
    # certificate is provided, it is assumed to contain both the certificate and
    # the key to convert to an X509 certificate. If both the certificate and
    # key are specified, Consul Replicate will automatically combine them into an
    # X509 certificate for you.
    cert = "/path/to/client/cert"
    key  = "/path/to/client/key"

    # This is the path to the certificate authority to use as a CA. This is
    # useful for self-signed certificates or for organizations using their own
    # internal certificate authority.
    ca_cert = "/path/to/ca"

    # This is the path to a directory of PEM-encoded CA cert files. If both
    # `ca_cert` and `ca_path` is specified, `ca_cert` is preferred.
    ca_path = "path/to/certs/"

    # This sets the SNI server name to use for validation.
    server_name = "my-server.com"
  }
}

# This is the list of keys to exclude if they are found in the prefix. This can
# be specified multiple times to exclude multiple keys from replication.
exclude {
  source = "my-key"
}

# This is the signal to listen for to trigger a graceful stop. The default value
# is shown below. Setting this value to the empty string will cause Consul
# Replicate to not listen for any graceful stop signals.
kill_signal = "SIGINT"

# This is the log level. If you find a bug in Consul Replicate, please enable
# debug logs so we can help identify the issue. This is also available as a
# command line flag.
log_level = "warn"

# This is the maximum interval to allow "stale" data. By default, only the
# Consul leader will respond to queries; any requests to a follower will
# forward to the leader. In large clusters with many requests, this is not as
# scalable, so this option allows any follower to respond to a query, so long
# as the last-replicated data is within these bounds. Higher values result in
# less cluster load, but are more likely to have outdated data.
max_stale = "10m"

# This is the path to store a PID file which will contain the process ID of the
# Consul Replicate process. This is useful if you plan to send custom signals
# to the process.
pid_file = "/path/to/pid"

# This is the prefix and datacenter to replicate and the resulting destination.
prefix {
  source      = "global"
  datacenter  = "nyc1"
  destination = "default"
}

# This is the signal to listen for to trigger a reload event. The default value
# is shown below. Setting this value to the empty string will cause Consul
# Replicate to not listen for any reload signals.
reload_signal = "SIGHUP"

# This is the path in Consul to store replication and leader status.
status_dir = "service/consul-replicate/statuses"

# This block defines the configuration for connecting to a syslog server for
# logging.
syslog {
  # This enables syslog logging. Specifying any other option also enables
  # syslog logging.
  enabled = true

  # This is the name of the syslog facility to log to.
  facility = "LOCAL5"
}

# This is the quiescence timers; it defines the minimum and maximum amount of
# time to wait for the cluster to reach a consistent state before rendering a
# replicating. This is useful to enable in systems that have a lot of flapping,
# because it will reduce the the number of times a replication occurs.
wait {
  min = "5s"
  max = "10s"
}

Note that not all fields are required. If you are not logging to syslog, you do not need to specify a syslog configuration.

For additional security, tokens may also be read from the environment using the CONSUL_TOKEN environment variable. It is highly recommended that you do not put your tokens in plain-text in a configuration file.

Instruct Consul Replicate to use a configuration file with the -config flag:

$ consul-replicate -config "/my/config.hcl"

This argument may be specified multiple times to load multiple configuration files. The right-most configuration takes the highest precedence. If the path to a directory is provided (as opposed to the path to a file), all of the files in the given directory will be merged in lexical order, recursively. Please note that symbolic links are not followed.

Commands specified on the CLI take precedence over a config file!

Debugging

Consul Replicate can print verbose debugging output. To set the log level for Consul Replicate, use the -log-level flag:

$ consul-replicate -log-level info ...
<timestamp> [INFO] (cli) received redis from Watcher
<timestamp> [INFO] (cli) invoking Runner
# ...

You can also specify the level as trace:

$ consul-replicate -log-level trace ...
<timestamp> [DEBUG] (cli) creating Runner
<timestamp> [DEBUG] (cli) creating Consul API client
<timestamp> [DEBUG] (cli) creating Watcher
<timestamp> [DEBUG] (cli) looping for data
<timestamp> [DEBUG] (watcher) starting watch
<timestamp> [DEBUG] (watcher) all pollers have started, waiting for finish
<timestamp> [DEBUG] (redis) starting poll
<timestamp> [DEBUG] (service redis) querying Consul with &{...}
<timestamp> [DEBUG] (service redis) Consul returned 2 services
<timestamp> [DEBUG] (redis) writing data to channel
<timestamp> [DEBUG] (redis) starting poll
<timestamp> [INFO] (cli) received redis from Watcher
<timestamp> [INFO] (cli) invoking Runner
<timestamp> [DEBUG] (service redis) querying Consul with &{...}
# ...

FAQ

Q: Can I use this for master-master replication?
A: Master-master replication is not possible. A leader would never be elected.

Contributing

To build and install Consul Replicate locally, you will need to install the Docker engine:

Clone the repository:

$ git clone https://github.com/hashicorp/consul-replicate.git

To compile the consul-replicate binary for your local machine:

$ make dev

This will compile the consul-replicate binary into bin/consul-replicate as well as your $GOPATH and run the test suite.

If you want to compile a specific binary, set XC_OS and XC_ARCH or run the following to generate all binaries:

$ make bin

If you just want to run the tests:

$ make test

Or to run a specific test in the suite:

go test ./... -run SomeTestFunction_name

More Repositories

1

terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
Go
42,550
star
2

vault

A tool for secrets management, encryption as a service, and privileged access management
Go
30,858
star
3

consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
Go
28,256
star
4

vagrant

Vagrant is a tool for building and distributing development environments.
Ruby
26,132
star
5

packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
Go
15,086
star
6

nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
Go
14,809
star
7

terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
Go
9,761
star
8

raft

Golang implementation of the Raft consensus protocol
Go
7,383
star
9

serf

Service orchestration and management tool.
Go
5,692
star
10

hcl

HCL is the HashiCorp configuration language.
Go
5,236
star
11

go-plugin

Golang plugin system over RPC.
Go
4,874
star
12

terraform-cdk

Define infrastructure resources using programming constructs and provision them using HashiCorp Terraform
TypeScript
4,868
star
13

waypoint

A tool to build, deploy, and release any application on any platform.
Go
4,789
star
14

consul-template

Template rendering, notifier, and supervisor for @HashiCorp Consul and Vault data.
Go
4,750
star
15

terraform-provider-azurerm

Terraform provider for Azure Resource Manager
Go
4,528
star
16

otto

Development and deployment made easy.
HTML
4,282
star
17

golang-lru

Golang LRU cache
Go
4,221
star
18

boundary

Boundary enables identity-based access management for dynamic infrastructure.
Go
3,847
star
19

memberlist

Golang package for gossip based membership and failure detection
Go
3,303
star
20

go-memdb

Golang in-memory database built on immutable radix trees
Go
2,937
star
21

next-mdx-remote

Load mdx content from anywhere through getStaticProps in next.js
TypeScript
2,405
star
22

terraform-provider-google

Terraform Provider for Google Cloud Platform
Go
2,325
star
23

go-multierror

A Go (golang) package for representing a list of errors as a single error.
Go
2,029
star
24

yamux

Golang connection multiplexing library
Go
2,003
star
25

envconsul

Launch a subprocess with environment variables using data from @HashiCorp Consul and Vault.
Go
1,967
star
26

go-retryablehttp

Retryable HTTP client in Go
Go
1,702
star
27

terraform-provider-kubernetes

Terraform Kubernetes provider
Go
1,580
star
28

go-getter

Package for downloading things from a string URL using a variety of protocols.
Go
1,541
star
29

best-practices

HCL
1,490
star
30

go-version

A Go (golang) library for parsing and verifying versions and version constraints.
Go
1,459
star
31

go-metrics

A Golang library for exporting performance and runtime metrics to external metrics systems (i.e. statsite, statsd)
Go
1,453
star
32

setup-terraform

Sets up Terraform CLI in your GitHub Actions workflow.
JavaScript
1,352
star
33

terraform-guides

Example usage of HashiCorp Terraform
HCL
1,324
star
34

mdns

Simple mDNS client/server library in Golang
Go
1,020
star
35

terraform-provider-helm

Terraform Helm provider
Go
997
star
36

vault-guides

Example usage of HashiCorp Vault secrets management
Shell
990
star
37

go-immutable-radix

An immutable radix tree implementation in Golang
Go
926
star
38

vault-helm

Helm chart to install Vault and other associated components.
Shell
904
star
39

terraform-ls

Terraform Language Server
Go
896
star
40

vscode-terraform

HashiCorp Terraform VSCode extension
TypeScript
870
star
41

levant

An open source templating and deployment tool for HashiCorp Nomad jobs
Go
825
star
42

vault-k8s

First-class support for Vault and Kubernetes.
Go
697
star
43

terraform-exec

Terraform CLI commands via Go.
Go
675
star
44

consul-k8s

First-class support for Consul Service Mesh on Kubernetes
Go
665
star
45

terraform-aws-vault

A Terraform Module for how to run Vault on AWS using Terraform and Packer
HCL
659
star
46

terraform-github-actions

Terraform GitHub Actions
Shell
624
star
47

terraform-provider-vsphere

Terraform Provider for VMware vSphere
Go
617
star
48

raft-boltdb

Raft backend implementation using BoltDB
Go
585
star
49

nextjs-bundle-analysis

A github action that provides detailed bundle analysis on PRs for next.js apps
JavaScript
562
star
50

go-discover

Discover nodes in cloud environments
Go
562
star
51

next-mdx-enhanced

A Next.js plugin that enables MDX pages, layouts, and front matter
JavaScript
500
star
52

docker-vault

Official Docker images for Vault
Shell
492
star
53

terraform-provider-kubernetes-alpha

A Terraform provider for Kubernetes that uses dynamic resource types and server-side apply. Supports all Kubernetes resources.
Go
490
star
54

vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
Go
466
star
55

terraform-k8s

Terraform Cloud Operator for Kubernetes
Go
454
star
56

puppet-bootstrap

A collection of single-file scripts to bootstrap your machines with Puppet.
Shell
444
star
57

cap

A collection of authentication Go packages related to OIDC, JWKs, Distributed Claims, LDAP
Go
436
star
58

terraform-provider-vault

Terraform Vault provider
Go
431
star
59

damon

A terminal UI (TUI) for HashiCorp Nomad
Go
427
star
60

nomad-autoscaler

Nomad Autoscaler brings autoscaling to your Nomad workloads.
Go
424
star
61

consul-helm

Helm chart to install Consul and other associated components.
Shell
422
star
62

terraform-provider-azuread

Terraform provider for Azure Active Directory
Go
419
star
63

vault-ssh-helper

Vault SSH Agent is used to enable one time keys and passwords
Go
404
star
64

terraform-provider-scaffolding

Quick start repository for creating a Terraform provider
Go
402
star
65

terraform-aws-consul

A Terraform Module for how to run Consul on AWS using Terraform and Packer
HCL
401
star
66

docker-consul

Official Docker images for Consul.
Dockerfile
399
star
67

nomad-pack

Go
393
star
68

hil

HIL is a small embedded language for string interpolations.
Go
392
star
69

vault-action

A GitHub Action that simplifies using HashiCorp Vaultโ„ข secrets as build variables.
JavaScript
391
star
70

learn-terraform-provision-eks-cluster

HCL
389
star
71

terraform-plugin-sdk

Terraform Plugin SDK enables building plugins (providers) to manage any service providers or custom in-house solutions
Go
383
star
72

terraform-config-inspect

A helper library for shallow inspection of Terraform configurations
Go
380
star
73

hcl2

Former temporary home for experimental new version of HCL
Go
375
star
74

errwrap

Errwrap is a Go (golang) library for wrapping and querying errors.
Go
373
star
75

go-cleanhttp

Go
366
star
76

design-system

Helios Design System
TypeScript
358
star
77

logutils

Utilities for slightly better logging in Go (Golang).
Go
356
star
78

vault-ruby

The official Ruby client for HashiCorp's Vault
Ruby
336
star
79

vault-rails

A Rails plugin for easily integrating Vault secrets
Ruby
334
star
80

next-remote-watch

Decorated local server for next.js that enables reloads from remote data changes
JavaScript
325
star
81

waypoint-examples

Example Apps that can be deployed with Waypoint
PHP
325
star
82

go-hclog

A common logging package for HashiCorp tools
Go
319
star
83

vault-csi-provider

HashiCorp Vault Provider for Secret Store CSI Driver
Go
308
star
84

nomad-guides

Example usage of HashiCorp Nomad
HCL
281
star
85

consul-haproxy

Consul HAProxy connector for real-time configuration
Go
279
star
86

terraform-provider-google-beta

Terraform Provider for Google Cloud Platform (Beta)
Go
265
star
87

consul-esm

External service monitoring for Consul
Go
262
star
88

http-echo

A tiny go web server that echos what you start it with!
Makefile
257
star
89

terraform-provider-awscc

Terraform AWS Cloud Control provider
HCL
256
star
90

terraform-aws-nomad

A Terraform Module for how to run Nomad on AWS using Terraform and Packer
HCL
254
star
91

faas-nomad

OpenFaaS plugin for Nomad
Go
252
star
92

go-sockaddr

IP Address/UNIX Socket convenience functions for Go
Go
250
star
93

terraform-foundational-policies-library

Sentinel is a language and framework for policy built to be embedded in existing software to enable fine-grained, logic-based policy decisions. This repository contains a library of Sentinel policies, developed by HashiCorp, that can be consumed directly within the Terraform Cloud platform.
HCL
233
star
94

nomad-driver-podman

A nomad task driver plugin for sandboxing workloads in podman containers
Go
226
star
95

vagrant-vmware-desktop

Official provider for VMware desktop products: Fusion, Player, and Workstation.
Go
225
star
96

go-tfe

HCP Terraform/Enterprise API Client/SDK in Golang
Go
224
star
97

nomad-pack-community-registry

A repo for Packs written and maintained by Nomad community members
HCL
215
star
98

boundary-reference-architecture

Example reference architecture for a high availability Boundary deployment on AWS.
HCL
211
star
99

terraform-plugin-framework

A next-generation framework for building Terraform providers.
Go
204
star
100

vault-plugin-auth-kubernetes

Vault authentication plugin for Kubernetes Service Accounts
Go
192
star