• This repository has been archived on 27/Jun/2019
  • Stars
    star
    279
  • Rank 147,939 (Top 3 %)
  • Language
    Go
  • License
    Mozilla Public Li...
  • Created over 10 years ago
  • Updated about 10 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Consul HAProxy connector for real-time configuration

DEPRECATED: consul-haproxy

Deprecated! This project is deprecated. Consul HAProxy has been replaced by Consul Template. This repository is kept for history and legacy purposes. Please use Consul Template instead.


This project provides consul-haproxy, a daemon for dynamically configuring HAProxy using data from Consul.

The daemon watches any number of backends for updates, and when appropriate renders an configuration template for HAProxy and then invokes a reload command, which can gracefully reload HAProxy. This allows for a zero-downtime reload of HAProxy which is populated by Consul.

Download & Compilation

Download a release from the releases page, or compile from source:

$ make
$ ./bin/consul-haproxy -h

Usage

The consul-haproxy command takes a number of CLI flags:

  • -addr - Provides the HTTP address of a Consul agent. By default this assumes a local agent at "127.0.0.1:8500".

  • -backend - Backend specification. Can be provided multiple times. The specification of a backend is documented below.

  • -dry - Dry run. Emit config file to stdout.

  • -f - Path to config file, overwrites CLI flags. The format of the file is documented below.

  • -in- Path to a template file. This is the template that is rendered to generate the configuration file at -out. It uses the Golang templating system. Docs for that are here. Can be provided multiple times. If specified multiple times, specify the same number of paths with -out.

  • -out - Path to output configuration file. This path must be writable by consul-haproxy or the file cannot be updated. This can be specified multiple times.

  • -reload - Command to invoke to reload configuration. This command can be any executable, and should be used to reload HAProxy. This is invoked only after the configuration file is updated.

  • -quiet - Quiet specifies a duration of time to wait for no updates before writing out the new configuration. This allows for waiting until a service stabilizes to prevent many different reloads.

  • -max-wait - Max wait is used to limit how waiting is done for a quiet period before forcing a reload. This defaults to 4x the -quiet value. As an example, if -quiet=30s but the backends are constantly flapping, a refresh will be forced after 2 minutes.

In addition to using CLI flags, consul-haproxy can be configured using a file given the -f flag. A configuration file overrides any values given by the CLI unless otherwise specified. The configuration file should be a JSON object with the following keys:

  • address - Same as -addr CLI flag.
  • backends - A list of backend specifications. This is merged with any backends provided via the CLI.
  • dry_run - Same as -dry CLI flag.
  • paths - Same as -out CLI flag. . This value should be a list of paths and is merged with any paths provided via the CLI.
  • reload_command - Same as -reload CLI flag.
  • templates - Same as -in CLI flag. This value should be a list of templates and is merged with any paths provided via the CLI.
  • quiet - Same as -quiet CLI flag.
  • max_wait - Same as -max-wait CLI flag.

Backend Specification

One of the key configuration values to consul-haproxy is the backends that it should monitor. These are the service entries that are watched for changes and used to populate the configuration file. The syntax for a backend is:

backend_name=tag.service@datacenter:port

The specification provides a variable name for the template backend_name, which is defined as the entries that match the given tag and service for a specific datacenter. A port can also be provided, which overrides the port specified by the service. The tag, datacenter, and port are all optional and can be omitted.

Below are a few examples:

  • app=release.webapp - This defines a template variable app which watches for the webapp service, filtering on the release tag.

  • db=mysql@east-aws:5500 - This defines a template variable db which watches for the mysql service in the east-aws datacenter, using port 5500.

A useful features is the ability to specify multiple backends with the same variable name. This causes the nodes to be merged. This can be used to merge nodes with various tags, or different datacenters together. As an example, we can define:

app=webapp@dc1
app=webapp@dc2
app=webapp@dc3

This backend specification sets app variable to be the union of the servers in the dc1, dc2, and dc3 datacenters.

Template Language

The template language is the Golang text/template package, which is fully documented here. However, the basic usage is quite simple.

As an example, suppose we define a simple backends with:

app=webapp@east-aws:8000
cache=redis

We might provide a very basic template like:

global
    daemon
    maxconn 256

defaults
    mode tcp
    timeout connect 5000ms
    timeout client 60000ms
    timeout server 60000ms

listen http-in
    bind *:80{{range .app}}
    {{.}} maxconn 32{{end}}

listen cache-in
    bind *:4444{{range .cache}}
    {{.}} maxconn 64{{end}}

This will populate the http-in block with the servers in the app backend, and the cache-in with the servers in the cache backend. This template will be re-rendered when any of those servers changing, allowing for dynamic updates.

Example

We run the example below against our NYC demo server. This lets you set quickly test consul-haproxy.

First lets create a simple template:

global
    daemon
    maxconn 256

defaults
    mode tcp
    timeout connect 5000ms
    timeout client 60000ms
    timeout server 60000ms

listen http-in
    bind *:8000{{range .c}}
    {{.}}{{end}}

Now, we can run the following to get our output configuration:

consul-haproxy -addr=demo.consul.io -in in.conf -backend "c=consul@nyc3:80" -backend "c=consul@sfo1:80" -dry

When this runs, we should see something like the following:

global
    daemon
    maxconn 256

defaults
    mode tcp
    timeout connect 5000ms
    timeout client 60000ms
    timeout server 60000ms

listen http-in
    bind *:8000
    server 0_nyc3-consul-1_consul 192.241.159.115:80
    server 0_nyc3-consul-2_consul 192.241.158.205:80
    server 0_nyc3-consul-3_consul 198.199.77.133:80
    server 1_sfo1-consul-2_consul 162.243.155.82:80
    server 1_sfo1-consul-1_consul 107.170.195.169:80
    server 1_sfo1-consul-3_consul 107.170.195.158:80

Varnish Example

derived from the varnish director example from the varnish page: https://www.varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#directors

import directors;    # load the directors
{{range .c}}
backend {{.Node}}_{{.ID}} { 
    .host = "{{.IP}}";
    .port = "{{.Port}}";
}{{end}}

sub vcl_init {
    new bar = directors.round_robin();
{{range .c}}
    bar.add_backend({{.Node}}_{{.ID}});{{end}}
}

sub vcl_recv {
    # send all traffic to the bar director:
    set req.backend_hint = bar.backend();
}

Now, we run the following command:

consul-haproxy -addr=demo.consul.io -in in.conf -backend "c=consul@nyc1:80" -backend "c=consul@sfo1:80" -dry

The following should return:

backend 0_nyc1-server-1_consul {
    .host = "192.241.159.115";
    .port = "80";
}
backend 0_nyc1-server-3_consul {
    .host = "198.199.77.133";
    .port = "80";
}
backend 0_nyc1-server-2_consul {
    .host = "162.243.162.228";
    .port = "80";
}
backend 1_sfo1-server-3_consul {
    .host = "107.170.196.151";
    .port = "80";
}
backend 1_sfo1-server-2_consul {
    .host = "107.170.195.154";
    .port = "80";
}
backend 1_sfo1-server-1_consul {
    .host = "162.243.153.242";
    .port = "80";
}

sub vcl_init {
    new bar = directors.round_robin();

    bar.add_backend(0_nyc1-server-1_consul);
    bar.add_backend(0_nyc1-server-3_consul);
    bar.add_backend(0_nyc1-server-2_consul);
    bar.add_backend(1_sfo1-server-3_consul);
    bar.add_backend(1_sfo1-server-2_consul);
    bar.add_backend(1_sfo1-server-1_consul);
}

sub vcl_recv {
    # send all traffic to the bar director:
    set req.backend_hint = bar.backend();
}

More Repositories

1

terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
Go
42,550
star
2

vault

A tool for secrets management, encryption as a service, and privileged access management
Go
30,858
star
3

consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
Go
28,256
star
4

vagrant

Vagrant is a tool for building and distributing development environments.
Ruby
26,132
star
5

packer

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.
Go
15,086
star
6

nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
Go
14,809
star
7

terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
Go
9,761
star
8

raft

Golang implementation of the Raft consensus protocol
Go
7,383
star
9

serf

Service orchestration and management tool.
Go
5,692
star
10

hcl

HCL is the HashiCorp configuration language.
Go
5,236
star
11

go-plugin

Golang plugin system over RPC.
Go
4,874
star
12

terraform-cdk

Define infrastructure resources using programming constructs and provision them using HashiCorp Terraform
TypeScript
4,868
star
13

waypoint

A tool to build, deploy, and release any application on any platform.
Go
4,789
star
14

consul-template

Template rendering, notifier, and supervisor for @HashiCorp Consul and Vault data.
Go
4,750
star
15

terraform-provider-azurerm

Terraform provider for Azure Resource Manager
Go
4,528
star
16

otto

Development and deployment made easy.
HTML
4,282
star
17

golang-lru

Golang LRU cache
Go
4,221
star
18

boundary

Boundary enables identity-based access management for dynamic infrastructure.
Go
3,847
star
19

memberlist

Golang package for gossip based membership and failure detection
Go
3,303
star
20

go-memdb

Golang in-memory database built on immutable radix trees
Go
2,937
star
21

next-mdx-remote

Load mdx content from anywhere through getStaticProps in next.js
TypeScript
2,405
star
22

terraform-provider-google

Terraform Provider for Google Cloud Platform
Go
2,325
star
23

go-multierror

A Go (golang) package for representing a list of errors as a single error.
Go
2,029
star
24

yamux

Golang connection multiplexing library
Go
2,003
star
25

envconsul

Launch a subprocess with environment variables using data from @HashiCorp Consul and Vault.
Go
1,967
star
26

go-retryablehttp

Retryable HTTP client in Go
Go
1,702
star
27

terraform-provider-kubernetes

Terraform Kubernetes provider
Go
1,580
star
28

go-getter

Package for downloading things from a string URL using a variety of protocols.
Go
1,541
star
29

best-practices

HCL
1,490
star
30

go-version

A Go (golang) library for parsing and verifying versions and version constraints.
Go
1,459
star
31

go-metrics

A Golang library for exporting performance and runtime metrics to external metrics systems (i.e. statsite, statsd)
Go
1,453
star
32

setup-terraform

Sets up Terraform CLI in your GitHub Actions workflow.
JavaScript
1,352
star
33

terraform-guides

Example usage of HashiCorp Terraform
HCL
1,324
star
34

mdns

Simple mDNS client/server library in Golang
Go
1,020
star
35

terraform-provider-helm

Terraform Helm provider
Go
997
star
36

vault-guides

Example usage of HashiCorp Vault secrets management
Shell
990
star
37

go-immutable-radix

An immutable radix tree implementation in Golang
Go
926
star
38

vault-helm

Helm chart to install Vault and other associated components.
Shell
904
star
39

terraform-ls

Terraform Language Server
Go
896
star
40

vscode-terraform

HashiCorp Terraform VSCode extension
TypeScript
870
star
41

levant

An open source templating and deployment tool for HashiCorp Nomad jobs
Go
825
star
42

vault-k8s

First-class support for Vault and Kubernetes.
Go
697
star
43

terraform-exec

Terraform CLI commands via Go.
Go
675
star
44

consul-k8s

First-class support for Consul Service Mesh on Kubernetes
Go
665
star
45

terraform-aws-vault

A Terraform Module for how to run Vault on AWS using Terraform and Packer
HCL
659
star
46

terraform-github-actions

Terraform GitHub Actions
Shell
624
star
47

terraform-provider-vsphere

Terraform Provider for VMware vSphere
Go
617
star
48

raft-boltdb

Raft backend implementation using BoltDB
Go
585
star
49

nextjs-bundle-analysis

A github action that provides detailed bundle analysis on PRs for next.js apps
JavaScript
562
star
50

go-discover

Discover nodes in cloud environments
Go
562
star
51

consul-replicate

Consul cross-DC KV replication daemon.
Go
504
star
52

next-mdx-enhanced

A Next.js plugin that enables MDX pages, layouts, and front matter
JavaScript
500
star
53

docker-vault

Official Docker images for Vault
Shell
492
star
54

terraform-provider-kubernetes-alpha

A Terraform provider for Kubernetes that uses dynamic resource types and server-side apply. Supports all Kubernetes resources.
Go
490
star
55

vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
Go
466
star
56

terraform-k8s

Terraform Cloud Operator for Kubernetes
Go
454
star
57

puppet-bootstrap

A collection of single-file scripts to bootstrap your machines with Puppet.
Shell
444
star
58

cap

A collection of authentication Go packages related to OIDC, JWKs, Distributed Claims, LDAP
Go
436
star
59

terraform-provider-vault

Terraform Vault provider
Go
431
star
60

damon

A terminal UI (TUI) for HashiCorp Nomad
Go
427
star
61

nomad-autoscaler

Nomad Autoscaler brings autoscaling to your Nomad workloads.
Go
424
star
62

consul-helm

Helm chart to install Consul and other associated components.
Shell
422
star
63

terraform-provider-azuread

Terraform provider for Azure Active Directory
Go
419
star
64

vault-ssh-helper

Vault SSH Agent is used to enable one time keys and passwords
Go
404
star
65

terraform-provider-scaffolding

Quick start repository for creating a Terraform provider
Go
402
star
66

terraform-aws-consul

A Terraform Module for how to run Consul on AWS using Terraform and Packer
HCL
401
star
67

docker-consul

Official Docker images for Consul.
Dockerfile
399
star
68

nomad-pack

Go
393
star
69

hil

HIL is a small embedded language for string interpolations.
Go
392
star
70

vault-action

A GitHub Action that simplifies using HashiCorp Vaultโ„ข secrets as build variables.
JavaScript
391
star
71

learn-terraform-provision-eks-cluster

HCL
389
star
72

terraform-plugin-sdk

Terraform Plugin SDK enables building plugins (providers) to manage any service providers or custom in-house solutions
Go
383
star
73

terraform-config-inspect

A helper library for shallow inspection of Terraform configurations
Go
380
star
74

hcl2

Former temporary home for experimental new version of HCL
Go
375
star
75

errwrap

Errwrap is a Go (golang) library for wrapping and querying errors.
Go
373
star
76

go-cleanhttp

Go
366
star
77

design-system

Helios Design System
TypeScript
358
star
78

logutils

Utilities for slightly better logging in Go (Golang).
Go
356
star
79

vault-ruby

The official Ruby client for HashiCorp's Vault
Ruby
336
star
80

vault-rails

A Rails plugin for easily integrating Vault secrets
Ruby
334
star
81

next-remote-watch

Decorated local server for next.js that enables reloads from remote data changes
JavaScript
325
star
82

waypoint-examples

Example Apps that can be deployed with Waypoint
PHP
325
star
83

go-hclog

A common logging package for HashiCorp tools
Go
319
star
84

vault-csi-provider

HashiCorp Vault Provider for Secret Store CSI Driver
Go
308
star
85

nomad-guides

Example usage of HashiCorp Nomad
HCL
281
star
86

terraform-provider-google-beta

Terraform Provider for Google Cloud Platform (Beta)
Go
265
star
87

consul-esm

External service monitoring for Consul
Go
262
star
88

http-echo

A tiny go web server that echos what you start it with!
Makefile
257
star
89

terraform-provider-awscc

Terraform AWS Cloud Control provider
HCL
256
star
90

terraform-aws-nomad

A Terraform Module for how to run Nomad on AWS using Terraform and Packer
HCL
254
star
91

faas-nomad

OpenFaaS plugin for Nomad
Go
252
star
92

go-sockaddr

IP Address/UNIX Socket convenience functions for Go
Go
250
star
93

terraform-foundational-policies-library

Sentinel is a language and framework for policy built to be embedded in existing software to enable fine-grained, logic-based policy decisions. This repository contains a library of Sentinel policies, developed by HashiCorp, that can be consumed directly within the Terraform Cloud platform.
HCL
233
star
94

nomad-driver-podman

A nomad task driver plugin for sandboxing workloads in podman containers
Go
226
star
95

vagrant-vmware-desktop

Official provider for VMware desktop products: Fusion, Player, and Workstation.
Go
225
star
96

go-tfe

HCP Terraform/Enterprise API Client/SDK in Golang
Go
224
star
97

nomad-pack-community-registry

A repo for Packs written and maintained by Nomad community members
HCL
215
star
98

boundary-reference-architecture

Example reference architecture for a high availability Boundary deployment on AWS.
HCL
211
star
99

terraform-plugin-framework

A next-generation framework for building Terraform providers.
Go
204
star
100

vault-plugin-auth-kubernetes

Vault authentication plugin for Kubernetes Service Accounts
Go
192
star