tEKS
- Terraform/Terragrunt
- Contributing
- Requirements
- Quickstart
- Main purposes
- What you get
- Curated Features
- Bottlerocket support
- AWS Session Manager by default
- From and to Zero scaling with EKS Managed Node Groups
- Automatic dependencies upgrade
- Enforced security
- Out of the box logging
- Out of the box monitoring
- Long term storage with Thanos
- Support for ARM instances
- Helm v3 provider
- Other and not limited to
- Always up to date
- Requirements
- Examples
- Additional infrastructure blocks
- Branches
- License
tEKS is a set of Terraform / Terragrunt modules designed to get you everything you need to run a production EKS cluster on AWS. It ships with sensible defaults, and add a lot of common addons with their configurations that work out of the box.
This is our opinionated view of what a well structred infrastructure as code repository should look like.
Terraform/Terragrunt
- Terragrunt implementation is available in the
terragruntfolder.
Contributing
Contribution are welcome, as well as issues, we are usually quite reactive. If you need more support for your project, do not hesitate to reach us directly.
Requirements
Terragrunt
Quickstart
Quickstart guide is available here or on the official documentation website
Main purposes
The main goal of this project is to glue together commonly used tooling with Kubernetes/EKS and to get from an AWS Account to a production cluster with everything you need without any manual configuration.
What you get
A production cluster all defined in IaaC with Terraform/Terragrunt:
- AWS VPC if needed based on
terraform-aws-vpc - EKS cluster base on
terraform-aws-eks - Kubernetes addons based on
terraform-kubernetes-addons: provides various addons that are often used on Kubernetes and specifically on EKS. This module is currated by Particule and well maintained.
Everything is tied together with Terragrunt and allows you to deploy a multi cluster architecture in a matter of minutes.
Curated Features
The additional features are provided by tEKS here as well as our curated addons module which support a bunch of various configuration.
Bottlerocket support
Bottlerocket OS is available for node groups (see example here). Bottle rocket is a container centric OS with less attack surface and no default shell.
AWS Session Manager by default
All the instances (Bottlerocket or Amazon Linux) are registered with AWS Session Manager. No SSH keys or SSH access is open on instances. Shell access on every instance can be given with SSM for added security.
aws ssm start-session --target INSTANCE_ID
From and to Zero scaling with EKS Managed Node Groups
tEKS support scaling to and from 0, even with using well know Kubernetes labels, there are a number of ongoing issues for support of EKS Managed node groups with Cluster Autoscaler. Thanks to automatic ASG tagging, tEKS adds the necessary tags on autoscaling group to balance similar node groups and allow you to scale to and from 0 and even to use well know labels such as node.kubernetes.io/instance-type or topology.kubernetes.io/zone . The logic can be extended to support other well known labels.
Automatic dependencies upgrade
We are using renovate to automatically open PR with the latest dependencies update (Terraform modules upgrade) so you never miss an upgrade and are alwasy up to date with the latest features.
Enforced security
- Encryption by default for root volume on instances with Custom KMS Key
- AWS EBS CSI volumes encrypted by default with Custom KMS Key
- No IAM credentials on instances, everything is enforced with IRSA.
- Each addons is deployed in it's own namespace with sensible default network policies.
- Calico Tigera Operator for network policy.
- PSP are enabled but not enforced because of depreciation.
Out of the box logging
Three stacks are supported:
- AWS for Fluent Bit: Forward containers logs to Cloudwatch Logs
- Grafana Loki: Uses Promtail to forward logs to Loki. Grafana or a tEKS supported monitoring stack (see below) is necessary to display logs.
Out of the box monitoring
- Prometheus Operator with defaults dashboards
- Addons that support metrics are enable along with their
serviceMonitor - Custom grafana dashboard are available by default
Two stacks are supported:
- Victoria Metrics Stack: Victoria Metrics is a Prometheus alertnative, compatible with prometheus CRDs
- Kube Prometheus Stack: Classic Prometheus Monitoring
Long term storage with Thanos
With Prometheus, tEKS includes Thanos by default. Thanos uses S3 to store and query metrics, offering long term storage without the costs. For more information check out our article on the CNCF Blog
Support for ARM instances
With either Amazon Linux or BottleRocket, you can use a mix of ARM and AMD64 instances. Check out our example
Helm v3 provider
- All addons support Helm v3 configuration
- All charts are easily customizable
Other and not limited to
- priorityClasses for addons and critical addons
- lot of manual stuff have been automated under the hood
Always up to date
We always support the latest modules and features for our addons module.
Our cutting edges addons include (not limited to):
- AWS EBS CSI Drivers: Support for Volume encryption by default, snapshot, etc
- AWS EFS CSI Drivers: Use AWS NFS shares.
- Secret Store CSI Driver: load
secret from Secret Managers with
aws-secret-store-csi-driverdriver - Linkerd2 or Certificate Manager CSI for mTLS
Requirements
Terragrunt is not a hard requirement but all the modules are tested with Terragrunt.
Pre-commit
This repository use pre-commit hooks, please see this on how to setup tooling
ASDF
ASDF is a package manager which is great for managing cloud native tooling. More info here (eg. French).
Enabling plugins
for p in $(cut -d " " .tool-versions -f1); do asdf plugin add $p; done
Installing tools
asdf install
Examples
terragrunt/live folder provides an opinionated directory structure for a production environment.
Additional infrastructure blocks
If you wish to extend your infrastructure you can pick up additional modules on the particuleio github page. Some modules can also be found on the clusterfrak-dynamics github page.
Branches
main: Backward incompatible with v1.X but compatible with v2.X, releases bumped to v3.X because a lot has changed.release-1.X: Compatible with Terraform < 0.12 and Terragrunt < 0.19. Be sure to target the same modules version.release-2.X: Compatible with Terraform >= 0.12 and Terragrunt >= 0.19. Be sure to target the same modules version.
