TKS
Warning: You must clone this repositiory with recursion in order to pull the submodules.
Summary
As infrastructure becomes more and more abstracted from the user, it becomes both easier to work with and harder to manage. Hypervisors, containers, orchestration platforms, etc. Cloud providers today manage to automate this complexity at scale for millions of enterprise customers. TKS is a collection of projects aiming to provide a similar experience with bare metal.
Each component of TKS is broken out into a dedicated subproject. The is that each component should be able to be used interchangeably with other platforms. For example, TKS-Deploy_Kubernetes_Apps is collection of Kubernetes manifests, Kustomizations, and Ansible projects that should allow you to deploy applications to any Kubernetes cluster.
Combining all of the components together will produce a platform that leverages:
| Technology | Description |
|---|---|
| Proxmox VE | Type 1 Hypervisor |
| ZFS | Block storage for VMs and file storage for containers |
| Kubernetes | Container Orchestration Platform |
| HAProxy | Virtual Load Balancer for Kubernetes Control Plane nodes |
| Grafana Stack | Federated monitoring platform |
| Vault | Encrypted & decentralized secrets management |
| Gitlab | Source Control & Continuous Integration |
| ArgoCD | Continuous Delivery |
| Harbor | Container Image Registry |
When possible, automation is leveraged using common tooling like Terraform, Ansible, Cloud Init, and Kustomize. When configuration is necessary, options are exposed through environment variables and defaults are configured as appropriate.
Requirements
Hardware
TKS requires a server, some storage, and an understanding of how to network everything together. You don't need much compute, I ran all of this on a 2008 Mac Pro for years. You could even re-purpose this to run on a cloud platform like AWS. TKS-Bootstrap_Proxmox provides instructions for getting started with a Bootable USB Flash Drive or Dell iDRAC.
Today I develop this on a Dell PowerEdge R730xd with 384GB of DDR4 ECC memory and 32 Xeon E5-2640 v3 cores. My storage is broken into four pieces:
- 512GB RAID 1 ZFS Pool consisting of two mirrored Samsung 970 Pro M.2 SSDs used for compute storage.
- 18TB RAID 10 ZFS Pool consisting of 6 6TB HGST Ultrastar He6 7200RPM hard drives used for file storage.
- 6TB RAID 10 Hardware Raid consisting of 6 2TB HGST Ultrastar He6 7200RPM hard drives exposed via LVM in Proxmox and used for additional file storage as well as backups, templates, etc in Proxmox.
- 16GB Dell K9R5M SATADOM for Proxmox
Software
I'm able to use all of the tooling here from both MacOS and Arch Linux. It will probably work on Windows too. An understanding of how to use the following tools will be helpful, but hopefully not necessary with the documentation.
Getting Started
Clone this repository to retrieve the submodules below. This repository is treated like a Release and each Submodule should reflect the most current stable commit from each project. You can review the master branch for each project for additional unstable updates if desired. Detailed instructions for how to use each project is located in the respective README.
The ./inventory.yml file at the root of this repository is used for Ansible in each of the submodules. Be sure to modify it as per your environment before starting.
| Name | Description |
|---|---|
| TKS-Bootstrap_Proxmox | * Prepare iDRAC or a Bootable USB Device * Provision and Configure Proxmox VE * Initialize Storage & Clustering |
| TKS-Build_Template | * Build a VM template with Ansible |
| TKS-Bootstrap_Kubernetes | * Deploy HAProxy with Terraform to load balance K8s * Deploy Kubernetes Cluster with Terraform * Deploy Calico CNI Plugin |
| TKS-Deploy_Kubernetes_Apps | * Deploy Kubernetes apps like MetalLB, Istio, etc. * Deploy enterprise apps like Jira, OpenVPN, etc. * Deploy homelab apps like Plex, ruTorrent, Sonarr, etc. * Leverage Kustomize when possible, Ansible when not * Support for Istio, External Secrets, resource management, etc. * Lean on NFS & ZFS for Persistent Volumes |
| TKS-Deploy_Harbor | * Deploy Harbor with Terraform * Leverage LetsEncrypt to receive a valid SSL Certificate * Integrate with Kubernetes to self host container images |
| TKS-Deploy_Grafana | * Deploy Grafana with Terraform * Configure Kubernetes to ship logs * Configure other apps to ship logs |
| TKS-Deploy_Vault | * Deploy Vault with Terraform * Configure Vault to act as a secret store for Kubernetes |
| TKS-Deploy_Argo | * Deploy Argo with Terraform * Configure Argo to perform continuous delivery for Kubernetes |
| TKS-Deploy_Gitlab | * Deploy Gitlab with Terraform * Configure Gitlab to manage continuous integration for Kubernetes |
Frequently Asked Questions
Q: Where did the older Ansible/QEMU based project go?
I retired that project in favor of TKS. You can find the code here, however.
Q: Why did you choose Debian instead of X?
- Debian is stable and easy to configure
- I tried to use Flatcar first unsuccessfully
- CentOS is enterprisey
- If you're still opposed, consider adding a PR with support for another OS? :)
Q: Why did you expose configuration through environment variables?
IaC and CaC tooling usually expose configuration through variables files, so I understand why you might ask that. My goal in exposing configuration through environment variables was to better support CI/CD with this tooling.
Q: Why didn't you use X? Why aren't you using Y?
Consider opening an issue informing me why you think that.
Q: Why do you make things so complicated?
It's fun. TKS is developed as a hobby.
Q: I found an issue! How should I notify you?
Please file a GitHub issue under the respective subproject. Please do not email me for support until you have initiated the issue process on GitHub. Pull requests are also welcome and encouraged. :)
Q: What are some ways that I can contribute?
- Add support for additional types of storage
- Add support for multiple Proxmox nodes
- Add support for alternative network configurations
- Improve Terraform & Ansible code quality
- Submit your Kustomize overlays as an example for others