IMPORTANT NOTE

This repository is really out of date, I have moved onto Flux v2 and do not use Raspberry Pis anymore. This is now archived and I will leave it up for people to read.

k3s-gitops-arm

Build a Kubernetes (k3s) cluster with RPis and utilize GitOps for managing cluster state. I would like to give a shout-out to k8s-gitops, the big brother of this repo, created by @billimek.

This repo uses a lot of multi-arch images provided by raspbernetes/multi-arch-images.

Note: A lot of files in this project have @CHANGEME comments, these are things that are specific to my set up that you may need to change.

Prerequisites

Hardware

• 3x Raspberry Pi 4 (recommended 4GB RAM model)
• 3x SD cards (recommended 32GB)
• 3x USB 3.x flash drives (recommended for local storage)
• A NFS server for storing persistent data (recommended for shared storage)

Software

Note: I use the fish shell for a lot of my commands. Some will work in Bash but others will not, see here for more information.

• ansible
• kubectl
• hypriot/flash
• alexellis/k3sup

Directory topology

.
├── ./ansible        # Ansible playbook to run after the RPis have been flashed
├── ./deployments    # Flux will only scan and deploy from this directory
├── ./setup          # Setup of the cluster
├── ./secrets        # Scripts to generate secrets for Sealed Secrets
└── ./docs           # Documentation

Network topology

Let's get started

1. Flash SD Card with Ubuntu

See ubuntu.md

2. Provision RPis with Ansible

Ansible is a great automation tool and here I am using it to provision the RPis.

See ansible.md and review the files in the ansible folder.

3. Install k3s on your RPis using k3sup

k3sup is a neat tool provided by @alexellis that helps get your k3s cluster up and running quick.

For manual deployment see k3sup.md, and for an automated script see bootstrap-cluster.sh

4. Flux and Helm Operator

Helm is a package manager for Kubernetes.

Flux is the GitOps tool I've chosen to have this Git Repository manage my clusters state.

For manual deployment see helm-flux.md, and for an automated script see bootstrap-cluster.sh

Additional Components

Sealed Secrets

Sealed Secrets are a "one-way" encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, give to the NSA, etc. Once the Sealed Secret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.

See sealed-secrets.md and review the files in the secrets folder.

MetalLB

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.

Review the file metallb.yaml

Cert Manager

Cert-Manager will automatically provision and manage TLS certificates in Kubernetes. In this setup I am using Cloudflare as the DNS challenge.

NGINX Ingress /engine x/

NGINX Ingress Controller is an Ingress controller that uses ConfigMap to store the NGINX configuration.

Review the file nginx-ingress.yaml