• Stars
    star
    144
  • Rank 255,590 (Top 6 %)
  • Language CUE
  • License
    Apache License 2.0
  • Created about 2 years ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Flux local dev environment with Docker and Kubernetes KIND

flux-local-dev

test license release

Spin up a local dev environment for Flux with Docker and Kubernetes KIND in under five minutes.

Who is this for?

  • Flux users who want to test Flux configs locally, without having to push changes to a Git repository. Config changes are pushed to a local registry and synced on the cluster by Flux automatically.
  • Flux contributors who want to test their changes to Flux controllers locally, without having to push the container images to an external registry.
  • Flux maintainers who want to test Flux prereleases on various Kubernetes versions and configurations.

How does it work?

This project spins up a Docker Registry container named kind-registry and a Kubernetes Kind cluster named flux under the same Docker network. Then it installs Flux and configures it to upgrade itself from the latest OCI artifact published at ghcr.io/fluxcd/flux-manifests. Before an upgrade, Flux verifies that the OCI artifacts are signed by the Flux team with Cosign and GitHub OIDC.

Component Role Host
Kubernetes KIND Local cluster Binds to port 80 and 443
Docker Registry Local registry Binds to port 5050
Flux Cluster reconciler -
ingress-nginx Ingress for *.flux.local -
cert-manager Self-signed ingress certs -
metrics-server Container resource metrics -
kube-prometheus-stack Prometheus Operator and Grafana Binds to grafana.flux.local
weave-gitops Flux UI Binds to ui.flux.local
podinfo Demo app Binds to podinfo.flux.local

The Docker registry is exposed on the local machine on localhost:5050 and inside the cluster on kind-registry:5000. The registry servers two purposes:

  • hosts container images e.g. docker push localhost:5050/podinfo:test1
  • hosts OCI artifacts e.g. flux push artifact oci://localhost:5050/podinfo-manifests:test1

To facilitate ingress access to the Flux UI and any other application running inside the cluster, the Kubernetes Kind container binds to port 80 and 443 on localhost. Ingress is handled by Kubernetes ingress-nginx and self-signed TLS certs are provided by cert-manager.

To monitor how the deployed applications perform on the cluster, the kube-prometheus-stack and metrics-server Helm charts are installed at bootstrap along with the Flux Grafana dashboards.

To monitor and debug Flux using a Web UI, the Weave GitOps Helm chart is installed at bootstrap.

How to get started?

Prerequisites

Start by cloning the repository locally:

git clone https://github.com/stefanprodan/flux-local-dev.git
cd flux-local-dev

Install Kubernetes kind, kubectl, flux and other CLI tools with Homebrew:

make tools

The complete list of tools can be found in the Brewfile.

Note that the minimum required version of Flux is v2.0.0-rc.1.

Bootstrap

Start the dev environment with:

make up

The make up command performs the following steps:

  • creates the Docker registry container if it's not already running
  • creates the Kubernetes Kind cluster if it's not already running
  • pushes the Kubernetes manifests as OCI artifacts to the local registry
    • locahost:5050/flux-cluster-sync is generated from kubernetes/clusters/local
    • locahost:5050/flux-infra-sync is generated from kubernetes/infra
    • locahost:5050/flux-apps-sync is generated from kubernetes/apps
  • installs Flux on the clusters and configures it to self upgrade from oci://ghcr.io/fluxcd/flux-manifests
  • waits for Flux to reconcile the cluster addons from oci://kind-registry:5000/flux-infra-sync
  • waits for Flux to reconcile the demo apps from oci://kind-registry:5000/flux-apps-sync

Access Flux UI

flux-ui

Add the following domains to /etc/hosts:

127.0.0.1 podinfo.flux.local
127.0.0.1 grafana.flux.local
127.0.0.1 ui.flux.local

Verify that the NGINX ingress self-signed TLS works:

make check

Access the Flux UI and Grafana using the username admin and password flux:

Access the demo application on http://podinfo.flux.local.

Sync local changes

Add a label to the apps namespace definition:

yq eval '.metadata.labels.env="dev"' -i ./kubernetes/apps/namespace.yaml

Validate the Kubernetes manifests and Flux custom resources:

make validate

Push the changes to the local registry with:

make sync

Verify that Flux has reconciled the namespace:

kubectl get ns apps --show-labels

Teardown

Delete the registry and the Kubernetes cluster with:

make down

How to use CUE for manifests generation?

In the cue directory you can find an example of how to use cuelang to define and generate Kubernetes resources.

List the CUE generated resources with make cue-ls:

$ make cue-ls

RESOURCE                                   API VERSION
Namespace/cue-apps                         v1
ServiceAccount/cue-apps/flux-cue-apps      v1
Service/cue-apps/podinfo                   v1
ServiceAccount/cue-apps/podinfo            v1
Deployment/cue-apps/podinfo                apps/v1
HorizontalPodAutoscaler/cue-apps/podinfo   autoscaling/v2beta2
Ingress/cue-apps/podinfo                   networking.k8s.io/v1
ServiceMonitor/cue-apps/podinfo            monitoring.coreos.com/v1
RoleBinding/cue-apps/flux-cue-apps         rbac.authorization.k8s.io/v1

Push the generated resources to the local registry with make cue-push:

$ make cue-push 
β–Ί pushing artifact to localhost:5050/flux-cue-apps-sync:local
βœ” artifact successfully pushed to localhost:5050/flux-cue-apps-sync@sha256:59676338abbb245a80345713fea24c2686c8c38cbd235691dd0af0fdc00fe116

To reconcile the resources on the cluster, add a file called cue-apps.yaml to the kubernetes/cluster/local directory:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: cue-apps-source
  namespace: flux-system
spec:
  insecure: true
  interval: 1m
  provider: generic
  ref:
    tag: local
  url: oci://kind-registry:5000/flux-cue-apps-sync
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cue-apps-sync
  namespace: flux-system
spec:
  dependsOn:
    - name: infra-config
  interval: 5m
  retryInterval: 30s
  timeout: 5m
  path: ./
  prune: true
  sourceRef:
    kind: OCIRepository
    name: cue-apps-source

Sync the changes on the cluster with:

make sync

List the reconciled objects with:

$ flux tree ks cue-apps-sync 
Kustomization/flux-system/cue-apps-sync
β”œβ”€β”€ Namespace/cue-apps
β”œβ”€β”€ ServiceAccount/cue-apps/flux-cue-apps
β”œβ”€β”€ ServiceAccount/cue-apps/podinfo
β”œβ”€β”€ RoleBinding/cue-apps/flux-cue-apps
β”œβ”€β”€ Service/cue-apps/podinfo
β”œβ”€β”€ Deployment/cue-apps/podinfo
β”œβ”€β”€ HorizontalPodAutoscaler/cue-apps/podinfo
β”œβ”€β”€ ServiceMonitor/cue-apps/podinfo
└── Ingress/cue-apps/podinfo

If you make changes to the CUE definitions, run make cue-push and Flux will apply the changes on its own.

How to test Flux controllers?

Assuming you are contributing a change to kustomize-controller, and you want to run a series of end-to-end tests before opening a PR. Most importantly, you want to make sure your changes don't break Flux capability to upgrade itself.

Build and push the controller image

From within the kustomize-controller local clone, run make docker-build to build the controller image. If your local machine is an Apple M1, set the arch to linux/arm64 and run:

IMG=localhost:5050/kustomize-controller:latest make docker-build docker-push
BUILD_PLATFORMS=linux/arm64 make docker-build

Tag and push the image to your local registry:

docker tag fluxcd/kustomize-controller:latest localhost:5050/kustomize-controller:test1
docker push localhost:5050/kustomize-controller:test1

Deploy the controller

From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-sync.yaml file and add an image patch:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-sync
  namespace: flux-system
spec:
  images:
    - name: ghcr.io/fluxcd/kustomize-controller
      newName: localhost:5050/kustomize-controller
      newTag: test1

Sync the changes on the cluster with make sync and verify that the image is being rolled out:

make sync
kubectl -n flux-system get deploy/kustomize-controller --watch

Finally, verify that the upgrade was successful with:

$ flux check 

βœ” kustomize-controller: deployment ready
β–Ί localhost:5050/kustomize-controller:test1

If you don't want to bump the image tag on every build, you can bypass the local registry and import the image directly in the cluster cache:

export IMG=localhost:5050/kustomize-controller:test1
IMG=${IMG} make docker-build &&
kind import docker-image ${IMG} &&
kubectl delete pod -n flux-system -l app=kustomize-controller

How to test Flux prereleases?

Assuming you are maintainer, and you want to test the Flux controller suite before a release.

Build and push the manifests

From within the flux2 local clone, run make build-dev to build the Flux CLI binary that embeds the install manifests.

Extract the manifests to a directory with:

mkdir -p flux-vnext

./bin/flux install --components-extra=image-reflector-controller,image-automation-controller \
--export > ./flux-vnext/install.yaml

Push the manifests to your local registry:

./bin/flux push artifact oci://localhost:5050/flux:latest --path ./flux-vnext \
--source="$(git config --get remote.origin.url)" \
--revision="$(git rev-parse HEAD)"

Deploy the controllers

From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-source.yaml file, change the URL to point to your local registry and enable the insecure flag:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: flux-source
  namespace: flux-system
spec:
  url: oci://kind-registry:5000/flux
  insecure: true

Sync the changes on the cluster with make sync and wait for the new version to rollout:

make sync
flux reconcile ks flux-sync --with-source

Finally, verify that the upgrade was successful with:

flux check 

How to test Flux CRDs?

Assuming you are contributing an API change to source-controller, and you want to run a series of end-to-end tests before opening a PR.

Build and push the controller image

From within the source-controller local clone, run make docker-build to build the controller image. If your local machine is an Apple M1, set the arch to linux/arm64 and run:

IMG=localhost:5050/source-controller \
TAG=oci1 \
BUILD_PLATFORMS=linux/arm64 \
BUILD_ARGS=--load \
make docker-build docker-push

Build and push the manifests

From within the source-controller local clone, extract the Flux manifests to a directory:

mkdir -p flux-vnext

flux install --components-extra=image-reflector-controller,image-automation-controller \
--export > ./flux-vnext/install.yaml

Replace the CRD with the one from your branch:

export CRD_NAME="ocirepositories.source.toolkit.fluxcd.io"
yq e 'select(.metadata.name != env(CRD_NAME))' -i ./flux-vnext/install.yaml

CRD_BASE_PATH="./config/crd/bases"
CRD_FILE_NAME="source.toolkit.fluxcd.io_ocirepositories.yaml"
cp ${CRD_BASE_PATH}/${CRD_FILE_NAME} ./flux-vnext/

Push the manifests to your local registry:

flux push artifact oci://localhost:5050/flux:latest --path ./flux-vnext \
--source="$(git config --get remote.origin.url)" \
--revision="$(git rev-parse HEAD)"

Deploy the controller

From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-source.yaml file, change the URL to point to your local registry and enable the insecure flag:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: flux-source
  namespace: flux-system
spec:
  url: oci://kind-registry:5000/flux
  insecure: true

Open the kubernetes/clusters/local/flux-system/flux-sync.yaml file and patch the controller deployment with the local image:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flux-sync
  namespace: flux-system
spec:
  images:
    - name: ghcr.io/fluxcd/source-controller
      newName: localhost:5050/source-controller
      newTag: oci1

Sync the changes on the cluster with make sync and wait for the new version to rollout:

make sync
flux reconcile ks flux-sync --with-source

Manual testing

Test the new feature by adding a Flux resource to the kubernetes/apps/source-test.yaml:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo-keyless
  namespace: apps
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    semver: "*"
  verify:
    provider: cosign

Sync the changes on the cluster and see the reconciliation result:

make sync
flux get source oci podinfo-keyless -n apps

More Repositories

1

dockprom

Docker hosts and containers monitoring with Prometheus, Grafana, cAdvisor, NodeExporter and AlertManager
5,839
star
2

podinfo

Go microservice template for Kubernetes
Go
5,097
star
3

AspNetCoreRateLimit

ASP.NET Core rate limiting middleware
C#
3,043
star
4

swarmprom

Docker Swarm instrumentation with Prometheus, Grafana, cAdvisor, Node Exporter and Alert Manager
Shell
1,862
star
5

timoni

Timoni is a package manager for Kubernetes, powered by CUE and inspired by Helm.
Go
1,306
star
6

WebApiThrottle

ASP.NET Web API rate limiter for IIS and Owin hosting
C#
1,284
star
7

mgob

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.
Go
770
star
8

gitops-istio

A GitOps recipe for Progressive Delivery with Flux v2, Flagger and Istio
646
star
9

k8s-prom-hpa

Kubernetes Horizontal Pod Autoscaler with Prometheus custom metrics
Makefile
560
star
10

kustomizer

An experimental package manager for distributing Kubernetes configuration as OCI artifacts.
Go
279
star
11

MvcThrottle

ASP.NET MVC Throttling filter
C#
226
star
12

istio-gke

Istio service mesh walkthrough (GKE, CloudDNS, Flagger, OpenFaaS)
217
star
13

kube-tools

Kubernetes tools for GitHub Actions CI
Shell
190
star
14

k8s-scw-baremetal

Kubernetes installer for Scaleway bare-metal AMD64 and ARMv7
HCL
177
star
15

mongo-swarm

Bootstrapping MongoDB sharded clusters on Docker Swarm
Go
126
star
16

aspnetcore-dockerswarm

ASP.NET Core orchestration scenarios with Docker
C#
119
star
17

istio-hpa

Configure horizontal pod autoscaling with Istio metrics and Prometheus
Dockerfile
106
star
18

helm-gh-pages

A GitHub Action for publishing Helm charts to Github Pages
Shell
102
star
19

flux-aio

Flux All-In-One distribution made with Timoni
CUE
97
star
20

openfaas-flux

OpenFaaS Kubernetes cluster state management with FluxCD
HTML
79
star
21

dockerdash

Docker dashboard built with ASP.NET Core, Docker.DotNet, SignalR and Vuejs
JavaScript
69
star
22

gitops-linkerd

Progressive Delivery workshop with Linkerd, Flagger and Flux
Shell
64
star
23

gitops-helm-workshop

Progressive Delivery for Kubernetes with Flux, Helm, Linkerd and Flagger
Smarty
61
star
24

hrval-action

Flux Helm Release validation GitHub action
Shell
59
star
25

scaleway-swarm-terraform

Setup a Docker Swarm Cluster on Scaleway with Terraform
HCL
46
star
26

dockes

Elasticsearch cluster with Docker
Shell
45
star
27

flagger-appmesh-gateway

A Kubernetes API Gateway for AWS App Mesh powered by Envoy
Go
44
star
28

kjob

Kubernetes job runner
Go
42
star
29

gitops-progressive-delivery

Progressive delivery with Istio, Weave Flux and Flagger
41
star
30

gh-actions-demo

GitOps pipeline with GitHub actions and Weave Cloud
Go
38
star
31

eks-hpa-profile

An eksctl gitops profile for autoscaling with Prometheus metrics on Amazon EKS on AWS Fargate
35
star
32

faas-grafana

OpenFaaS Grafana
Shell
35
star
33

dockelk

ELK log transport and aggregation at scale
Shell
32
star
34

openfaas-gke

Running OpenFaaS on Google Kubernetes Engine
Shell
30
star
35

prometheus.aspnetcore

Prometheus instrumentation for .NET Core
C#
29
star
36

syros

DevOps tool for managing microservices
Go
28
star
37

gh-actions

GitHub actions for Kubernetes and Helm workflows
Dockerfile
27
star
38

gitops-app-distribution

GitOps workflow for managing app delivery on multiple clusters
Shell
22
star
39

gitops-kyverno

Kubernetes policy managed with Flux and Kyverno
Shell
21
star
40

gitops-appmesh

Progressive Delivery on EKS with AppMesh, Flagger and Flux v2
Shell
19
star
41

flux-workshop-2023

Flux Workshop 2023-08-10
18
star
42

dockerd-exporter

Prometheus Docker daemon metrics exporter
Dockerfile
17
star
43

caddy-builder

Build Caddy with plugins as an Ingress/Proxy for OpenFaaS
Go
16
star
44

jenkins

Continuous integration with disposable containers
Shell
16
star
45

swarm-gcp-faas

Setup OpenFaaS on Google Cloud with Terraform, Docker Swarm and Weave
HCL
16
star
46

nexus

A Sonatype Nexus Repository Manager Docker image based on Alpine with OpenJDK 8
16
star
47

podinfo-deploy

A GitOps workflow for multi-env deployments
14
star
48

es-curator-cron

Docker Alpine image with Elasticsearch Curator cron job
Shell
13
star
49

openfaas-certinfo

OpenFaaS function that returns SSL/TLS certificate information for a given URL
Go
12
star
50

caddy-dockerd

Caddy reverse proxy for Docker Remote API with IP filter
Shell
12
star
51

eks-contour-ingress

Securing EKS Ingress with Contour and Let's Encrypt the GitOps way
Shell
11
star
52

gomicro

golang microservice prototype
Go
10
star
53

ngrok

ngrok docker image
10
star
54

rancher-swarm-weave

Rancher + Docker Swarm + Weave Cloud Scope integration
HCL
9
star
55

kubernetes-cue-schema

CUE schema of the Kubernetes API
CUE
9
star
56

swarm-logspout-logstash

Logspout adapter to send Docker Swarm logs to Logstash
Go
9
star
57

openfaas-promq

OpenFaaS function that executes Prometheus queries
Go
8
star
58

appmesh-eks

AWS App Mesh installer for EKS
Smarty
6
star
59

fninfo

OpenFaaS Kubernetes info function
Go
5
star
60

alpine-base

Alpine Linux base image
Dockerfile
4
star
61

gloo-flagger-demo

GitOps Progressive Delivery demo with Gloo, Flagger and Flux
4
star
62

k8s-grafana

Kubernetes Grafana v5.0 dashboards
Smarty
4
star
63

stefanprodan

My open source portfolio and tech blog
4
star
64

appmesh-dev

Testing eks-appmesh-profile
Shell
3
star
65

AndroidDevLab

Android developer laboratory setup
Batchfile
2
star
66

my-k8s-fleet

2
star
67

RequireJsNet.Samples

RequireJS.NET samples
JavaScript
2
star
68

BFormsTemplates

BForms Visual Studio Project Template
JavaScript
2
star
69

EsnServiceBus

Service Bus and Service Registry implementation based on RabbitMQ
C#
2
star
70

klog

Go
2
star
71

homebrew-tap

Homebrew formulas
Ruby
1
star
72

evomon

Go
1
star
73

loadtest

Hey load test container
1
star
74

openfaas-colorisebot-gke-weave

OpenFaaS colorisebot on GKE and Weave Cloud
Shell
1
star
75

xmicro

microservice HA prototype
Go
1
star
76

goc-proxy

A dynamic reverse proxy backed by Consul
Go
1
star