flux-local-dev
Spin up a local dev environment for Flux with Docker and Kubernetes KIND in under five minutes.
Who is this for?
- Flux users who want to test Flux configs locally, without having to push changes to a Git repository. Config changes are pushed to a local registry and synced on the cluster by Flux automatically.
- Flux contributors who want to test their changes to Flux controllers locally, without having to push the container images to an external registry.
- Flux maintainers who want to test Flux prereleases on various Kubernetes versions and configurations.
How does it work?
This project spins up a Docker Registry container named kind-registry
and a Kubernetes Kind cluster
named flux
under the same Docker network. Then it installs Flux and configures it to upgrade itself
from the latest OCI artifact published at ghcr.io/fluxcd/flux-manifests
. Before an upgrade, Flux
verifies that the OCI artifacts are signed by the Flux team with Cosign and GitHub OIDC.
Component | Role | Host |
---|---|---|
Kubernetes KIND | Local cluster | Binds to port 80 and 443 |
Docker Registry | Local registry | Binds to port 5050 |
Flux | Cluster reconciler | - |
ingress-nginx | Ingress for *.flux.local |
- |
cert-manager | Self-signed ingress certs | - |
metrics-server | Container resource metrics | - |
kube-prometheus-stack | Prometheus Operator and Grafana | Binds to grafana.flux.local |
weave-gitops | Flux UI | Binds to ui.flux.local |
podinfo | Demo app | Binds to podinfo.flux.local |
The Docker registry is exposed on the local machine on localhost:5050
and inside the cluster
on kind-registry:5000
. The registry servers two purposes:
- hosts container images e.g.
docker push localhost:5050/podinfo:test1
- hosts OCI artifacts e.g.
flux push artifact oci://localhost:5050/podinfo-manifests:test1
To facilitate ingress access to the Flux UI and any other
application running inside the cluster, the Kubernetes Kind container
binds to port 80
and 443
on localhost.
Ingress is handled by Kubernetes ingress-nginx and self-signed TLS certs
are provided by cert-manager.
To monitor how the deployed applications perform on the cluster, the kube-prometheus-stack and metrics-server Helm charts are installed at bootstrap along with the Flux Grafana dashboards.
To monitor and debug Flux using a Web UI, the Weave GitOps Helm chart is installed at bootstrap.
How to get started?
Prerequisites
Start by cloning the repository locally:
git clone https://github.com/stefanprodan/flux-local-dev.git
cd flux-local-dev
Install Kubernetes kind, kubectl, flux and other CLI tools with Homebrew:
make tools
The complete list of tools can be found in the Brewfile
.
Note that the minimum required version of Flux is v2.0.0-rc.1
.
Bootstrap
Start the dev environment with:
make up
The make up
command performs the following steps:
- creates the Docker registry container if it's not already running
- creates the Kubernetes Kind cluster if it's not already running
- pushes the Kubernetes manifests as OCI artifacts to the local registry
locahost:5050/flux-cluster-sync
is generated fromkubernetes/clusters/local
locahost:5050/flux-infra-sync
is generated fromkubernetes/infra
locahost:5050/flux-apps-sync
is generated fromkubernetes/apps
- installs Flux on the clusters and configures it to self upgrade from
oci://ghcr.io/fluxcd/flux-manifests
- waits for Flux to reconcile the cluster addons from
oci://kind-registry:5000/flux-infra-sync
- waits for Flux to reconcile the demo apps from
oci://kind-registry:5000/flux-apps-sync
Access Flux UI
Add the following domains to /etc/hosts
:
127.0.0.1 podinfo.flux.local
127.0.0.1 grafana.flux.local
127.0.0.1 ui.flux.local
Verify that the NGINX ingress self-signed TLS works:
make check
Access the Flux UI and Grafana using the username admin
and password flux
:
- http://ui.flux.local/applications
- http://grafana.flux.local/d/flux-control-plane
- http://grafana.flux.local/d/flux-cluster
Access the demo application on http://podinfo.flux.local.
Sync local changes
Add a label to the apps
namespace definition:
yq eval '.metadata.labels.env="dev"' -i ./kubernetes/apps/namespace.yaml
Validate the Kubernetes manifests and Flux custom resources:
make validate
Push the changes to the local registry with:
make sync
Verify that Flux has reconciled the namespace:
kubectl get ns apps --show-labels
Teardown
Delete the registry and the Kubernetes cluster with:
make down
How to use CUE for manifests generation?
In the cue directory you can find an example of how to use cuelang to define and generate Kubernetes resources.
List the CUE generated resources with make cue-ls
:
$ make cue-ls
RESOURCE API VERSION
Namespace/cue-apps v1
ServiceAccount/cue-apps/flux-cue-apps v1
Service/cue-apps/podinfo v1
ServiceAccount/cue-apps/podinfo v1
Deployment/cue-apps/podinfo apps/v1
HorizontalPodAutoscaler/cue-apps/podinfo autoscaling/v2beta2
Ingress/cue-apps/podinfo networking.k8s.io/v1
ServiceMonitor/cue-apps/podinfo monitoring.coreos.com/v1
RoleBinding/cue-apps/flux-cue-apps rbac.authorization.k8s.io/v1
Push the generated resources to the local registry with make cue-push
:
$ make cue-push
βΊ pushing artifact to localhost:5050/flux-cue-apps-sync:local
β artifact successfully pushed to localhost:5050/flux-cue-apps-sync@sha256:59676338abbb245a80345713fea24c2686c8c38cbd235691dd0af0fdc00fe116
To reconcile the resources on the cluster, add a file called cue-apps.yaml
to the kubernetes/cluster/local
directory:
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: cue-apps-source
namespace: flux-system
spec:
insecure: true
interval: 1m
provider: generic
ref:
tag: local
url: oci://kind-registry:5000/flux-cue-apps-sync
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cue-apps-sync
namespace: flux-system
spec:
dependsOn:
- name: infra-config
interval: 5m
retryInterval: 30s
timeout: 5m
path: ./
prune: true
sourceRef:
kind: OCIRepository
name: cue-apps-source
Sync the changes on the cluster with:
make sync
List the reconciled objects with:
$ flux tree ks cue-apps-sync
Kustomization/flux-system/cue-apps-sync
βββ Namespace/cue-apps
βββ ServiceAccount/cue-apps/flux-cue-apps
βββ ServiceAccount/cue-apps/podinfo
βββ RoleBinding/cue-apps/flux-cue-apps
βββ Service/cue-apps/podinfo
βββ Deployment/cue-apps/podinfo
βββ HorizontalPodAutoscaler/cue-apps/podinfo
βββ ServiceMonitor/cue-apps/podinfo
βββ Ingress/cue-apps/podinfo
If you make changes to the CUE definitions, run make cue-push
and Flux will apply the changes on its own.
How to test Flux controllers?
Assuming you are contributing a change to kustomize-controller, and you want to run a series of end-to-end tests before opening a PR. Most importantly, you want to make sure your changes don't break Flux capability to upgrade itself.
Build and push the controller image
From within the kustomize-controller local clone, run make docker-build
to build the controller image.
If your local machine is an Apple M1, set the arch to linux/arm64
and run:
IMG=localhost:5050/kustomize-controller:latest make docker-build docker-push
BUILD_PLATFORMS=linux/arm64 make docker-build
Tag and push the image to your local registry:
docker tag fluxcd/kustomize-controller:latest localhost:5050/kustomize-controller:test1
docker push localhost:5050/kustomize-controller:test1
Deploy the controller
From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-sync.yaml
file
and add an image patch:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-sync
namespace: flux-system
spec:
images:
- name: ghcr.io/fluxcd/kustomize-controller
newName: localhost:5050/kustomize-controller
newTag: test1
Sync the changes on the cluster with make sync
and verify that the image is being rolled out:
make sync
kubectl -n flux-system get deploy/kustomize-controller --watch
Finally, verify that the upgrade was successful with:
$ flux check
β kustomize-controller: deployment ready
βΊ localhost:5050/kustomize-controller:test1
If you don't want to bump the image tag on every build, you can bypass the local registry and import the image directly in the cluster cache:
export IMG=localhost:5050/kustomize-controller:test1
IMG=${IMG} make docker-build &&
kind import docker-image ${IMG} &&
kubectl delete pod -n flux-system -l app=kustomize-controller
How to test Flux prereleases?
Assuming you are maintainer, and you want to test the Flux controller suite before a release.
Build and push the manifests
From within the flux2 local clone, run make build-dev
to build the Flux CLI
binary that embeds the install manifests.
Extract the manifests to a directory with:
mkdir -p flux-vnext
./bin/flux install --components-extra=image-reflector-controller,image-automation-controller \
--export > ./flux-vnext/install.yaml
Push the manifests to your local registry:
./bin/flux push artifact oci://localhost:5050/flux:latest --path ./flux-vnext \
--source="$(git config --get remote.origin.url)" \
--revision="$(git rev-parse HEAD)"
Deploy the controllers
From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-source.yaml
file,
change the URL to point to your local registry and enable the insecure flag:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: flux-source
namespace: flux-system
spec:
url: oci://kind-registry:5000/flux
insecure: true
Sync the changes on the cluster with make sync
and wait for the new version to rollout:
make sync
flux reconcile ks flux-sync --with-source
Finally, verify that the upgrade was successful with:
flux check
How to test Flux CRDs?
Assuming you are contributing an API change to source-controller, and you want to run a series of end-to-end tests before opening a PR.
Build and push the controller image
From within the source-controller local clone, run make docker-build
to build the controller image.
If your local machine is an Apple M1, set the arch to linux/arm64
and run:
IMG=localhost:5050/source-controller \
TAG=oci1 \
BUILD_PLATFORMS=linux/arm64 \
BUILD_ARGS=--load \
make docker-build docker-push
Build and push the manifests
From within the source-controller local clone, extract the Flux manifests to a directory:
mkdir -p flux-vnext
flux install --components-extra=image-reflector-controller,image-automation-controller \
--export > ./flux-vnext/install.yaml
Replace the CRD with the one from your branch:
export CRD_NAME="ocirepositories.source.toolkit.fluxcd.io"
yq e 'select(.metadata.name != env(CRD_NAME))' -i ./flux-vnext/install.yaml
CRD_BASE_PATH="./config/crd/bases"
CRD_FILE_NAME="source.toolkit.fluxcd.io_ocirepositories.yaml"
cp ${CRD_BASE_PATH}/${CRD_FILE_NAME} ./flux-vnext/
Push the manifests to your local registry:
flux push artifact oci://localhost:5050/flux:latest --path ./flux-vnext \
--source="$(git config --get remote.origin.url)" \
--revision="$(git rev-parse HEAD)"
Deploy the controller
From within the flux-local-dev clone, open the kubernetes/clusters/local/flux-system/flux-source.yaml
file,
change the URL to point to your local registry and enable the insecure flag:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: flux-source
namespace: flux-system
spec:
url: oci://kind-registry:5000/flux
insecure: true
Open the kubernetes/clusters/local/flux-system/flux-sync.yaml
file
and patch the controller deployment with the local image:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-sync
namespace: flux-system
spec:
images:
- name: ghcr.io/fluxcd/source-controller
newName: localhost:5050/source-controller
newTag: oci1
Sync the changes on the cluster with make sync
and wait for the new version to rollout:
make sync
flux reconcile ks flux-sync --with-source
Manual testing
Test the new feature by adding a Flux resource to the kubernetes/apps/source-test.yaml
:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
name: podinfo-keyless
namespace: apps
spec:
interval: 5m
url: oci://ghcr.io/stefanprodan/manifests/podinfo
ref:
semver: "*"
verify:
provider: cosign
Sync the changes on the cluster and see the reconciliation result:
make sync
flux get source oci podinfo-keyless -n apps