Kubeinvaders

A.K.A. k-inv

We have embedded a Kubeinvaders demo directly on our website to le try out the tool.

• Launch the demo at this link https://kubeinvaders.devopstribe.it/
• Watch Grafana dashboard here https://grafana.devopstribe.it
• Monitor the pod status here https://kubeopsview.devopstribe.it

These are the slides from the Chaos Engineering speech I prepared for Fosdem 2023. Unfortunately I could not be present at my talk :D but I would still like to share them with the community

Table of Contents

1. Description
2. Installation
3. Usage
4. Architecture
5. Persistence
6. Generic Troubleshooting & Known Problems
7. Troubleshooting Unknown Namespace
8. Metrics
9. Security
10. Community
11. Community blogs and videos
12. License

Description

Through k-inv a.k.a. KubeInvaders you can stress a Kubernetes cluster in a fun way and check how resilient it is.

Installation

Try with Docker (only for development purpose)

docker run -p 8080:8080 \
--env K8S_TOKEN=<k8s_service_account_token>  \
--env ENDPOINT=localhost:8080 \
--env INSECURE_ENDPOINT=true \
--env KUBERNETES_SERVICE_HOST=<k8s_controlplane_host> \
--env KUBERNETES_SERVICE_PORT_HTTPS=<k8s_controlplane_port> \
--env NAMESPACE=<comma_separated_namespaces_to_stress> \
luckysideburn/kubeinvaders:develop

Install to Kubernetes with Helm (v3+)

helm repo add kubeinvaders https://lucky-sideburn.github.io/helm-charts/
helm repo update

kubectl create namespace kubeinvaders

helm install kubeinvaders --set-string config.target_namespace="namespace1\,namespace2" \
-n kubeinvaders kubeinvaders/kubeinvaders --set ingress.enabled=true --set ingress.hostName=kubeinvaders.io --set deployment.image.tag=v1.9.6

Install to Kubernetes with Helm (v3+) - LoadBalancer / HTTP (tested with GKE)

helm install kubeinvaders --set-string config.target_namespace="namespace1\,namespace2" -n kubeinvaders kubeinvaders/kubeinvaders --set ingress.enabled=true --set ingress.hostName=kubeinvaders.local --set deployment.image.tag=v1.9.6 --set service.type=LoadBalancer --set service.port=80

kubectl set env deployment/kubeinvaders INSECURE_ENDPOINT=true -n kubeinvaders

SCC for Openshift

oc adm policy add-scc-to-user anyuid -z kubeinvaders

Route for Openshift

I should add this to the helm chart...

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: kubeinvaders
  namespace: "kubeinvaders"
spec:
  host: "kubeinvaders.io"
  to:
    name: kubeinvaders
  tls:
    termination: Edge

Usage

At the top you will find some metrics as described below:

Current Replicas State Delay is a metric that show how much time the cluster takes to come back at the desired state of pods replicas.

This is a control-plane you can use to switch off & on various features.

YouTube HowTo

Video How To of version v1.9

Start The Chaos Experiment

Press the button "Start" to start automatic pilot (button changes to "Stop" to disable this feature).

Enable Shuffle

Press the button "Enable Shuffle" to randomly switch the positions of pods or k8s nodes (button changes to "Disable Shuffle" to disable this feature).

Enable Auto Jump Between Namespace

Press the button "Auto NS Switch" to randomly switch between namespaces (button changes to "Disable Auto NS Switch" to disable this feature).

Show / Hide pods name

Press the button "Hide Pods Name" to hide the name of the pods under the aliens (button changes to "Show Pods Name" to disable this feature).

Information about current status and events

As described below, on the game screen, near the spaceship, there are details about current cluster, namespace and some configurations.

Under + and - buttons appears a bar with the latest occurred game events.

Do Kube-linter Lint

It is possibile using kube-linter through KubeInvaders in order to scan resources looking for best-practices or improvements to apply.

Example from YouTube

Show Special Keys

Press 'h' or select 'Show Special Keys' from the menu.

Zoom In / Out

Press + or - buttons to increase or decrease the game screen.

Chaos Containers for masters and workers nodes

• Select from the menu "Show Current Chaos Container for nodes" to watch which container start when you fire against a worker node (not an alien, they are pods).

• Select from the menu "Set Custom Chaos Container for nodes" to use your preferred image or configuration against nodes.

Architecture

Persistence

"Kinv" uses Redis for save and manage data. Redis is configured with "appendonly".

At moment the helm chart does not support PersistentVolumes but this task is in the to do list...

Generic Troubleshooting And Known Problems

• It seems that KubeInvaders does not work with EKS because of problems with ServiceAccount.
• At moment the installation of KubeInvaders into a namespace that is not named "kubeinvaders" is not supported
• I have only tested KubeInvaders with a Kubernetes cluster installed through KubeSpray
• If you don't see aliens please do the following steps:

1. Open a terminal and do "kubectl logs <pod_of_kubeinvader> -n kubeinvaders -f"
2. Execute from another terminal curl "https://<your_kubeinvaders_url>/kube/pods?action=list&namespace=namespace1" -k
3. Open an issue with attached logs

Troubleshooting Unknown Namespace

• Check if namespaces declared with helm config.target_namespace (es: config.target_namespace="namespace1,namespace2") exist and contain some pods
• Check from your browser developer console if there some HTTP requests that fails (send them to luckysideburn[at]gmail[dot]com or open an issue on this repo
• Try v1.9.6_debug and send logs to luckysideburn[at]gmail[dot]com or open an issue on this repo

Prometheus Metrics

KubeInvaders exposes metrics for Prometheus through the standard endpoint /metrics

This is an example of Prometheus configuration:

scrape_configs:
- job_name: kubeinvaders
  static_configs:
  - targets:
    - kubeinvaders.kubeinvaders.svc.cluster.local:8080

Example of metrics:

Security

In order to restrict the access to the Kubeinvaders endpoint add this annotation into the ingress.

nginx.ingress.kubernetes.io/whitelist-source-range: <your_ip>/32

Community

Please reach out for news, bugs, feature requests, and other issues via:

• On Twitter: @kubeinvaders & @luckysideburn
• New features are published on YouTube too in this channel

Community blogs and videos

• Kubernetes.io blog: KubeInvaders - Gamified Chaos Engineering Tool for Kubernetes
• acloudguru: cncf-state-of-the-union
• DevNation RedHat Developer: Twitter
• Flant: Open Source solutions for chaos engineering in Kubernetes
• Reeinvent: KubeInvaders - gamified chaos engineering
• Adrian Goins: K8s Chaos Engineering with KubeInvaders
• dbafromthecold: Chaos engineering for SQL Server running on AKS using KubeInvaders
• Pklinker: Gamification of Kubernetes Chaos Testing
• Openshift Commons Briefings: OpenShift Commons Briefing KubeInvaders: Chaos Engineering Tool for Kubernetes
• GitHub: awesome-kubernetes repo
• William Lam: Interesting Kubernetes application demos
• The Chief I/O: 5 Fun Ways to Use Kubernetes
• LuCkySideburn: Talk @ Codemotion
• Chaos Carnival: Chaos Engineering is fun!
• Kubeinvaders (old version) + OpenShift 4 Demo: YouTube_Video
• KubeInvaders (old version) Vs Openshift 4.1: YouTube_Video
• Chaos Engineering for SQL Server | Andrew Pruski | Conf42: Chaos Engineering: YouTube_Video

License

KubeInvaders is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.