• Stars
    star
    134
  • Rank 270,967 (Top 6 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 5 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

chaos engineering via kubernetes operator

Litmus chaos-operator for injecting chaos experiments on Kubernetes

Slack Channel GitHub Workflow Docker Pulls GitHub issues Twitter Follow Codacy Badge Go Report Card CII Best Practices BCH compliance FOSSA Status codecov YouTube Channel

Litmus chaos operator is used by Kubernetes application developers and SREs to inject chaos into the applications and Kubernetes infrastructure in a managed fashion. Its objective is to make the process of validation and hardening of application workloads on Kubernetes easy by automating the execution of chaos experiments. A sample chaos injection workflow could be as simple as:

  • Install the Litmus infrastructure components (RBAC, CRDs), the Operator & Experiment custom resource bundles via the operator manifest
  • Annotate the application under test (AUT), enabling it for chaos
  • Create a ChaosEngine custom resource tied to the AUT, which describes the experiment to be executed

Benefits provided by the Chaos Operator include:

  • Standardised chaos experiment spec
  • Categorized chaos bundles for stateless/stateful/vendor-specific
  • Test-Run resiliency
  • Ability to chaos run as a background service based on annotations

What is a chaos operator and how is it built?

The Chaos Operator is a Kubernetes Operator, which are nothing but custom-controllers with direct access to Kubernetes API that can manage the lifecycle of certain resources or applications, while always trying to ensure the resource is in the "desired state". The logic that ensures this is commonly called "reconcile" function.

The Chaos Operator is built using the popular Operator-SDK framework, which provides bootstrap support for new operator projects, allowing teams to focus on business/operational logic.

The Litmus Chaos Operator helps reconcile the state of the ChaosEngine, a custom resource that holds the chaos intent specified by a developer/devops engineer against a particular stateless/stateful Kubernetes deployment. The operator performs specific actions upon CRUD of the ChaosEngine, its primary resource. The operator also defines a secondary resource (the engine runner pod), which is created & managed by it in order to implement the reconcile functions.

What is a chaos engine?

The ChaosEngine is the core schema that defines the chaos workflow for a given application. Currently, it defines the following:

  • Application info (namespace, labels, kind) of primary (AUT) and auxiliary (dependent) applications
  • ServiceAccount used for execution of the experiment
  • Flag to turn on/off chaos annotation checks on applications
  • Chaos Experiment to be executed on the application
  • Attributes of the experiments (overrides defaults specified in the experiment CRs)
  • Flag to retain/cleanup chaos resources after experiment execution

The ChaosEngine is the referenced as the owner of the secondary (reconcile) resource with Kubernetes deletePropagation ensuring these also are removed upon deletion of the ChaosEngine CR.

Here is a sample ChaosEngineSpec for reference: https://v1-docs.litmuschaos.io/docs/getstarted/#prepare-chaosengine

What is a litmus chaos chart and how can I use it?

Litmus Chaos Charts are used to install "Chaos Experiment Bundles" & are categorized based on the nature of the experiments (general Kubernetes chaos, vendor/provider specific chaos - such as, OpenEBS or application-specific chaos, say NuoDB). They consist of custom resources that hold low-level chaos(test) parameters which are queried by the operator in order to execute the experiments. The spec.definition.fields and their corresponding values are used to construct the eventual execution artifact that runs the chaos experiment (typically, the litmusbook, which is a K8s job resource). It also defines the permissions necessary to execute the experiment.

Here is a sample ChaosEngineSpec for reference:

apiVersion: litmuschaos.io/v1alpha1
description:
  message: |
    Deletes a pod belonging to a deployment/statefulset/daemonset
kind: ChaosExperiment
metadata:
  name: pod-delete
  labels:
    name: pod-delete
    app.kubernetes.io/part-of: litmus
    app.kubernetes.io/component: chaosexperiment
    app.kubernetes.io/version: latest
spec:
  definition:
    scope: Namespaced
    permissions:
      - apiGroups:
          - ""
          - "apps"
          - "batch"
          - "litmuschaos.io"
        resources:
          - "deployments"
          - "jobs"
          - "pods"
          - "configmaps"
          - "chaosengines"
          - "chaosexperiments"
          - "chaosresults"
        verbs:
          - "create"
          - "list"
          - "get"
          - "patch"
          - "update"
          - "delete"
    image: "litmuschaos/go-runner:latest"
    imagePullPolicy: Always
    args:
    - -c
    - ./experiments -name pod-delete
    command:
    - /bin/bash
    env:

    - name: TOTAL_CHAOS_DURATION
      value: '15'

    # Period to wait before/after injection of chaos in sec
    - name: RAMP_TIME
      value: ''

    - name: FORCE
      value: 'true'

    - name: CHAOS_INTERVAL
      value: '5'

    ## percentage of total pods to target
    - name: PODS_AFFECTED_PERC
      value: ''

    - name: LIB
      value: 'litmus'    

    - name: TARGET_PODS
      value: ''

    ## it defines the sequence of chaos execution for multiple target pods
    ## supported values: serial, parallel
    - name: SEQUENCE
      value: 'parallel'
    labels:
      name: pod-delete
      app.kubernetes.io/part-of: litmus
      app.kubernetes.io/component: experiment-job
      app.kubernetes.io/version: latest

How to get started?

Refer the LitmusChaos documentation litmus docs

How do I contribute?

You can contribute by raising issues, improving the documentation, contributing to the core framework and tooling, etc.

Head over to the Contribution guide

License

FOSSA Status

More Repositories

1

litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
Go
4,399
star
2

chaos-charts

Repository to hold chaos experiments resource YAML bundles
Go
78
star
3

litmus-go

Go
67
star
4

litmus-helm

Helm Charts for the Litmus Chaos Operator & CRDs
YAML
45
star
5

chaos-exporter

Prometheus Exporter for Litmus Chaos Metrics
Go
35
star
6

test-tools

A set of containerized applications, workload generators that will be consumed by openebs/e2e & openebs/litmus
C
29
star
7

github-chaos-actions

Github actions to trigger chaos on your review apps
Shell
29
star
8

chaos-runner

The bridge between chaos operator and chaos experiment! Lifecycle manager for chaos experiments
Go
24
star
9

litmusctl

Command-Line tool to manage Litmuschaos's agent plane
Go
24
star
10

chaos-workflows

Argo integration with LitmusChaos to create Chaos Workflows
19
star
11

v1-litmus-docs

Documentation for the Litmus Project
JavaScript
18
star
12

charthub.litmuschaos.io

front-end and back-end for litmus community charts
TypeScript
16
star
13

litmus-e2e

Test the Chaos!!! An e2e (end-to-end) validation pipeline for the LitmusChaos infrastructure
Go
15
star
14

litmus-docs

Documentation for the Litmus Project
JavaScript
15
star
15

litmus-python

Litmus Chaos Experiments written in Python !
Python
12
star
16

website-litmuschaos

Website source code for litmuschaos.io and staging-litmuschaos.io etc
TypeScript
9
star
17

litmus-ansible

Ansible-based chaos experiments
Python
9
star
18

litmus-cd-old

8
star
19

admission-controllers

Admission Controllers used in LitmusChaos
Go
7
star
20

chaos-scheduler

Schedule your Chaos!!
Go
7
star
21

chaos-ci-lib

Go
7
star
22

litmus-ui

Litmus UI - component library
TypeScript
6
star
23

chaos-observability

Observability for LitmusChaos Experiments
6
star
24

litmus-java-sdk

Java
6
star
25

tutorials

Consists of codelabs to perform standard user flows for LitmusChaos
HTML
5
star
26

gitlab-remote-templates

5
star
27

m-agent

LitmusChaos Machine Agent
Go
4
star
28

backstage-plugin

Backstage plugin where you can find all the information you need to run chaos engineering on Litmus in one place
TypeScript
3
star
29

litmus-portal

Contains the frontend and backend for Litmus portal
CSS
3
star
30

litmus-website-2

litmus-website-2
JavaScript
1
star