Chaosblade-operator: A Chaos Engineering Tool for Cloud-native
中文版 README
Introduction
Chaosblade Operator is a chaos experiments injection tool for cloud-native on kubernetes platform. By defining Kubernetes CRD to manage chaos experiments, each experiment has a very clear execution status. The tool has the characteristics of simple deployment, convenient execution, standardized implementation, and rich experiments. The chaos experimental model in chaosblade is well integrated with Kubernetes, which can realize the reuse of experiments such as basic resources, application services, and containers on the Kubernetes platform, which facilitates the expansion of resource experiments under Kubernetes, and can be executed uniformly through chaosblade cli tool.
Supported experiments (continuously adding ...)
The current experimental scenarios involve resources including Node, Pod, and Container. The specific supported experimental scenarios are as follows:
- Node:
- CPU: specify CPU usage
- Network: specify network card, port, IP, etc. packet delay, packet loss, packet blocking, packet duplication, packet re-ordering, packet corruption, etc.
- Process: specify process Hang, kill process, etc.
- Disk: specify the directory disk occupation, disk IO read and write load, etc.
- Memory: specify memory usage
- Pod:
- Network: specify network card, port, IP, etc. packet delay, packet loss, packet blocking, packet duplication, packet re-ordering, packet corruption, etc.
- Disk: specify the directory disk occupation, disk IO read and write load, etc.
- Memory: specify memory usage
- Pod: kill pod
- IO: specify the file system io exception. Supports 31 file operations and 11 exception scenarios, such as "Too many open files", "Device or resource busy" and so on.
- Container:
- CPU: specify CPU usage
- Network: specify network card, port, IP, etc. packet delay, packet loss, packet blocking, packet duplication, packet re-ordering, packet corruption, etc.
- Process: specify process Hang, kill process, etc.
- Disk: specify the directory disk occupation, disk IO read and write load, etc.
- Memory: specify memory usage
- Container: remove container
Install and uninstall
The lowest version of kubernetes supported is 1.12. Chaosblade operator can be installed through kubectl or helm, the installation method is as follows:
Note: For the following VERSION, please use the latest version number instead
Helm v2
- Download the latest
chaosblade-operator-VERSION-v2.tgzpackage at Release - Install using
helm install --namespace chaosblade --name chaosblade-operator chaosblade-operator-VERSION-v2.tgz - Use
kubectl get pod -l part-of=chaosblade -n chaosbladeto check the installation status of the Pod. If both are running, the installation was successful - Use the following command to uninstall, pay attention to the execution order:
kubectl delete crd chaosblades.chaosblade.io
helm del --purge chaosblade-operatorHelm v3
- Download the latest
chaosblade-operator-VERSION-v3.tgzpackage at Release - Use
helm install chaosblade-operator chaosblade-operator-VERSION-v3.tgz --namespace chaosbladecommand to install - Use
kubectl get pod -l part-of=chaosblade -n chaosbladeto check the installation status of the Pod. If both are running, the installation was successful - Use the following command to uninstall, pay attention to the execution order:
kubectl delete crd chaosblades.chaosblade.io
helm uninstall chaosblade-operator -n chaosbladeKubectl
- Download the latest
chaosblade-operator-yaml-VERSION.tar.gzpackage at Release - After decompression, execute
kubectl apply -f chaosblade-operator-yaml-VERSION/installation - Use
kubectl get pod -l part-of=chaosblade -n chaosbladeto check the installation status of the Pod. If both are running, the installation was successful - Use the following command to uninstall, pay attention to the execution order:
kubectl delete crd chaosblades.chaosblade.io
kubectl delete -f chaosblade-operator-yaml-VERSION/How to use
You can run chaos experiments after installing the chaosblade operator. There are three ways to execute chaos experiments:
- By configuring yaml file, use kubectl to execute
- Executed using chaosblade cli tool
- Use Kubernetes API to execute by writing code
The following uses a specific case to illustrate the use of chaosblade-operator: simulate cn-hangzhou.192.168.0.205 node local port 40690 60% network packet loss.
By configuring the yaml file, use kubectl to execute
apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata:
name: loss-node-network-by-names
spec:
experiments:
- scope: node
target: network
action: loss
desc: "node network loss"
matchers:
- name: names
value: ["cn-hangzhou.192.168.0.205"]
- name: percent
value: ["60"]
- name: interface
value: ["eth0"]
- name: local-port
value: ["40690"]
Execute experiment：
kubectl apply -f loss-node-network-by-names.yaml
Query the experimental status, the returned information is as follows (spec and other contents are omitted):
~ » kubectl get blade loss-node-network-by-names -o json
{
"apiVersion": "chaosblade.io/v1alpha1",
"kind": "ChaosBlade",
"metadata": {
"creationTimestamp": "2019-11-04T09:56:36Z",
"finalizers": [
"finalizer.chaosblade.io"
],
"generation": 1,
"name": "loss-node-network-by-names",
"resourceVersion": "9262302",
"selfLink": "/apis/chaosblade.io/v1alpha1/chaosblades/loss-node-network-by-names",
"uid": "63a926dd-fee9-11e9-b3be-00163e136d88"
},
"status": {
"expStatuses": [
{
"action": "loss",
"resStatuses": [
{
"id": "057acaa47ae69363",
"kind": "node",
"name": "cn-hangzhou.192.168.0.205",
"nodeName": "cn-hangzhou.192.168.0.205",
"state": "Success",
"success": true,
"uid": "e179b30d-df77-11e9-b3be-00163e136d88"
}
],
"scope": "node",
"state": "Success",
"success": true,
"target": "network"
}
],
"phase": "Running"
}
}
From the above, you can clearly see the running status of the chaos experiment. Run the following command to stop the experiment:
kubectl delete -f loss-node-network-by-names.yaml
Or delete this blade resource directly:
kubectl delete blade loss-node-network-by-names
You can also edit the yaml file to update the content of the experiment and the chaosblade operator will complete the update of the experiment. See more examples: Examples
Execute with chaosblade cli tool
blade create k8s node-network loss --percent 60 --interface eth0 --local-port 40690 --names cn-hangzhou.192.168.0.205 --kubeconfig config
If the execution fails, a detailed error message is returned; if the execution is successful, the experiment UID is returned:
{"code":200,"success":true,"result":"e647064f5f20953c"}
You can query the status of the experiment with the following command:
blade query k8s create e647064f5f20953c --kubeconfig config
{
"code": 200,
"success": true,
"result": {
"uid": "e647064f5f20953c",
"success": true,
"error": "",
"statuses": [
{
"id": "fa471a6285ec45f5",
"uid": "e179b30d-df77-11e9-b3be-00163e136d88",
"name": "cn-hangzhou.192.168.0.205",
"state": "Success",
"kind": "node",
"success": true,
"nodeName": "cn-hangzhou.192.168.0.205"
}
]
}
}
Destroy experiment:
blade destroy e647064f5f20953c
In addition to the above two methods, you can also use the kubernetes client-go api for execution. For details, please refer to: executor.go code implementation.
Questions & Suggestions
If you encounter problems during installation and use, or suggestions and new features, all projects (including other projects) can be submitted to Github Issues
You can also contact us via:
- Dingding group: 23177705
- Gitter room: chaosblade community
- Email: [email protected]
- Twitter: chaosblade.io
Contributions
We welcome every issue and PR. Even a punctuation mark, how to participate in the contribution please read the CONTRIBUTING document, or contact us through the above method.
Open source license
Chaosblade-operator is licensed under the Apache 2.0 license. For details, please read LICENSE