k8s-bench-suite
Bash scripts collection to benchmark kubernetes cluster performance
knb : Kubernetes Network Benchmark
knb is a bash script that will start a networking benchmark on a target Kubernetes cluster.
Here are some highlights:
- Plain bash script with very few dependencies
- Complete benchmark takes only 2 minutes
- Ability to select only a subset of benchmark tests to run
- Testing both TCP and UDP bandwidth
- Automatic detection of CNI MTU
- Includes host cpu and ram monitoring in benchmark report
- Ability to create static graph images based on the result data using plotly/orca (see examples below)
- No ssh access required, just an access to the target cluster through standard kubectl
- No need for high privileges, the script will just launch very lightweight pods on two nodes.
- Based on very lights containers images :
infrabuilder/bench-iperf3, is used to run benchmark tests
infrabuilder/bench-custom-monitor, is used to monitor nodes
- Ability to run the whole suite in a container olegeech/k8s-bench-suite:
- Image is based on the bitnami/kubectl
- Nodes for testing can be auto-preselected
Requirements
This script needs a valid kubectl
setup with an access to the target cluster.
Binaries dependencies for the host that will execute knb :
- awk
- grep
- tail
- date
- kubectl
- jq (for plotting)
Quickstart
Choose two nodes to act as server/client on your cluster (for example node1 and node2) . Then start the knb :
./knb --verbose --client-node node1 --server-node node2
If you omit the --verbose
flag, it will also complete, but you will have no output until the end of the benchmark.
Docker quickstart
Environment variables:
NODE_AUTOSELECT
Auto-selects a few nodes from cluster for running testsMASTER_ELIGIBLE
Master nodes can also be chosen
You need to mount a valid kubeconfig inside the container and provide all other required flags to knb:
docker run -e NODE_AUTOSELECT=1 -it --hostname knb --name knb --rm -v /home/user/my-graphs:/my-graphs -v /path/to/my/kubeconfig:/.kube/config olegeech/k8s-bench-suite --verbose --plot --plot-dir /my-graphs
Examples
-
Simple benchmark from "node1" to "node2" in verbose mode :
knb -v -cn node1 -sn node2
-
Benchmark from "nA" to "nB" and save data in file
mybench.knbdata
knb -cn nA -sn nB -o data -f mybench.knbdata
-
Generate report in json from previous benchmark data file
mybench.knbdata
knb -fd mybench.knbdata -o json
-
Plot graphs from previous benchmark data file
mybench.knbdata
knb -fd mybench.knbdata --plot --plot-args '--width 900 --height 600'
-
To run benchmark from node A to node B, showing only result in yaml format :
knb -cn A -sn B -o yaml
-
To run benchmark from node Asterix to node Obelix, with the most verbose output and a result as json in a
res.json
file :knb --debug -cn Asterix -sn Obelix -o json -f res.json
-
Running benchmark in namespace
myns
:knb -n myns -cn node1 -sn node2
-
Run only idle and tcp benchmark :
knb -cn clientnode -sn servernode -ot idle,tcp
Usage
To display usage, use the -h
flag :
aducastel@infrabuilder:~/k8s-bench-suite$ ./knb -h
knb is a network benchmark tool for Kubernetes CNI
There are two modes :
- benchmark mode : will actually run benchmark on a cluster
- from data mode : read data generated by previous benchmark with "-o data" flag
=====[ Benchmark mode ]====================================================
Mandatory flags :
-cn <nodename>
--client-node <nodename> : Define kubernetes node name that will host the client part
-sn <nodename>
--server-node <nodename> : Define kubernetes node name that will host the server part
Optionnal flags :
-d <time-in-scd>
--duration <time-in-scd> : Set the benchmark duration for each test in seconds (Default 10)
-k
--keep : Keep data directory instead of cleaning it (tmp dir that contains raw benchmark data)
-n <namespace>
--namespace <namespace> : Set the target kubernetes namespace
--name <name> : Set the name of this benchmark run
-ot <testlist>
--only-tests <testlist> : Only run a subset of benchmark tests, comma separated (Ex: -ot tcp,idle)
Possible values: all, tcp, udp, p2p, p2s , p2ptcp, p2pudp, p2stcp, p2sudp, idle
-sbs <size>
--socket-buffer-size <size> : Set the UDP socket buffer size with unit, or 'auto'. ex: '256K' (Default: auto)
-t <time-in-scd>
--timeout <time-in-scd> : Set the pod ready wait timeout in seconds (Default 30)
=====[ From Data mode ]====================================================
Mandatory flags :
-fd <path>
--from-data <path> : Define the path to the data to read from
Data file must be rendered with '--output data'
=====[ Common optionnal flags ]============================================
--debug : Set the debug level to "debug"
-dl <level>
--debug-level <level> : Set the debug level
Possible values: standard, warn, info, debug
-f <filepath>
--file <filepath> : Set the output file
-h
--help : Display this help message
-p
--plot : Plot data using plotly/orca
-pd
--plot-dir : Directory where to save graphs
Defaults to the current directory
-pa
--plot-args : Arguments to the plotly's 'orca graph' function
Defaults to '--width 900 --height 500'
-o <format>
--output <format> : Set the output format. Defaults to 'text'
Possible values: text, yaml, json, data
-v
--verbose : Activate the verbose mode by setting debug-level to 'info'
-V
--version : Show current script version
=====[ Examples ]==========================================================
Simple benchmark from "node1" to "node2" in verbose mode
-------------------------------------------------------------------------
| knb -v -cn node1 -sn node2 |
-------------------------------------------------------------------------
Benchmark from "nA" to "nB" with data saved in file "mybench.knbdata"
-------------------------------------------------------------------------
| knb -cn nA -sn nB -o data -f mybench.knbdata |
-------------------------------------------------------------------------
Generate report in json from previous benchmark file "mybench.knbdata"
-------------------------------------------------------------------------
| knb -fd mybench.knbdata -o json |
-------------------------------------------------------------------------
Create graph images from previous benchmark file "mybench.knbdata"
-------------------------------------------------------------------------
| knb -fd mybench.knbdata --plot --plot-dir ./my-graphs |
-------------------------------------------------------------------------
Run only idle and tcp benchmark :
-------------------------------------------------------------------------
| knb -cn clientnode -sn servernode -ot idle,tcp |
-------------------------------------------------------------------------