Kube-Prometheus-Top [ kptop ]
A Python tool that provides Monitoring for Kubernetes Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metircs
kptop2.mp4
📌 Try out the new beta features
Motivation
The resources metrics provided by the K8s APIs are very limited compared what what's scraped by Prometheus.
This tool is using Prometheus as a data source for metrics to display all the needed informations right on the terminal.
Project Status
Additional
- Check the connection with Prometheus Server
- Check the existence of the needed Prometheus metrics [ good for similar use cases ]
Installation
Compatible with Python 3.6+
pip3 install kptop --upgradeEnvironment Variables
| ENV | Description | Default | Required |
|---|---|---|---|
KPTOP_PROMETHEUS_SERVER |
Prometheus server URL | Yes | |
KPTOP_BASIC_AUTH_ENABLED |
Whether basic authentication is needed to connect to Prometheus | False | No |
KPTOP_PROMETHEUS_USERNAME |
Prometheus username | No | |
KPTOP_PROMETHEUS_PASSWORD |
Prometheus password | No | |
KPTOP_INSECURE |
Verify SSL certificate | False | No |
KPTOP_NODE_EXPORTER_NODE_LABEL |
node exporter "node label" | "node" | NO |
KPTOP_START_GRAPHS_WITH_ZERO |
By default graphs begin with '0' to let the graph take its full hight | True | NO |
KPTOP_LOGGING_DIR |
Choose a different logging directory | /tmp/ | NO |
KPTOP_GRAPH_WIDTH |
Choose a custom graphs width | 45 | NO |
CLI Arguments
| ENV | Description | Default |
|---|---|---|
--namespace, -n |
Specify a Kubernetes Namespace | default |
--all-namespaces, -A |
||
--container, -c |
Specify a container | |
--interval, -i |
Live monitoring update interval | 8 [NOTE: the actuall update depends on the Prometheus scaping interval (15s by default)] |
--debug, -d |
Enable debugging logging mode | False |
--verify-prometheus, -V |
Verify connectivity to Prometheus server & check the existence of the needed exporters | |
--sort-by-mem-usage, -s |
Sort top result by memory usage | False |
--check-metrics , -C |
Check the existence of the needed metrics, needs to be done with -V |
False |
Usage
Different ways to connect to Prometheus server:
- You have direct access to it (Like in dev environments)
- Prometheus is exposed publically/over-vpn (mostly with an Ingress)
- You can use kubectl port-forward command
- You also can run
kptopas a Kubernetes pod (to be built))
export KPTOP_PROMETHEUS_SERVER="http://prometheus.home-lab.com"Top nodes
kptop nodesNODE MEM TOTAL MEM USAGE MEM FREE CPU CORES CPU USAGE% RUNNING PODS
worker-1 19.6 gb 16.92 gb 2.69 gb 6 9% 14
worker-2 19.6 gb 9.52 gb 10.08 gb 6 9% 27kptop nodes -o json --colorize-jsonUnder testing (currently tested with EKS)
kptop nodes -o cloudLive monitoring for Nodes
kptop node <NODE>
Top pods
kptop pods -n <NAMESPACE>kptop pods -n elk-stack
NAMESPACE POD MEM LIMIT MEM USAGE MEM USAGE % MEM USAGE MAX MEM FREE CPU LIMIT CPU USAGE
elk-stack elasticsearch-master-0 2.0 gb 1.38 gb 68% 2.0 gb 635.25 mb 1000m 0.04m
elk-stack elasticsearch-master-1 2.0 gb 1.49 gb 74% 2.0 gb 522.05 mb 1000m 0.03m
elk-stack strimzi-filebeat-filebeat-f8ms7 200.0 mb 85.73 mb 42% 174.16 mb 114.27 mb 1000m 0.02m
elk-stack haproxy-ingress-filebeat-filebeat-pq2wf 200.0 mb 87.73 mb 43% 171.31 mb 112.27 mb 1000m 0.03m
elk-stack strimzi-filebeat-filebeat-r7dht 200.0 mb 119.12 mb 59% 199.52 mb 80.88 mb 1000m 0.02m
elk-stack haproxy-ingress-filebeat-filebeat-lzqdt 200.0 mb 98.66 mb 49% 199.57 mb 101.34 mb 1000m 0.02m
elk-stack my-kibana-kibana-79448f7fb7-wf4t6 2.0 gb 342.87 mb 16% 618.07 mb 1.67 gb 1000m 0.02m
elk-stack my-logstash-logstash-0 1.5 gb 1008.22 mb 65% 1.21 gb 527.78 mb 1000m 0.02m
kptop pod -n kube-system
NAMESPACE POD MEM LIMIT MEM USAGE MEM USAGE% MEM USAGE MAX MEM FREE CPU LIMIT CPU USAGE
kube-system coredns-558bd4d5db-nfcjq 170.0 mb 26.0 mb 15% 42.77 mb 144.0 mb --- 0.0m
kube-system coredns-558bd4d5db-vcstr 170.0 mb 17.45 mb 10% 24.04 mb 152.55 mb --- 0.0m
kube-system etcd-master --- 85.02 mb --- 391.7 mb --- --- 0.02m
kube-system kube-apiserver-master --- 635.97 mb --- 731.13 mb --- --- 0.09m
kube-system kube-controller-manager-master --- 95.55 mb --- 145.41 mb --- --- 0.03m
kube-system kube-proxy-q6nr7 --- 27.46 mb --- 58.91 mb --- --- 0.0m
kube-system kube-proxy-q489q --- 21.98 mb --- 63.0 mb --- --- 0.0m
kube-system kube-proxy-bghp6 --- 22.35 mb --- 64.1 mb --- --- 0.0m
kube-system kube-scheduler-master --- 37.04 mb --- 61.68 mb --- --- 0.0m
kube-system nfs-subdir-external-provisioner-b97f4d9f5-bjp2h --- 9.43 mb --- 37.48 mb --- --- 0.0m
Live monitoring for Pods
kptop pod <POD> -n <NAMESPACE>
Live monitoring for Containers
kptop pod <POD> -n <NAMESPACE> -c <CONTAINER>
Top PVCs
kptop pvcs <NAMESPACE>NOTE: in this example, all VPCs have the same capacity because this is a testing environment (using nfs-provisioner)
kptop pvcs --all-namespacesNAMESPACE PVC VOLUME CAPACITY USED USED % FREE FREE %
elk-stack elasticsearch-master-elasticsearch-master-0 elasticsearch-master 123.14 gb 21.42 gb 17% 95.43 gb 77%
elk-stack elasticsearch-master-elasticsearch-master-1 elasticsearch-master 123.14 gb 21.42 gb 17% 95.43 gb 77%
elk-stack elasticsearch-master-elasticsearch-master-2 elasticsearch-master ? ? ? ? ?
kafka data-0-kafka-cluster-region1-kafka-0 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region1-kafka-1 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region1-kafka-2 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region2-kafka-0 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region2-kafka-1 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region2-kafka-2 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-1-kafka-cluster-region1-kafka-0 data-1 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-1-kafka-cluster-region1-kafka-1 data-1 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-1-kafka-cluster-region1-kafka-2 data-1 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-2-kafka-cluster-region1-kafka-0 data-2 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-2-kafka-cluster-region1-kafka-1 data-2 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-2-kafka-cluster-region1-kafka-2 data-2 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-3-kafka-cluster-region1-kafka-0 data-3 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-3-kafka-cluster-region1-kafka-1 data-3 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-3-kafka-cluster-region1-kafka-2 data-3 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-kafka-cluster-region1-zookeeper-0 data 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-kafka-cluster-region2-zookeeper-0 data 123.14 gb 21.42 gb 17% 95.43 gb 77%
prometheus my-prometheus-alertmanager storage-volume 123.14 gb 21.42 gb 17% 95.43 gb 77%
prometheus my-prometheus-server storage-volume 123.14 gb 21.42 gb 17% 95.43 gb 77%
Verify Prometheus connectivity
kptop --verify-prometheusSample output
Verifying Prometheus connection: Connected
{
"connected": true,
"status_code": 200,
"reason": "",
"fail_reason": ""
}
Verifying Prometheus Exporters:
* Node Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_versions": {
"1.3.1": "2"
}
}
}
* Kubernetes Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_git_versions": {
"v1.21.0": "3",
"v1.21.14": "1"
}
}
}Check Prometheus metrics
kptop --verify-prometheus --check-metricsSample output
Verifying Prometheus connection: Connected
{
"connected": true,
"status_code": 200,
"reason": "",
"fail_reason": ""
}
Verifying Prometheus Exporters:
* Node Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_versions": {
"1.3.1": "2"
}
}
}
* Kubernetes Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_git_versions": {
"v1.21.0": "3",
"v1.21.14": "1"
}
}
}
0:00:00 0:00:00 Checking Metrics ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% [ container_fs_writes_bytes_total ]
+---------------------------------------------------+---------------+-----------+-----------+
| METRIC | EXPORTER | STATE | COMMENT |
+===================================================+===============+===========+===========+
| node_memory_MemFree_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemAvailable_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemTotal_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Cached_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Buffers_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapTotal_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapFree_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapCached_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_cpu_seconds_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load1 | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load5 | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load15 | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_physical_cores | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_sockets | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| up | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_boot_time_seconds | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_size_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_receive_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_transmit_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_written_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_read_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_cores | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_running_pods | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_last_seen | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_working_set_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_max_usage_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_limit_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_cache | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_swap_limit_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_load_average_10s | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_usage_seconds_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_system_seconds_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_user_seconds_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_cpu_quota | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kube_pod_spec_volumes_persistentvolumeclaims_info | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_capacity_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_used_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_available_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_receive_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_transmit_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_start_time_seconds | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_file_descriptors | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_threads | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_processes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_reads_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+Logging
Default log file location is "/tmp/kptop.log"
Known Issues
[1] Node Exporter metrics don't return data
- This is NOT an issue, the node exporter NODE label change from version to another, currently we encountered only "kubernetes_node" or "node"
- "node" is the default, to fix the it you can change it with the "KPTOP_NODE_EXPORTER_NODE_LABEL" Environment variables
export KPTOP_NODE_EXPORTER_NODE_LABEL="node" # default
export KPTOP_NODE_EXPORTER_NODE_LABEL="kubernetes_node"auto detection of exporters verstions can be implemented later (if needed).
Reach me anytime on Linkedin