Kube-Prometheus-Top [ kptop ]

A Python tool that provides Monitoring for Kubernetes Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metircs

kptop2.mp4

📌 Try out the new beta features

kptop_0.0.7 (Introducing K8s Pod port-forward connection method))

Environment variables

Motivation

The resources metrics provided by the K8s APIs are very limited compared what what's scraped by Prometheus.

This tool is using Prometheus as a data source for metrics to display all the needed informations right on the terminal.

Project Status

Additional

Check the connection with Prometheus Server
Check the existence of the needed Prometheus metrics _{[ good for similar use cases ]}

Installation

Compatible with Python 3.6+

on PyPi

pip3 install kptop --upgrade

Environment Variables

ENV	Description	Default	Required
`KPTOP_PROMETHEUS_SERVER`	Prometheus server URL		Yes
`KPTOP_BASIC_AUTH_ENABLED`	Whether basic authentication is needed to connect to Prometheus	False	No
`KPTOP_PROMETHEUS_USERNAME`	Prometheus username		No
`KPTOP_PROMETHEUS_PASSWORD`	Prometheus password		No
`KPTOP_INSECURE`	Verify SSL certificate	False	No
`KPTOP_NODE_EXPORTER_NODE_LABEL`	node exporter "node label"	"node"	NO
`KPTOP_START_GRAPHS_WITH_ZERO`	By default graphs begin with '0' to let the graph take its full hight	True	NO
`KPTOP_LOGGING_DIR`	Choose a different logging directory	/tmp/	NO
`KPTOP_GRAPH_WIDTH`	Choose a custom graphs width	45	NO

CLI Arguments

ENV	Description	Default
`--namespace`, `-n`	Specify a Kubernetes Namespace	default
`--all-namespaces`, -A
`--container`, `-c`	Specify a container
`--interval`, `-i`	Live monitoring update interval	8 _{[NOTE: the actuall update depends on the Prometheus scaping interval (15s by default)]}
`--debug`, `-d`	Enable debugging logging mode	False
`--verify-prometheus`, `-V`	Verify connectivity to Prometheus server & check the existence of the needed exporters
`--sort-by-mem-usage`, `-s`	Sort top result by memory usage	False
`--check-metrics` , `-C`	Check the existence of the needed metrics, needs to be done with `-V`	False

Usage

Different ways to connect to Prometheus server:

You have direct access to it (Like in dev environments)
Prometheus is exposed publically/over-vpn (mostly with an Ingress)
You can use kubectl port-forward command
You also can run kptop as a Kubernetes pod (to be built))

export KPTOP_PROMETHEUS_SERVER="http://prometheus.home-lab.com"

Top nodes

kptop nodes

NODE      MEM TOTAL    MEM USAGE    MEM FREE      CPU CORES  CPU USAGE%      RUNNING PODS
worker-1  19.6 gb      16.92 gb     2.69 gb               6  9%                        14
worker-2  19.6 gb      9.52 gb      10.08 gb              6  9%                        27

kptop nodes --option / -o 🎉

kptop nodes -o json --colorize-json

Under testing (currently tested with EKS)

kptop nodes -o cloud

Live monitoring for Nodes

kptop node <NODE>

Top pods

kptop pods -n <NAMESPACE>

kptop pods -n elk-stack

NAMESPACE    POD                                      MEM LIMIT    MEM USAGE    MEM USAGE %    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
elk-stack    elasticsearch-master-0                   2.0 gb       1.38 gb      68%            2.0 gb           635.25 mb   1000m        0.04m
elk-stack    elasticsearch-master-1                   2.0 gb       1.49 gb      74%            2.0 gb           522.05 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-f8ms7          200.0 mb     85.73 mb     42%            174.16 mb        114.27 mb   1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-pq2wf  200.0 mb     87.73 mb     43%            171.31 mb        112.27 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-r7dht          200.0 mb     119.12 mb    59%            199.52 mb        80.88 mb    1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-lzqdt  200.0 mb     98.66 mb     49%            199.57 mb        101.34 mb   1000m        0.02m
elk-stack    my-kibana-kibana-79448f7fb7-wf4t6        2.0 gb       342.87 mb    16%            618.07 mb        1.67 gb     1000m        0.02m
elk-stack    my-logstash-logstash-0                   1.5 gb       1008.22 mb   65%            1.21 gb          527.78 mb   1000m        0.02m

kptop pod -n kube-system

NAMESPACE    POD                                              MEM LIMIT    MEM USAGE    MEM USAGE%    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
kube-system  coredns-558bd4d5db-nfcjq                         170.0 mb     26.0 mb      15%           42.77 mb         144.0 mb    ---          0.0m
kube-system  coredns-558bd4d5db-vcstr                         170.0 mb     17.45 mb     10%           24.04 mb         152.55 mb   ---          0.0m
kube-system  etcd-master                                      ---          85.02 mb     ---           391.7 mb         ---         ---          0.02m
kube-system  kube-apiserver-master                            ---          635.97 mb    ---           731.13 mb        ---         ---          0.09m
kube-system  kube-controller-manager-master                   ---          95.55 mb     ---           145.41 mb        ---         ---          0.03m
kube-system  kube-proxy-q6nr7                                 ---          27.46 mb     ---           58.91 mb         ---         ---          0.0m
kube-system  kube-proxy-q489q                                 ---          21.98 mb     ---           63.0 mb          ---         ---          0.0m
kube-system  kube-proxy-bghp6                                 ---          22.35 mb     ---           64.1 mb          ---         ---          0.0m
kube-system  kube-scheduler-master                            ---          37.04 mb     ---           61.68 mb         ---         ---          0.0m
kube-system  nfs-subdir-external-provisioner-b97f4d9f5-bjp2h  ---          9.43 mb      ---           37.48 mb         ---         ---          0.0m

Live monitoring for Pods

kptop pod <POD> -n <NAMESPACE>

Live monitoring for Containers

kptop pod <POD> -n <NAMESPACE> -c <CONTAINER>

Top PVCs

kptop pvcs <NAMESPACE>

NOTE: in this example, all VPCs have the same capacity because this is a testing environment (using nfs-provisioner)

kptop pvcs --all-namespaces

NAMESPACE    PVC                                          VOLUME                CAPACITY    USED      USED %    FREE      FREE %
elk-stack    elasticsearch-master-elasticsearch-master-0  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-1  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-2  elasticsearch-master  ?           ?         ?         ?         ?
kafka        data-0-kafka-cluster-region1-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-0         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-1         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-2         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-0         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-1         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-2         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-0         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-1         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-2         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region1-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region2-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-alertmanager                   storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-server                         storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%

Verify Prometheus connectivity

kptop --verify-prometheus

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}

Check Prometheus metrics

kptop --verify-prometheus --check-metrics

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}
 
  0:00:00 0:00:00 Checking Metrics  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  [ container_fs_writes_bytes_total ]
+---------------------------------------------------+---------------+-----------+-----------+
| METRIC                                            | EXPORTER      | STATE     | COMMENT   |
+===================================================+===============+===========+===========+
| node_memory_MemFree_bytes                         | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemAvailable_bytes                    | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemTotal_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Cached_bytes                          | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Buffers_bytes                         | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapTotal_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapFree_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapCached_bytes                      | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_cpu_seconds_total                            | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load1                                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load5                                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load15                                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_physical_cores                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_sockets                               | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| up                                                | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_boot_time_seconds                            | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_size_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_receive_bytes_total                  | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_transmit_bytes_total                 | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_written_bytes_total                     | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_read_bytes_total                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_cores                                 | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_running_pods                              | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_last_seen                               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_working_set_bytes                | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_max_usage_bytes                  | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_limit_bytes                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_cache                            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_swap_limit_bytes            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_load_average_10s                    | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_usage_seconds_total                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_system_seconds_total                | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_user_seconds_total                  | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_cpu_quota                          | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kube_pod_spec_volumes_persistentvolumeclaims_info | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_capacity_bytes               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_used_bytes                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_available_bytes              | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_receive_bytes_total             | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_transmit_bytes_total            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_start_time_seconds                      | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_file_descriptors                        | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_threads                                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_processes                               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_reads_bytes_total                    | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+

Logging

Default log file location is "/tmp/kptop.log"

Known Issues

[1] Node Exporter metrics don't return data

This is NOT an issue, the node exporter NODE label change from version to another, currently we encountered only "kubernetes_node" or "node"
"node" is the default, to fix the it you can change it with the "KPTOP_NODE_EXPORTER_NODE_LABEL" Environment variables

export KPTOP_NODE_EXPORTER_NODE_LABEL="node" # default
export KPTOP_NODE_EXPORTER_NODE_LABEL="kubernetes_node"

auto detection of exporters verstions can be implemented later (if needed).

Reach me anytime on Linkedin

eslam-gomaa/kptop

eslam-gomaa

Reviews

Repository Details