• Stars
    star
    4,761
  • Rank 8,863 (Top 0.2 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Scalable and efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.

Kubernetes Metrics Server

Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.

Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler. Metrics API can also be accessed by kubectl top, making it easier to debug autoscaling pipelines.

Metrics Server is not meant for non-autoscaling purposes. For example, don't use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics. In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.

Metrics Server offers:

  • A single deployment that works on most clusters (see Requirements)
  • Fast autoscaling, collecting metrics every 15 seconds.
  • Resource efficiency, using 1 mili core of CPU and 2 MB of memory for each node in a cluster.
  • Scalable support up to 5,000 node clusters.

Use cases

You can use Metrics Server for:

Don't use Metrics Server when you need:

  • Non-Kubernetes clusters
  • An accurate source of resource usage metrics
  • Horizontal autoscaling based on other resources than CPU/Memory

For unsupported use cases, check out full monitoring solutions like Prometheus.

Requirements

Metrics Server has specific requirements for cluster and network configuration. These requirements aren't the default for all cluster distributions. Please ensure that your cluster distribution supports these requirements before using Metrics Server:

  • The kube-apiserver must enable an aggregation layer.
  • Nodes must have Webhook authentication and authorization enabled.
  • Kubelet certificate needs to be signed by cluster Certificate Authority (or disable certificate validation by passing --kubelet-insecure-tls to Metrics Server)
  • Container runtime must implement a container metrics RPCs (or have cAdvisor support)
  • Network should support following communication:
    • Control plane to Metrics Server. Control plane node needs to reach Metrics Server's pod IP and port 10250 (or node IP and custom port if hostNetwork is enabled). Read more about control plane to node communication.
    • Metrics Server to Kubelet on all nodes. Metrics server needs to reach node address and Kubelet port. Addresses and ports are configured in Kubelet and published as part of Node object. Addresses in .status.addresses and port in .status.daemonEndpoints.kubeletEndpoint.port field (default 10250). Metrics Server will pick first node address based on the list provided by kubelet-preferred-address-types command line flag (default InternalIP,ExternalIP,Hostname in manifests).

Installation

Metrics Server can be installed either directly from YAML manifest or via the official Helm chart. To install the latest Metrics Server release from the components.yaml manifest, run the following command.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Installation instructions for previous releases can be found in Metrics Server releases.

Compatibility Matrix

Metrics Server Metrics API group/version Supported Kubernetes version
0.6.x metrics.k8s.io/v1beta1 1.19+
0.5.x metrics.k8s.io/v1beta1 *1.8+
0.4.x metrics.k8s.io/v1beta1 *1.8+
0.3.x metrics.k8s.io/v1beta1 1.8-1.21

*Kubernetes versions lower than v1.16 require passing the --authorization-always-allow-paths=/livez,/readyz command line flag

High Availability

Metrics Server can be installed in high availability mode directly from a YAML manifest or via the official Helm chart by setting the replicas value greater than 1. To install the latest Metrics Server release in high availability mode from the high-availability.yaml manifest, run the following command.

On Kubernetes v1.21+:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml

On Kubernetes v1.19-1.21:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml

Note that this configuration requires having a cluster with at least 2 nodes on which Metrics Server can be scheduled.

Also, to maximize the efficiency of this highly available configuration, it is recommended to add the --enable-aggregator-routing=true CLI flag to the kube-apiserver so that requests sent to Metrics Server are load balanced between the 2 instances.

Helm Chart

The Helm chart is maintained as an additional component within this repo and released into a chart repository backed on the gh-pages branch. A new version of the chart will be released for each Metrics Server release and can also be released independently if there is a need. The chart on the master branch shouldn't be referenced directly as it might contain modifications since it was last released, to view the chart code use the chart release tag.

Security context

Metrics Server requires the CAP_NET_BIND_SERVICE capability in order to bind to a privileged ports as non-root. If you are running Metrics Server in an environment that uses PSSs or other mechanisms to restrict pod capabilities, ensure that Metrics Server is allowed to use this capability. This applies even if you use the --secure-port flag to change the port that Metrics Server binds to a non-privileged port.

Scaling

Starting from v0.5.0 Metrics Server comes with default resource requests that should guarantee good performance for most cluster configurations up to 100 nodes:

  • 100m core of CPU
  • 200MiB of memory

Metrics Server resource usage depends on multiple independent dimensions, creating a Scalability Envelope. Default Metrics Server configuration should work in clusters that don't exceed any of the thresholds listed below:

Quantity Namespace threshold Cluster threshold
#Nodes n/a 100
#Pods per node 70 70
#Deployments with HPAs 100 100

Resources can be adjusted proportionally based on number of nodes in the cluster. For clusters of more than 100 nodes, allocate additionally:

  • 1m core per node
  • 2MiB memory per node

You can use the same approach to lower resource requests, but there is a boundary where this may impact other scalability dimensions like maximum number of pods per node.

Configuration

Depending on your cluster setup, you may also need to change flags passed to the Metrics Server container. Most useful flags:

  • --kubelet-preferred-address-types - The priority of node address types used when determining an address for connecting to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])
  • --kubelet-insecure-tls - Do not verify the CA of serving certificates presented by Kubelets. For testing purposes only.
  • --requestheader-client-ca-file - Specify a root certificate bundle for verifying client certificates on incoming requests.
  • --node-selector -Can complete to scrape the metrics from the Specified nodes based on labels

You can get a full list of Metrics Server configuration flags by running:

docker run --rm registry.k8s.io/metrics-server/metrics-server:v0.6.0 --help

Design

Metrics Server is a component in the core metrics pipeline described in Kubernetes monitoring architecture.

For more information, see:

Have a question?

Before posting an issue, first checkout Frequently Asked Questions and Known Issues.

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this project at:

This project is maintained by SIG Instrumentation

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

More Repositories

1

kubespray

Deploy a Production Ready Kubernetes Cluster
Jinja
14,679
star
2

kind

Kubernetes IN Docker - local clusters for testing Kubernetes
Go
13,222
star
3

kustomize

Customization of kubernetes YAML configurations
Go
10,363
star
4

kubebuilder

Kubebuilder - SDK for building Kubernetes APIs using CRDs
Go
7,716
star
5

external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
Go
6,672
star
6

krew

📦 Find and install kubectl plugins
Go
6,132
star
7

aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
Go
3,921
star
8

descheduler

Descheduler for Kubernetes
Go
3,444
star
9

cluster-api

Home for Cluster API, a subproject of sig-cluster-lifecycle
Go
2,944
star
10

kui

A hybrid command-line/UI development experience for cloud-native development
TypeScript
2,746
star
11

nfs-subdir-external-provisioner

Dynamic sub-dir volume provisioner on a remote NFS server.
Shell
2,378
star
12

kwok

Kubernetes WithOut Kubelet - Simulates thousands of Nodes and Clusters.
Go
2,304
star
13

controller-runtime

Repo for the controller-runtime subproject of kubebuilder (sig-apimachinery)
Go
2,240
star
14

aws-iam-authenticator

A tool to use AWS IAM credentials to authenticate to a Kubernetes cluster
Go
2,008
star
15

prometheus-adapter

An implementation of the custom.metrics.k8s.io API using Prometheus
Go
1,662
star
16

gateway-api

Repository for the next iteration of composite service (e.g. Ingress) and load balancing APIs.
Go
1,582
star
17

cri-tools

CLI and validation tools for Kubelet Container Runtime Interface (CRI) .
Go
1,333
star
18

secrets-store-csi-driver

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a CSI volume.
Go
1,177
star
19

kueue

Kubernetes-native Job Queueing
Go
1,144
star
20

scheduler-plugins

Repository for out-of-tree scheduler plugins based on scheduler framework.
Go
1,015
star
21

sig-storage-local-static-provisioner

Static provisioner of local volumes
Go
1,009
star
22

aws-ebs-csi-driver

CSI driver for Amazon EBS https://aws.amazon.com/ebs/
Go
923
star
23

apiserver-builder-alpha

apiserver-builder-alpha implements libraries and tools to quickly and easily build Kubernetes apiservers/controllers to support custom resource types based on APIServer Aggregation
Go
787
star
24

etcdadm

Go
758
star
25

kube-scheduler-simulator

The simulator for the Kubernetes scheduler
Go
715
star
26

aws-efs-csi-driver

CSI Driver for Amazon EFS https://aws.amazon.com/efs/
Go
683
star
27

controller-tools

Tools to use with the controller-runtime libraries
Go
682
star
28

security-profiles-operator

The Kubernetes Security Profiles Operator
C
649
star
29

krew-index

Plugin index for https://github.com/kubernetes-sigs/krew. This repo is for plugin maintainers.
628
star
30

cluster-api-provider-aws

Kubernetes Cluster API Provider AWS provides consistent deployment and day 2 operations of "self-managed" and EKS Kubernetes clusters on AWS.
Go
618
star
31

node-feature-discovery

Node feature discovery for Kubernetes
Go
595
star
32

hierarchical-namespaces

Home of the Hierarchical Namespace Controller (HNC). Adds hierarchical policies and delegated creation to Kubernetes namespaces for improved in-cluster multitenancy.
Go
583
star
33

cluster-proportional-autoscaler

Kubernetes Cluster Proportional Autoscaler Container
Go
519
star
34

sig-storage-lib-external-provisioner

Go
515
star
35

alibaba-cloud-csi-driver

CSI Plugin for Kubernetes, Support Alibaba Cloud EBS/NAS/OSS/CPFS
Go
511
star
36

application

Application metadata descriptor CRD
Go
488
star
37

custom-metrics-apiserver

Framework for implementing custom metrics support for Kubernetes
Go
457
star
38

e2e-framework

A Go framework for end-to-end testing of components running in Kubernetes clusters.
Go
439
star
39

nfs-ganesha-server-and-external-provisioner

NFS Ganesha Server and Volume Provisioner.
Shell
399
star
40

cluster-capacity

Cluster capacity analysis
Go
390
star
41

karpenter

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
Go
356
star
42

cluster-api-provider-vsphere

Go
349
star
43

apiserver-network-proxy

Go
349
star
44

image-builder

Tools for building Kubernetes disk images
Shell
344
star
45

kubetest2

Kubetest2 is the framework for launching and running end-to-end tests on Kubernetes.
Go
323
star
46

ingress2gateway

Convert Ingress resources to Gateway API resources
Go
301
star
47

bom

A utility to generate SPDX-compliant Bill of Materials manifests
Go
300
star
48

cluster-api-provider-nested

Cluster API Provider for Nested Clusters
Go
294
star
49

vsphere-csi-driver

vSphere storage Container Storage Interface (CSI) plugin
Go
289
star
50

cluster-api-provider-azure

Cluster API implementation for Microsoft Azure
Go
283
star
51

blixt

Layer 4 Kubernetes load-balancer
Rust
268
star
52

cluster-api-provider-openstack

Go
266
star
53

kubebuilder-declarative-pattern

A toolkit for building declarative operators with kubebuilder
Go
248
star
54

kpng

Reworking kube-proxy's architecture
Go
240
star
55

cloud-provider-azure

Cloud provider for Azure
Go
222
star
56

aws-encryption-provider

APIServer encryption provider, backed by AWS KMS
Go
192
star
57

mcs-api

This repository hosts the Multi-Cluster Service APIs. Providers can import packages in this repo to ensure their multi-cluster service controller implementations will be compatible with MCS data planes.
Go
187
star
58

ip-masq-agent

Manage IP masquerade on nodes
Go
180
star
59

zeitgeist

Zeitgeist: the language-agnostic dependency checker
Go
171
star
60

contributor-playground

Dockerfile
171
star
61

cluster-api-provider-gcp

The GCP provider implementation for Cluster API
Go
168
star
62

cluster-addons

Addon operators for Kubernetes clusters.
Go
156
star
63

azurefile-csi-driver

Azure File CSI Driver
Go
155
star
64

gcp-compute-persistent-disk-csi-driver

The Google Compute Engine Persistent Disk (GCE PD) Container Storage Interface (CSI) Storage Plugin.
Go
151
star
65

cli-utils

This repo contains binaries that built from libraries in cli-runtime.
Go
147
star
66

azuredisk-csi-driver

Azure Disk CSI Driver
Go
145
star
67

promo-tools

Container and file artifact promotion tooling for the Kubernetes project
Go
138
star
68

cluster-api-operator

Home for Cluster API Operator, a subproject of sig-cluster-lifecycle
Go
134
star
69

kube-storage-version-migrator

Go
125
star
70

lws

LeaderWorkerSet: An API for deploying a group of pods as a unit of replication
Go
124
star
71

blob-csi-driver

Azure Blob Storage CSI driver
Go
123
star
72

aws-fsx-csi-driver

CSI Driver of Amazon FSx for Lustre https://aws.amazon.com/fsx/lustre/
Go
118
star
73

usage-metrics-collector

High fidelity and scalable capacity and usage metrics for Kubernetes clusters
Go
117
star
74

boskos

Boskos is a resource management service that provides reservation and lifecycle management of a variety of different kinds of resources.
Go
117
star
75

sig-windows-tools

Repository for tools and artifacts related to the sig-windows charter in Kubernetes. Scripts to assist kubeadm and wincat and flannel will be hosted here.
PowerShell
117
star
76

downloadkubernetes

Download kubernetes binaries more easily
Go
115
star
77

cluster-api-provider-digitalocean

The DigitalOcean provider implementation of the Cluster Management API
Go
108
star
78

cluster-api-provider-kubevirt

Cluster API Provider for KubeVirt
Go
103
star
79

kubectl-validate

Go
103
star
80

jobset

JobSet: An API for managing a group of Jobs as a unit
Go
97
star
81

cluster-api-provider-packet

Cluster API Provider Packet (now Equinix Metal)
Go
94
star
82

structured-merge-diff

Test cases and implementation for "server-side apply"
Go
92
star
83

slack-infra

Tooling for kubernetes.slack.com
Go
90
star
84

cluster-api-addon-provider-helm

Cluster API Add-on Provider for Helm is a extends the functionality of Cluster API by providing a solution for managing the installation, configuration, upgrade, and deletion of Cluster add-ons using Helm charts.
Go
85
star
85

dashboard-metrics-scraper

Container to scrape, store, and retrieve a window of time from the Metrics Server.
Go
84
star
86

apiserver-runtime

Libraries for implementing aggregated apiservers
Go
83
star
87

kube-scheduler-wasm-extension

All the things to make the scheduler extendable with wasm.
Go
83
star
88

container-object-storage-interface-controller

Container Object Storage Interface (COSI) controller responsible to manage lifecycle of COSI objects.
Go
83
star
89

cli-experimental

Experimental Kubectl libraries and commands.
Go
82
star
90

gcp-filestore-csi-driver

The Google Cloud Filestore Container Storage Interface (CSI) Plugin.
Go
82
star
91

lwkd

Last Week in Kubernetes Development
HTML
78
star
92

sig-windows-dev-tools

This is a batteries included local development environment for Kubernetes on Windows.
PowerShell
77
star
93

cloud-provider-kind

Cloud provider for KIND clusters
Go
75
star
94

kernel-module-management

The kernel module management operator builds, signs and loads kernel modules in Kubernetes clusters.
Go
75
star
95

cloud-provider-equinix-metal

Kubernetes Cloud Provider for Equinix Metal (formerly Packet Cloud Controller Manager)
Go
71
star
96

reference-docs

Tools to build reference documentation for Kubernetes APIs and CLIs.
HTML
69
star
97

hydrophone

Hydrophone is a lightweight Kubernetes conformance tests runner
Go
63
star
98

community-images

kubectl plugin that displays images running in a Kubernetes cluster that were pulled from community owned repositories and warn the user to switch repositories if needed
Go
61
star
99

wg-policy-prototypes

A place for policy work group related proposals and prototypes.
Go
60
star
100

cluster-api-ipam-provider-in-cluster

An IPAM provider for Cluster API that manages pools of IP addresses using Kubernetes resources.
Go
59
star