• Stars
    star
    164
  • Rank 225,067 (Top 5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

K8s Event Exporting, Filtering and Alerting in Multi-Tenant Environment

kube-events

Kube-events revolves around Kubernetes Event, covering multi-dimensional processing of them, such as emitting events to sinks, issuing notifications and generating alerts. In some of these dimensions, configurable filtering rules are provided to meet different business needs.

Kube-events contains two major business components, Exporter and Ruler. Exporter watches Kubernetes Event and emits events to sinks. Ruler receives events, filters them by rules, then issues notifications or processes them as alerts which will eventually be sent to alertmanager or webhooks.

Another component called Operator is responsible for the desired state of the previous business components. This relies on the following three crds to achieve:

  • Exporter, which defines a desired Exporter deployment. Operator ensures at all times that a Exporter matching the resource definition is running.
  • Ruler, which defines a desired Ruler deployment. The Operator ensures at all times that a Ruler matching the resource definition is running.
  • Rule, which defines a desired Rule set, which will be used to filter events by Ruler.

To learn more about the CRDs introduced by the kube-events have a look at the design doc.

The architecture diagram is as follows:
Architecture

Quickstart

Install quickly kube-events into your kubernetes cluster with the following commands:

kubectl apply -f https://raw.githubusercontent.com/kubesphere/kube-events/master/config/bundle.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kube-events/master/config/crs/bundle.yaml

Or install it by helm chart.

Videos & blogs

K8s events exporting, filtering, alerting and notification in multi-tenant env

Usage

Rule

Rule is used to define filtering rules for events. Each Rule instance can be configured with a set of rules. When configuring these instances, set a label called kubesphere.io/rule-scope to distinguish the scope of rules, to meet the needs of multi-tenancy.

  • kubesphere.io/rule-scope: cluster
    This level of Rule does not need to specify the namespace when it is configured, it will be stored in the namespace which the operator belongs to. Filtering rules in it will be applied to the events of the entire cluster.
  • kubesphere.io/rule-scope: namespace When configuring a Rule instance at this level, you need to specify the namespace. Rules in it will be applied to the events whose involvedObject.namespace equals to the namespace which this Rule instance belongs to.
  • kubesphere.io/rule-scope: workspace This is an intermediate level of scope control. It will act on events of multiple namespaces. It needs to be specified a label called kubesphere.io/workspace, which will be used to select the namespaces with the same label. Rules in it will be applied to the events whose involvedObject.namespace is in these selected namespaces. (It will be stored in the same namespace to the cluster-level instances)

Exporter

Exporter component is configured by Exporter. Exporter currently supports output events to stdout and webhook. In fact, Ruler component is such a webhook.

If you want to collect events from the exporter's stdout and output them to a specific place, fluentbit is a good choice. To achieve this, by fluentbit-operator which provides flexibility and convenience of fluentbit configuration, just add some Input/Filter/Output plugins as needed. For example, the following provides plugins which will finally output events to es:

apiVersion: logging.kubesphere.io/v1alpha2
kind: Input
metadata:
  name: tail-events
  namespace: kubesphere-logging-system
  labels:
    logging.kubesphere.io/enabled: "true"
    logging.kubesphere.io/component: "events"
spec:
  tail:
    tag: kube_events
    path: /var/log/containers/*_kubesphere-logging-system_events-exporter*.log
    parser: docker
    refreshIntervalSeconds: 10
    memBufLimit: 5MB
    skipLongLines: true
    db: /fluent-bit/tail/pos-events.db
    dbSync: Normal
---
apiVersion: logging.kubesphere.io/v1alpha2
kind: Output
metadata:
  name: es-events
  namespace: kubesphere-logging-system
  labels:
    logging.kubesphere.io/enabled: "true"
    logging.kubesphere.io/component: "events"
spec:
  match: kube_events
  es:
    host: elasticsearch-logging-data.kubesphere-logging-system.svc
    port: 9200
    logstashPrefix: ks-logstash-events
    logstashFormat: true
---
apiVersion: logging.kubesphere.io/v1alpha2
kind: Filter
metadata:
  name: filter-events
  namespace: kubesphere-logging-system
  labels:
    logging.kubesphere.io/enabled: "true"
    logging.kubesphere.io/component: "events"
spec:
  match: kube_events
  filters:
  - parser:
      keyName: log
      parser: json

Ruler

Ruler component is configured by Ruler. Ruler component filters the received events through rules and may generate event notifications or alerts. Alert messages can be configured to be sent to the alertmanager service.

And webhooks can be also configured to receive these notifications or alerts.

More Repositories

1

kubesphere

The container platform tailored for Kubernetes multi-cloud, datacenter, and edge management โŽˆ ๐Ÿ–ฅ โ˜๏ธ
Go
12,855
star
2

kubekey

Install Kubernetes/K3s only, both Kubernetes/K3s and KubeSphere, and related cloud-native add-ons, it supports all-in-one, multi-node, and HA ๐Ÿ”ฅ โŽˆ ๐Ÿณ
Go
2,027
star
3

kubeeye

KubeEye aims to find various problems on Kubernetes, such as application misconfiguration, unhealthy cluster components and node problems.
Go
786
star
4

console

KubeSphere Console is the web-based UI for KubeSphere clusters.
JavaScript
558
star
5

ks-installer

Install KubeSphere on existing Kubernetes cluster
Jinja
507
star
6

notification-manager

K8s native notification management with multi-tenancy support
Go
218
star
7

ks-devops

This is a cloud-native application that focuses on the DevOps area.
Go
158
star
8

community

KubeSphere Community
120
star
9

devops-java-sample

SpringBoot demo for DevOps on KubeSphere
Java
117
star
10

helm-charts

Source & Repo of https://charts.kubesphere.io/main & https://charts.kubesphere.io/test
Mustache
97
star
11

kube-design

Kube Design for KubeSphere Console
JavaScript
96
star
12

tower

Proxy for multiple Kubernetes cluster communication
Go
92
star
13

website

KubeSphere website and documentation 3.x
SCSS
78
star
14

devops-agent

Agents for Kubesphere DevOps
Dockerfile
59
star
15

s2ioperator

Operator for Source to image
Go
41
star
16

devops-maven-sample

Java
36
star
17

alert

Go
27
star
18

logsidecar-injector

A MutatingAdmissionWebhook that adds a sidecar to your pod. This sidecar is just for forwarding file log on the volume.
Go
25
star
19

ks-jenkins

Jenkins distribution for Kubesphere
Python
24
star
20

dev-guide

A set of development guidelines for KubeSphere developers.
HTML
19
star
21

s2irun

Run s2i in pod
Go
18
star
22

monitoring-dashboard

Custom Metrics Monitoring Dashboard for Apps on KubeSphere
Go
16
star
23

s2i-java-container

Java S2I Builder image
Shell
15
star
24

event-rule-engine

Go
10
star
25

tutorial

KubeSphere tutorial repo, include samples code, docs etc.
Shell
10
star
26

storageclass-accessor

The storageclass-accessor webhook is an HTTP callback which responds to admission requests.
Go
9
star
27

client-go

Go
9
star
28

devops-go-sample

Go
8
star
29

image-sync-config

8
star
30

im

A general identity management system
Go
8
star
31

api

Go
7
star
32

ksbuilder

A CLI tool helps you to manage the development of kubesphere extensions
Go
7
star
33

s2i-java-runtimeImage

a java runtime image for s2i
Shell
5
star
34

alert-adapter

Go
5
star
35

s2i-python-container

Shell
5
star
36

pvc-autoresizer

pvc-autoresizer resizes PersistentVolumeClaims (PVCs) when the free amount of storage is below the threshold.
Go
5
star
37

test-infra

Dockerfile
4
star
38

prometheus-example-app

Smarty
4
star
39

devops-python-sample

devops-python-sample for Kubesphere
Python
4
star
40

kubeeye-console

TypeScript
3
star
41

sonargo

Go
3
star
42

cluster-api-provider-qingcloud

The QingCloud provider implementation of the Cluster Management API
Go
3
star
43

create-ks-project

A CLI tool enables you to quickly set up a KubeSphere Console extension project.
TypeScript
3
star
44

s2i-nodejs-container

Shell
3
star
45

ks-prometheus

Jsonnet
2
star
46

aks-kubesphere-linux

KubeSphere on Azure AKS
2
star
47

openpitrix-jobs

Go
2
star
48

alertmanager-kit

alertmanager-kit encapsulates the interface to alertmanager in a neat way.
Go
2
star
49

container-common-scripts

Shell
2
star
50

s2i-binary-container

Dockerfile
1
star
51

devops-docs-sample

JavaScript
1
star
52

springboot-sample

Java
1
star
53

extension-samples

JavaScript
1
star
54

s2i-base-container

Shell
1
star
55

artwork

๐ŸŽจKubeSphere-related logos and artwork
1
star
56

.github

1
star
57

kubeocean-api

Go
1
star
58

kubectl

kubectl docker image used in kubesphere.
Dockerfile
1
star
59

design-prototypes

Design prototypes for kubesphere console.
HTML
1
star