• Stars
    star
    161
  • Rank 233,470 (Top 5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 6 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Generate a Kubernetes Event when a Pod's container has been OOMKilled

kubernetes-oom-event-generator

Build Status

Generates Kubernetes Event when a container is starting and indicates that it was previously out-of-memory killed.

Design

The Controller listens to the Kubernetes API for new Events and changes to Events. Every time a notification regarding an Event is received it checks whether this Event refers to a "ContainerStarted" event, based on the Reason for the Event and the Kind of the involved object. If this is the case and the Event constitutes a change (meaning it is not a not-changing update, which happens when the resync, that is executed every two minutes, is run) it checks the underlying Pod resource. Should the LastTerminationState of the Pod refer to an OOM kill the controller will emit a Kubernetes Event with a level of Warning and a reason of PreviousContainerWasOOMKilled.

Usage

Usage:
  kubernetes-oom-event-generator [OPTIONS]

Application Options:
  -v, --verbose= Show verbose debug information [$VERBOSE]
      --version  Print version information

Help Options:
  -h, --help     Show this help message

Run the pre-built image xingse/kubernetes-oom-event-generator locally (with local permission):

echo VERBOSE=2 >> .env
docker run --env-file=.env -v $HOME/.kube/config:/root/.kube/config xingse/kubernetes-oom-event-generator

Deployment

Example Clusterrole:

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: xing:controller:kubernetes-oom-event-generator
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
      - list
      - watch

Run this controller on Kubernetes with the following commands:

kubectl create serviceaccount kubernetes-oom-event-generator \
  --namespace=kube-system

kubectl create -f path/to/example-clusterrole.yml
# alternatively run: `cat | kubectl create -f -` and paste the above example, hit Ctrl+D afterwards.

kubectl create clusterrolebinding xing:controller:kubernetes-oom-event-generator \
  --clusterrole=xing:controller:kubernetes-oom-event-generator \
  --serviceaccount=kube-system:kubernetes-oom-event-generator

kubectl run kubernetes-oom-event-generator \
  --image=xingse/kubernetes-oom-event-generator \
  --env=VERBOSE=2 \
  --serviceaccount=kubernetes-oom-event-generator \
  --namespace=kube-system

Alerting on OOM killed pods

There are many different ways to send alerts when an OOM occurs. We just want to mention two of them here.

Forwarding OOM events to Graylog

Graylog is a popular log management solution, and it includes an alerting feature. See the Graylog docs for more details.

At XING we forward all Kubernetes cluster events to Graylog using our kubernetes-event-forwarder-gelf. This allows us to configure alerts whenever a PreviousContainerWasOOMKilled event generated by the kubernetes-oom-event-generator occurs.

Using kube-state-metrics and Prometheus alerts

When kube-state-metrics is deployed in the cluster and a Prometheus installation is scraping the metrics, you can alert on OOM-killed pods using the prometheus alert manager.

Example alert:

alert: ComponentOutOfMemory
expr: sum_over_time(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[5m])
  > 0
for: 10s
labels:
  severity: warning
annotations:
  description: Critical Pod {{$labels.namespace}}/{{$labels.pod}} was OOMKilled.

The downside is that kube_pod_container_status_terminated_reason always returns to 0 once a container starts back up. See the introduction of kube_pod_container_status_last_terminated_reason for more details.

Developing

You will need a working Go installation (1.11+) and the make program. You will also need to clone the project to a place outside you normal go code hierarchy (usually ~/go), as it uses the new Go module system.

All build and install steps are managed in the central Makefile. make test will fetch external dependencies, compile the code and run the tests. If all goes well, hack along and submit a pull request. You might need to modify the go.mod to specify desired constraints on dependencies.

Make sure to run go mod tidy before you check in after changing dependencies in any way.

Releases

Releases are a two-step process, beginning with a manual step:

  • Create a release commit
  • Run make release, which will create an image, retrieve the version from the binary, create a git tag and push both your commit and the tag

The Travis CI run will then realize that the current tag refers to the current master commit and will tag the built docker image accordingly.

More Repositories

1

hops

Universal Development Environment
JavaScript
168
star
2

beetle

High Availability AMQP Messaging With Redundant Queues
Ruby
161
star
3

rails_cursor_pagination

Add cursor pagination to your ActiveRecord backed application
Ruby
112
star
4

XNGMarkdownParser

A Markdown NSAttributedString parser.
Mercury
103
star
5

kubernetes-deployment-restart-controller

Kubernetes controller that restarts Deployments when referenced ConfigMaps or Secrets change.
Go
100
star
6

paperclip-storage-ftp

Allow Paperclip attachments to be stored on FTP servers
Ruby
53
star
7

mango

A fastlane plugin that lets you execute tasks in a managed docker container
Ruby
51
star
8

XNGPodsSynchronizer

Mirrors the CocoaPods you need to GitHub Enterprise
Ruby
40
star
9

kubernetes-event-forwarder-gelf

Forward Kubernetes Events using the Graylog Extended Log Format
Go
36
star
10

xing-android-sdk

The Official XING API client for Java/Android
Java
34
star
11

racktables_api

REST access to racktables objects
Ruby
33
star
12

calabash-launcher

iOS Calabash Launcher is a macOS app that helps you run and manage Calabash tests on your Mac.
Swift
30
star
13

XNGMacBuildNodeSetup

These are the ansible playbooks we use to configure our Mac machines for usage with Jenkins CI.
Shell
25
star
14

xing-api-samples

Code samples to get started with the XING API
HTML
24
star
15

pandan

Retrieve Xcode dependency information with ease
Ruby
24
star
16

jungle

Complexity metrics for Cocoapods and SwiftPM
Swift
21
star
17

cloudinit-vmware-guestinfo

Python
19
star
18

xing_api

Offical ruby client for the XING-API, providing easy access to all API endpoints and simplifies response parsing, error handling and tries to ease the oauth pain.
Ruby
19
star
19

selector-specificity

Mean selector specificity of a CSS style sheet
Ruby
18
star
20

fpm-fry

deep-fried package builder
Ruby
18
star
21

jira-bookmarklets

A collection of bookmarklets that helps you to print your JIRA tickets
JavaScript
16
star
22

vpim

a clone of the vpim project on rubyforge (patched geolocation support)
Ruby
16
star
23

israkel

Collection of common rake tasks for the iPhone Simulator.
Ruby
15
star
24

oembed_provider_engine

Mountable engine to turn your Rails 3 app into an oembed provider
Ruby
14
star
25

recrep

Generate crash reports with ease πŸ‘Œ
Rust
13
star
26

passenger-prometheus-exporter-app

Passenger application that exports Passenger metrics in a Prometheus format
HTML
11
star
27

mire

A Ruby method usage analyzer
Ruby
10
star
28

pdf_cover

Generate PDF cover images with Paperclip and Carrierwave
Ruby
8
star
29

cross-application-csrf-prevention

Cross-site request forgery prevention done right
8
star
30

jQuery.autocompletr

A new, better and more extensible jQuery plugin for autocompletion and (ajax) suggesting
JavaScript
8
star
31

perl-beetle

High availability AMQP messaging with redundant queues
Perl
8
star
32

UnifiedActionHandling

A sample project for the CocoaHeads HH presentation in January 2018. It shows our approach to handle actions like cell and button taps in a unified way.
Swift
8
star
33

amiando

Amiando Api Ruby implementation
Ruby
8
star
34

rbenv-patchsets

A collection of definitions for ruby-build to install rubies patched from https://github.com/skaes/rvm-patchsets for rbenv
Ruby
7
star
35

slouch

Backbone Scaffolding
Ruby
7
star
36

canarist

TypeScript
7
star
37

gem_version_check

Check gem dependencies of ruby apps and generate report
Ruby
7
star
38

java-beetle

Java version of the Beetle client library
Java
6
star
39

curl-useragent

UserAgent based on libcurl
Perl
6
star
40

rlsr

create npm releses and changelogs from a mono repo
TypeScript
5
star
41

test-spec-script

A script that converts tests from test/spec to Rails 2.3+ standard test syntax
4
star
42

elixir_logjam_agent

Logjam agent implementation for Elixir / Phoenix
Elixir
3
star
43

gearman-server

Gearman C Server
C
3
star
44

XNGOAuth1Client

Objective-C
3
star
45

logjam-agent-go

Golang logjam agent
Go
3
star
46

game_of_life_web

Web visualisation for Game of Life
Elixir
2
star
47

scratcher

Backbone lightweight framework giving simple structure to make life a little easier
JavaScript
2
star
48

XNGAPIClientTester

The client tester for the XINGAPIClient
Objective-C
2
star
49

cookiebara

session access for capybara
Ruby
2
star
50

xing_ad_tracker

Official XNGAdTracker plugin for iOS/Android
Kotlin
2
star
51

actionlint

Actionlint as wasm
JavaScript
1
star
52

rbenv-mygemfile

Tiny rbenv plugin which sets `BUNDLE_GEMFILE` to `Mygemfile` if file exists
Shell
1
star
53

unified_csrf_prevention

Self-healing, stateless and programming language agnostic CSRF prevention library for Rails
Ruby
1
star
54

msteams_hermes

Send messages to any Microsoft Teams' channel as easy as possible
Ruby
1
star
55

icu4r

Ruby Bindings for icu4c (derived from an installed version of icu4r)
C
1
star
56

btn

Bitrise Teams Notifier - Send out messages to MS Teams if Bitrise builds failed, succeeded, were aborted.
Ruby
1
star