• Stars
    star
    615
  • Rank 72,947 (Top 2 %)
  • Language
    C
  • License
    Other
  • Created over 1 year ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

bpftune uses BPF to auto-tune Linux systems

bpftune - BPF driven auto-tuning

bpftune aims to provide lightweight, always-on auto-tuning of system behaviour. The key benefit it provides are

  • by using BPF observability features, we can continuously monitor and adjust system behaviour
  • because we can observe system behaviour at a fine grain (rather than using coarse system-wide stats), we can tune at a finer grain too (individual socket policies, individual device policies etc)

The problem

The Linux kernel contains a large number of tunables; these often take the form of sysctl(8) parameters, and are usually introduced for situations where there is no one "right" answer for a configuration choice. The number of tunables available is quite daunting. On a 6.2 kernel we see

# sysctl --all 2>/dev/null|wc -l
1624

See here for an excellent writeup on network-related tunables..

At the same time, individual systems get a lot less care and adminstrator attention than they used to; phrases like "cattle not pets" exemplify this. Given the modern cloud architectures used for most deployments, most systems never have any human adminstrator interaction after initial provisioning; in fact given the scale requirements, this is often an explicit design goal - "no ssh'ing in!".

These two observations are not unrelated; in an earlier era of fewer, larger systems, tuning by administrators was more feasible.

These trends - system complexity combined with minimal admin interaction suggest a rethink in terms of tunable management.

A lot of lore accumulates around these tunables, and to help clarify why we developed bpftune, we will use a straw-man version of the approach taken with tunables:

"find the set of magic numbers that will work for the system forever"

This is obviously a caricature of how administrators approach the problem, but it does highlight a critical implicit assumption - that systems are static.

And that gets to the "BPF" in bpftune; BPF provides means to carry out low-overhead observability of systems. So not only can we observe the system and tune appropriately, we can also observe the effect of that tuning and re-tune if necessary.

Key design principles

  • Minimize overhead. Use observability features sparingly; do not trace very high frequency events.
  • Be explicit about policy changes providing both a "what" - what change was made - and a "why" - how does it help? syslog logging makes policy actions explicit with explanations
  • Get out of the way of the administrator. We can use BPF observability to see if the admin sets tunable values that we are auto-tuning; if they do, we need to get out of the way and disable auto-tuning of the related feature set.
  • Don't replace tunables with more tunables! bpftune is designed to be zero configuration; there are no options, and we try to avoid magic numbers where possible.
  • Use push-pull approaches. For example, with tcp buffer sizing, we often want to get out of the way of applications and bump up tcp sndbuf and rcvbuf, but at a certain point we run the risk of exhausting TCP memory. We can however monitor if we are approaching TCP memory pressure and if so we can tune down values that we've tuned up. In this way, we can let the system find a balance between providing resources and exhausting them. In some cases, we won't need to tune up values; they may be fine as they are. But in other cases these limits block optimal performance, and if they are raised safely - with awareness of global memory limits - we can get out the way of improved performance. Another concern is that increasing buffer size leads to latency - to handle that, we correlate buffer size changes and TCP smoothed round-trip time; if the correlation between these exceeds a threshold (0.7) we stop increasing buffer size.

Concepts

The key components are

  • tuners: each tuner manages tunables and handles events sent from BPF programs to userspace via the shared ring buffer. Each tuner has an associated set of tunables that it manages.

  • optional strategies: a tuner can specify multiple strategies; after running for a while a strategy times out and we assess if a better strategy is available. Each strategy specifies a

    • name
    • description
    • timeout
    • evaluation function
    • set of BPF program names in tuner associated with strategy

    Strategies are optional and should be set in the tuner init() method via bpftune_strategies_add(). See test/strategy for a coded example. When a strategy times out, the various evaluation functions are called and the highest-value evaluation dictates the next stratgey.

    Strategies provide a way of providing multiple schemes for auto-tuning the same set of tunables, where the choice is guided by an evaluation of the effectiveness of the strategies.

  • events specify a

    • tuner id: which tuner the event is destined for
    • a scenario: what happened
    • an associated netns (if supported)
    • information about the event (IP address etc)
  • the tuner then responds to the event guided by the active strategy; increase or decrease a tunable value, etc. Describing the event in the log is key; this allows an admin to understand what changed and why.

Architecture

  • bpftune is a daemon which manages a set of .so plugin tuners; each of these is a shared object that is loaded on start-up.
  • tuners can be enabled or disabled; a tuner is automatically disabled if the admin changes associated tunables manually.
  • tuners share a global BPF ring buffer which allows posting of events from BPF programs to userspace. For example, if the sysctl tuner sees a systl being set, it posts an event.
  • each tuner has an associated id (set when it is loaded), and events posted contain the tuner id.
  • each tuner has a BPF component (built using a BPF skeleton) and a userspace component. The latter has init(), fini() and event_handler() entrypoints. When an event is received, the tuner id is used to identify the appropriate event handler and its event_handler() callback function is run.
  • init, fini and event_handler functions are loaded from the tuner .so object.
  • BPF components should include bpftune.bpf.h; it contains the common map definitions (ringbuf, etc) and shared variables such as learning rate and tuner ids that each tuner needs.

Supported tuners

  • TCP connection tuner: auto-tune choice of congestion control algorithm. See bpftune-tcp-conn (8).
  • neighbour table tuner: auto-tune neighbour table sizes by growing tables when approaching full. See bpftune-neigh (8).
  • route table tuner: auto-tune route table size by growing tables when approaching full. See bpftune-route (8).
  • sysctl tuner: monitor sysctl setting and if it collides with an auto-tuned sysctl value, disable the associated tuner. See bpftune-sysctl (8).
  • TCP buffer tuner: auto-tune max and initial buffer sizes. See bpftune-tcp-buffer (8).
  • net buffer tuner: auto-tune tunables related to core networking. See bpftune-net-buffer (8).
  • netns tuner: notices addition and removal of network namespaces, which helps power namespace awareness for bpftune as a whole. Namespace awareness is important as we want to be able to auto-tune containers also. See bpftune-netns (8).

Code organization

Both core bpftune.c and individual tuners use the libbpftune library. It handles logging, tuner init/fini, and BPF init/fini.

Each tuner shared object defines an init(), fini() and event_handler() function. These respectively set up and clean up BPF and handle events that originate from the BPF code.

Getting Started

If building the repository manually, simply run

$ make ; sudo make install

at the top-level of the repository. bpftune also supports a

$ make pkg

target, which will make a bpftune RPM. See ./buildrpm/bpftune.spec

To build the following packages are needed (names may vary by distro);

  • libbpf, libbpf-devel >= 0.6
  • libcap-devel
  • bpftool >= 4.18
  • libnl3-devel
  • clang >= 11
  • llvm >= 11
  • python3-docutils

From the kernel side, the kernel needs to support BPF ring buffer (around the 5.6 kernel, though 5.4 is supported on Oracle Linux as ring buffer support was backported), and kernel BTF is required (CONFIG_DEBUG_INFO_BTF=y). Verify /sys/kernel/btf/vmlinux is present.

To enable bpftune as a service

$ sudo service bpftune start

...and to enable it by default

$ sudo systemctl enable bpftune

bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out.

bpftune can also be run in the foreground as a program; to redirect output to stdout/stderr, run

$ sudo bpftune -s

On exit, bpftune will summarize any tuning done.

Tests

Tests are supplied for each tuner in the tests/ subdirectory. "make test" runs all the tests. Tests use network namespaces to simulate interactions with remote hosts. See ./TESTING.md for more details.

Does my system support bpftune?

Simply run "bpftune -S" to see:

$ bpftune -S
bpftune works fully
bpftune supports per-netns policy (via netns cookie)

Two aspects are important here

  • does the system support fentry/fexit etc? If so full support is likely.
  • does the system support network namespace cookies? If so per-network-namespace policy is supported.

Demo

Simply starting bpftune and observing changes made via /var/log/messages can be instructive. For example, on a standard VM with sysctl defaults, I ran

$ service bpftune start

...and went about normal development activities such as cloning git trees from upstream, building kernels, etc. From the log we see some of the adjustments bpftune made to accommodate these activities

$ sudo grep bpftune /var/log/messages
...
Apr 19 16:14:59 bpftest bpftune[2778]: bpftune works fully
Apr 19 16:14:59 bpftest bpftune[2778]: bpftune supports per-netns policy (via netns cookie)
Apr 19 16:18:40 bpftest bpftune[2778]: Scenario 'specify bbr congestion control' occurred for tunable 'TCP congestion control' in global ns. Because loss rate has exceeded 1 percent for a connection, use bbr congestion control algorithm instead of default
Apr 19 16:18:40 bpftest bpftune[2778]: due to loss events for 145.40.68.75, specify 'bbr' congestion control algorithm
Apr 19 16:26:53 bpftest bpftune[2778]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Apr 19 16:26:53 bpftest bpftune[2778]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 6291456) -> (4096 131072 7864320)
Apr 19 16:26:53 bpftest bpftune[2778]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Apr 19 16:26:53 bpftest bpftune[2778]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 7864320) -> (4096 131072 9830400)
Apr 19 16:29:04 bpftest bpftune[2778]: Scenario 'specify bbr congestion control' occurred for tunable 'TCP congestion control' in global ns. Because loss rate has exceeded 1 percent for a connection, use bbr congestion control algorithm instead of default
Apr 19 16:29:04 bpftest bpftune[2778]: due to loss events for 140.91.12.81, specify 'bbr' congestion control algorithm

To deterministically trigger bpftune behaviour, one approach we can take is to download a large file with inappropriate settings.

In one window, set tcp rmem max to a too-low value, and run bpftune as a program logging to stdout/stderr (-s):

$ sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 1310720"
net.ipv4.tcp_rmem = 4096 131072 1310720
$ sudo bpftune -s

In another window, wget a large file:

$ wget https://yum.oracle.com/ISOS/OracleLinux/OL8/u7/x86_64/OracleLinux-R8-U7-x86_64-dvd.iso

In the first window, we see bpftune tuning up rmem:

bpftune: bpftune works in legacy mode
bpftune: bpftune does not support per-netns policy (via netns cookie)
bpftune: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
bpftune: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 1310720) -> (4096 131072 1638400)

This occurs multiple times, and on exit (Ctrl+C) we see the summary of changes made:

bpftune: Summary: scenario 'need to increase TCP buffer size(s)' occurred 9 times for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
bpftune: sysctl 'net.ipv4.tcp_rmem' changed from (4096 131072 1310720 ) -> (4096 131072 9765625 )

For more info

See the docs/ subdirectory for manual pages covering bpftune and associated tuners.

Contributing

This project welcomes contributions from the community. Before submitting a pull request, please review our contribution guide

Security

Please consult the security guide for our responsible security vulnerability disclosure process

License

Copyright (c) 2023 Oracle and/or its affiliates.

This software is available to you under

SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note

Being under the terms of the GNU General Public License version 2.

SPDX-URL: https://spdx.org/licenses/GPL-2.0.html

See the license file for more details.

More Repositories

1

graal

GraalVM compiles Java applications into native executables that start instantly, scale fast, and use fewer compute resources 🚀
Java
20,237
star
2

docker-images

Official source of container configurations, images, and examples for Oracle products and projects
Shell
6,160
star
3

opengrok

OpenGrok is a fast and usable source code search and cross reference engine, written in Java
Java
3,971
star
4

truffleruby

A high performance implementation of the Ruby programming language, built on GraalVM.
Ruby
3,018
star
5

helidon

Java libraries for writing microservices
Java
2,596
star
6

visualvm

VisualVM is an All-in-One Java Troubleshooting Tool
Java
2,563
star
7

node-oracledb

Oracle Database driver for Node.js maintained by Oracle Corp.
JavaScript
2,174
star
8

graaljs

A ECMAScript 2022 compliant JavaScript implementation built on GraalVM. With polyglot language interoperability support. Running Node.js applications!
C++
1,418
star
9

tribuo

Tribuo - A Java machine learning library
Java
1,211
star
10

railcar

RailCar: Rust implementation of the Open Containers Initiative oci-runtime
Rust
1,115
star
11

oracle-db-examples

Examples of applications and tool usage for Oracle Database
Java
958
star
12

graalpython

A Python 3 implementation built on GraalVM
Python
957
star
13

mysql-operator

Create, operate and scale self-healing MySQL clusters in Kubernetes
Go
868
star
14

python-cx_Oracle

Python interface to Oracle Database now superseded by python-oracledb
C
861
star
15

vagrant-projects

Vagrant projects for Oracle products and other examples
Shell
840
star
16

graphpipe

Machine Learning Model Deployment Made Simple
Makefile
725
star
17

terraform-provider-oci

Terraform Oracle Cloud Infrastructure provider
Go
622
star
18

smith

Smith: A microcontainer builder
Go
602
star
19

fastr

A high-performance implementation of the R programming language, built on GraalVM.
Java
598
star
20

oraclejet

Oracle JET is a modular JavaScript Extension Toolkit for developers working on client-side applications.
479
star
21

db-sample-schemas

Oracle Database Sample Schemas
PLSQL
435
star
22

coherence

Oracle Coherence Community Edition
Java
408
star
23

dotnet-db-samples

.NET code samples for Oracle database developers #OracleDotNet
C#
381
star
24

apex

Official Oracle APEX repo for sample code, starter apps, plug-ins, and more! #orclapex
354
star
25

graalvm-reachability-metadata

Repository which contains community-driven collection of GraalVM reachability metadata for open-source libraries.
Java
352
star
26

oci-cli

Command Line Interface for Oracle Cloud Infrastructure
Python
343
star
27

centos2ol

Script and documentation to switch CentOS/Rocky Linux to Oracle Linux
Shell
330
star
28

oci-python-sdk

Oracle Cloud Infrastructure SDK for Python
Python
303
star
29

crashcart

CrashCart: sideload binaries into a running container
Rust
275
star
30

oracle-db-tools

This project is a repository of sample code that will demonstrate various concepts to assist developers in building applications around Oracle Database technologies. SDKs and scripts will be available to integrate with SQL Developer, Data Modeler, Oracle REST Data Services and DBaaS.
JavaScript
274
star
31

python-oracledb

Python driver for Oracle Database conforming to the Python DB API 2.0 specification. This is the renamed, new major release of cx_Oracle
Python
256
star
32

oci-designer-toolkit

OCI designer toolKIT (OKIT) is a set of tools for enabling design, deploy and visualise OCI environments through a graphical web based interface.
JavaScript
254
star
33

linux-uek

Oracle Linux UEK: Unbreakable Enterprise Kernel
253
star
34

odpi

ODPI-C: Oracle Database Programming Interface for Drivers and Applications
C
235
star
35

weblogic-kubernetes-operator

WebLogic Kubernetes Operator
Java
225
star
36

netsuite-suitecloud-sdk

SuiteCloud Software Development Kit (SuiteCloud SDK) are the set of tools that allow you to customize accounts and create SuiteApps through SuiteCloud Development Framework (SDF).
JavaScript
184
star
37

oracle-r2dbc

R2DBC Driver for Oracle Database
Java
183
star
38

javavscode

Java platform support for Visual Studio Code for full featured Java development (edit-compile-debug & test cycle)
TypeScript
179
star
39

terraform-kubernetes-installer

Terraform Installer for Kubernetes on Oracle Cloud Infrastructure
HCL
178
star
40

oci-java-sdk

Oracle Cloud Infrastructure SDK for Java
Java
163
star
41

pgql-lang

PGQL is an SQL-based query language for property graphs
Java
158
star
42

speedle

Speedle is an open source project for access control.
Go
156
star
43

oci-ansible-collection

Oracle Cloud Infrastructure Ansible Collection provides an easy way to provision and manage resources in Oracle Cloud using Ansible.
Python
152
star
44

oci-go-sdk

Go SDK for Oracle Cloud Infrastructure
Go
150
star
45

wookiee

Scala based lightweight service framework using zookeeper, gRPC, and other popular technologies.
Scala
143
star
46

cordova-plugin-wkwebview-file-xhr

Cordova Plugin for WebView File XHR
JavaScript
140
star
47

weblogic-deploy-tooling

WebLogic Deploy Tooling
Python
140
star
48

solaris-userland

Open Source software in Solaris using gmake based build system to drive building various software components.
C
137
star
49

macaron

Macaron is an extensible supply-chain security analysis framework from Oracle Labs that supports a wide range of build systems and CI/CD services. It can be used to prevent supply chain attacks, detect malicious Python packages, or check conformance to frameworks, such as SLSA. Documentation:
Python
134
star
50

sd4j

Stable diffusion pipeline in Java using ONNX Runtime
Java
131
star
51

analytical-sql-examples

NO LONGER MAINTAINED. Code samples for Oracle's analytical SQL features
PLSQL
127
star
52

container-images

Oracle Linux container images
124
star
53

oracle-database-operator

The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases. It eliminates the dependency on a human operator or administrator for the majority of database operations.
Go
120
star
54

oracle-linux

Scripts, examples, and tutorials to get started with Oracle Linux
Shell
115
star
55

kernel-fuzzing

Fuzzers for the Linux kernel
Hack
109
star
56

oci-cloud-controller-manager

Kubernetes Cloud Controller Manager implementation for Oracle Cloud Infrastucture
Go
109
star
57

oci-ansible-modules

DEPRECATED - Please migrate to the new OCI Ansible collection (https://github.com/oracle/oci-ansible-collection).
Python
106
star
58

cloud-native-devops-workshop

Oracle's Cloud Native and DevOps Workshop on Oracle Cloud
JavaScript
106
star
59

weblogic-monitoring-exporter

WebLogic Monitoring Exporter exposes metrics and monitoring data through REST APIs for consumption by other tools (e.g. Prometheus)
Java
105
star
60

macest

Model Agnostic Confidence Estimator (MACEST) - A Python library for calibrating Machine Learning models' confidence scores
Jupyter Notebook
99
star
61

hospitality-api-docs

This repository stores REST API specifications and accompanying Postman collections for Oracle Hospitality APIs.
HTML
99
star
62

cloudtestdrive

HTML
95
star
63

font-apex

Font APEX is an open source icon library from the Oracle APEX team.
CSS
93
star
64

coherence-operator

Oracle Coherence Operator
Go
93
star
65

ktf

Kernel Test Framework - a unit test framework for the Linux kernel
Shell
88
star
66

kubernetes-vault-kms-plugin

Go
74
star
67

dtrace-utils

DTrace-utils contains the DTrace port to Linux
C
74
star
68

oci-grafana-metrics

Grafana datasource plugin for OCI metrics
Go
71
star
69

hiq

HiQ - Observability And Optimization In Modern AI Era
Python
68
star
70

accelerated-data-science

ADS is the Oracle Data Science Cloud Service's python SDK supporting, model ops (train/eval/deploy), along with running workloads on Jobs and Pipeline resources.
Python
67
star
71

graphpipe-go

GraphPipe for go
Go
66
star
72

soda-for-java

SODA (Simple Oracle Document Access) for Java is an Oracle library for writing Java apps that work with JSON (and not only JSON!) in the Oracle Database. SODA allows your Java app to use the Oracle Database as a NoSQL document store.
Java
65
star
73

oci-typescript-sdk

Oracle Cloud Infrastructure SDK for TypeScript and JavaScript
TypeScript
64
star
74

oracle-functions-samples

Examples demonstrating how to use Oracle Functions
61
star
75

yo

A fast and simple command line OCI client
Python
60
star
76

solaris-ips

Solaris IPS: Image Packaging System
Python
57
star
77

weblogic-image-tool

WebLogic Image Tool
Java
57
star
78

nvm-direct

A C library to support applications that map Non-Volatile Memory into their address space for load/store access.
C
56
star
79

microservices-datadriven

Sample code of application examples to build microservices with converged Oracle database and multi-cloud / hybrid cloud services
CSS
56
star
80

bots-node-sdk

Oracle Bots Node.js SDK
TypeScript
51
star
81

db-appdev-vm

Database Application Development Virtual Machine
Shell
50
star
82

souffle

DEPRECATED. Soufflé is a translator of declarative Datalog programs into the C++ language.
C++
49
star
83

xml-sample-demo

Oracle Database XMLDB Code samples
VBScript
48
star
84

oci-service-broker

Oracle Cloud Infrastructure Service Broker is an open source implementation of Open service broker API Spec for OCI services. Customers can use this implementation to install Open Service Broker in Oracle Container Engine for Kubernetes or in other Kubernetes clusters.
Java
48
star
85

free

Free Oracle technologies for Developers
HTML
47
star
86

oraclesolaris-contrib

oraclesolaris-contrib is a repository focussed on the Oracle Solaris 11.4 StatsStore, using methodologies like REST to connect to Oracle Solaris 11.4 and the new features being introduced in Oracle Solaris 11.4 OS.
Jupyter Notebook
46
star
87

oci-utils

Oracle Cloud Infrastructure utilities
Python
45
star
88

content-and-experience-toolkit

The Oracle Content Management Toolkit and SDKs help you develop custom applications that consume content that is managed in the OCM repository. These applications can be developed in the Content Management Cloud or using 3rd party tools.
JavaScript
45
star
89

navarkos

Enables a Kubernetes federation to automatically manage multi-cluster infrastructure
Go
44
star
90

nosql-examples

This is a top level repository for code examples related to the use of Oracle NoSQL Database.
HTML
44
star
91

rwloadsim

RWP*Load Simulator - your tool for scripting, simulation and much more. Like having a bit of bash and SQL, a nip of C or Java, a dash of awk, a grain of sed plus drops of secret sauce in one single tool. See https://blogs.oracle.com/database/rwploadsim-oracle-db-performance-simluator for the announcement on the Oracle database blog.
C
44
star
92

fmw-chef-cookbook

Official repository of samples that show how to use Chef to provision Oracle Fusion Middleware (FMW) products.
Ruby
43
star
93

oci-dotnet-sdk

Oracle Cloud Infrastructure SDK for .NET
C#
43
star
94

graphpipe-py

GraphPipe for python
Python
42
star
95

terraform-examples

Terraform Examples for Oracle Cloud Infrastructure and Platfrom
41
star
96

heatwave-tpch

SQL scripts for HeatWave benchmarking
41
star
97

oci-data-science-ai-samples

This repo contains a series of tutorials and code examples highlighting different features of the OCI Data Science and AI services, along with a release vehicle for experimental programs.
Jupyter Notebook
41
star
98

oracle-db-appdev-monitoring

Metrics exporter and samples for unified observability for data-centric app dev and microservices
Go
41
star
99

dbt-oracle

dbt (data build tool) adapter for Oracle Autonomous Database
Python
40
star
100

offline-persistence-toolkit

Offline Persistence Toolkit for Javascript Applications
JavaScript
40
star