• Stars
    star
    387
  • Rank 106,123 (Top 3 %)
  • Language
    Kotlin
  • License
    Apache License 2.0
  • Created over 5 years ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Kotlin multiplatform benchmarking toolkit

kotlinx-benchmark

Kotlin Alpha JetBrains incubator project GitHub license Build status Maven Central Gradle Plugin Portal

kotlinx-benchmark is a toolkit for running benchmarks for multiplatform code written in Kotlin. It is designed to work with Kotlin/JVM, Kotlin/JS, Kotlin/Native, and Kotlin/Wasm (experimental) targets.

To get started, ensure you're using Kotlin 1.9.20 or newer and Gradle 7.4 or newer.

Features

  • Low noise and reliable results
  • Statistical analysis
  • Detailed performance reports

Table of contents

Setting Up a Kotlin Multiplatform Project for Benchmarking

To configure a Kotlin Multiplatform project for benchmarking, follow the steps below. If you want to benchmark only Kotlin/JVM and Java code, you may refer to our comprehensive guide dedicated to setting up benchmarking in those specific project types.

Kotlin DSL
  1. Applying Benchmark Plugin: Apply the benchmark plugin.

    // build.gradle.kts
    plugins {
        id("org.jetbrains.kotlinx.benchmark") version "0.4.10"
    }
  2. Specifying Plugin Repository: Ensure you have the Gradle Plugin Portal for plugin lookup in the list of repositories:

    // settings.gradle.kts
    pluginManagement {
        repositories {
            gradlePluginPortal()
        }
    }
  3. Adding Runtime Dependency: Next, add the kotlinx-benchmark-runtime dependency to the common source set:

    // build.gradle.kts
    kotlin {
        sourceSets {
            commonMain {
                dependencies {
                    implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.10")
                }
            }
        }
    }
  4. Specifying Runtime Repository: Ensure you have mavenCentral() for dependencies lookup in the list of repositories:

    // build.gradle.kts
    repositories {
        mavenCentral()
    }
Groovy DSL
  1. Applying Benchmark Plugin: Apply the benchmark plugin.

    // build.gradle
    plugins {
        id 'org.jetbrains.kotlinx.benchmark' version '0.4.10'
    }
  2. Specifying Plugin Repository: Ensure you have the Gradle Plugin Portal for plugin lookup in the list of repositories:

    // settings.gradle
    pluginManagement {
        repositories {
            gradlePluginPortal()
        }
    }
  3. Adding Runtime Dependency: Next, add the kotlinx-benchmark-runtime dependency to the common source set:

    // build.gradle
    kotlin {
        sourceSets {
            commonMain {
                dependencies {
                    implementation 'org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.10'
                }
            }
        }
    }
  4. Specifying Runtime Repository: Ensure you have mavenCentral() for dependencies lookup in the list of repositories:

    // build.gradle
    repositories {
        mavenCentral()
    }

Target-specific configurations

To run benchmarks on a platform ensure your Kotlin Multiplatform project targets that platform. For different platforms, there may be distinct requirements and settings that need to be configured. The guide below contains the steps needed to configure each supported platform for benchmarking.

Kotlin/JVM

To run benchmarks in Kotlin/JVM:

  1. Create a JVM target:

    // build.gradle.kts
    kotlin {
        jvm()
    }
  2. Register jvm as a benchmark target:

    // build.gradle.kts
    benchmark {
        targets { 
            register("jvm")
        }
    }
  3. Apply allopen plugin to ensure your benchmark classes and methods are open.

    // build.gradle.kts
    plugins {
        kotlin("plugin.allopen") version "1.9.20"
    }
    
    allOpen {
        annotation("org.openjdk.jmh.annotations.State")
    }
    Explanation

    Assume that you've annotated each of your benchmark classes with @State(Scope.Benchmark):

    // MyBenchmark.kt
    @State(Scope.Benchmark)
    class MyBenchmark {
        // Benchmarking-related methods and variables
        @Benchmark
        fun benchmarkMethod() {
            // benchmarking logic
        }
    }

    In Kotlin, classes are final by default, which means they can't be overridden. This conflicts with the Java Microbenchmark Harness (JMH) operation, which kotlinx-benchmark uses under the hood for running benchmarks on JVM. JMH requires benchmark classes and methods to be open to be able to generate subclasses and conduct the benchmark.

    This is where the allopen plugin comes into play. With the plugin applied, any class annotated with @State is treated as open, which allows JMH to work as intended:

    // build.gradle.kts
    plugins {
        kotlin("plugin.allopen") version "1.9.20"
    }
    
    allOpen {
        annotation("org.openjdk.jmh.annotations.State")
    }

    This configuration ensures that your MyBenchmark class and its benchmarkMethod function are treated as open.

    You can alternatively mark your benchmark classes and methods open manually, but using the allopen plugin enhances code maintainability.

Kotlin/JS

To run benchmarks in Kotlin/JS:

  1. Create a JS target with Node.js execution environment:

    // build.gradle.kts
    kotlin {
        js { 
            nodejs() 
        }
    }
  2. Register js as a benchmark target:

    // build.gradle.kts
    benchmark {
        targets {
            register("js")
        }
    }

Kotlin/Native

To run benchmarks in Kotlin/Native:

  1. Create a Native target:

    // build.gradle.kts
    kotlin {
        linuxX64()
    }
  2. Register linuxX64 as a benchmark target:

    // build.gradle.kts
    benchmark {
        targets {
            register("linuxX64")
        }
    }

It is possible to register multiple native targets. However, benchmarks can be executed only for the host target. This library supports all targets supported by the Kotlin/Native compiler.

Kotlin/Wasm

To run benchmarks in Kotlin/Wasm:

  1. Create a Wasm target with Node.js execution environment:

    // build.gradle.kts
    kotlin {
        wasm { 
            nodejs() 
        }
    }
  2. Register wasm as a benchmark target:

    // build.gradle.kts
    benchmark {
        targets {
            register("wasm")
        }
    }

Note: Kotlin/Wasm is an experimental compilation target for Kotlin. It may be dropped or changed at any time. Refer to Kotlin/Wasm documentation for up-to-date information about the target stability.

Writing Benchmarks

After setting up your project and configuring targets, you can start writing benchmarks. As an example, let's write a simplified benchmark that tests how fast we can add up numbers in an ArrayList:

  1. Create Benchmark Class: Create a class in your source set where you'd like to add the benchmark. Annotate this class with @State(Scope.Benchmark).

    @State(Scope.Benchmark)
    class MyBenchmark {
    
    }
  2. Set Up Variables: Define variables needed for the benchmark.

    private val size = 10
    private val list = ArrayList<Int>()
  3. Initialize Resources: Within the class, you can define any setup or teardown methods using @Setup and @TearDown annotations respectively. These methods will be executed before and after the entire benchmark run.

    @Setup
    fun prepare() {
        for (i in 0..<size) {
            list.add(i)
        }
    }
    
    @TearDown
    fun cleanup() {
        list.clear()
    }
  4. Define Benchmark Methods: Next, create methods that you would like to be benchmarked within this class and annotate them with @Benchmark.

    @Benchmark
    fun benchmarkMethod(): Int {
        return list.sum()
    }

Your final benchmark class will look something like this:

import kotlinx.benchmark.*

@State(Scope.Benchmark)
class MyBenchmark {
    private val size = 10
    private val list = ArrayList<Int>()

    @Setup
    fun prepare() {
        for (i in 0..<size) {
            list.add(i)
        }
    }

    @TearDown
    fun cleanup() {
        list.clear()
    }

    @Benchmark
    fun benchmarkMethod(): Int {
        return list.sum()
    }
}

Note: Benchmark classes located in the common source set will be run in all platforms, while those located in a platform-specific source set will be run only in the corresponding platform.

See writing benchmarks for a complete guide for writing benchmarks.

Running Benchmarks

To run your benchmarks in all registered platforms, run benchmark Gradle task in your project. To run only on a specific platform, run <target-name>Benchmark, e.g., jvmBenchmark.

For more details about the tasks created by the kotlinx-benchmark plugin, refer to this guide.

Benchmark Configuration Profiles

The kotlinx-benchmark library provides the ability to create multiple configuration profiles. The main configuration is already created by the toolkit. Additional profiles can be created as needed in the configurations section of the benchmark block:

// build.gradle.kts
benchmark {
    configurations {
        named("main") {
            warmups = 20
            iterations = 10
            iterationTime = 3
            iterationTimeUnit = "s"
        }
        register("smoke") {
            include("<pattern of fully qualified name>")
            warmups = 5
            iterations = 3
            iterationTime = 500
            iterationTimeUnit = "ms"
        }
    }
}

Refer to our comprehensive guide to learn about configuration options and how they affect benchmark execution.

Separate source set for benchmarks

Often you want to have benchmarks in the same project, but separated from main code, much like tests. Refer to our detailed documentation on configuring your project to set up a separate source set for benchmarks.

Examples

To help you better understand how to use the kotlinx-benchmark library, we've provided an examples subproject. These examples showcase various use cases and offer practical insights into the library's functionality.

Contributing

We welcome contributions to kotlinx-benchmark! If you want to contribute, please refer to our Contribution Guidelines.

More Repositories

1

anko

Pleasant Android application development
Kotlin
15,927
star
2

kotlinx.coroutines

Library support for Kotlin coroutines
Kotlin
12,203
star
3

kotlinx.serialization

Kotlin multiplatform / multi-format serialization
Kotlin
4,951
star
4

dokka

API documentation engine for Kotlin
Kotlin
3,220
star
5

kotlin-examples

Various examples for Kotlin
3,177
star
6

KEEP

Kotlin Evolution and Enhancement Process
Markdown
3,109
star
7

kotlin-koans

Kotlin workshop
Kotlin
2,602
star
8

kotlinx-datetime

KotlinX multiplatform date/time library
Kotlin
2,056
star
9

kmm-production-sample

This is an open-source, mobile, cross-platform application built with Kotlin Multiplatform Mobile. It's a simple RSS reader, and you can download it from the App Store and Google Play. It's been designed to demonstrate how KMM can be used in real production projects.
Kotlin
1,868
star
10

coroutines-examples

Examples for coroutines design in Kotlin
1,465
star
11

kotlindl

High-level Deep Learning Framework written in Kotlin and inspired by Keras
Kotlin
1,373
star
12

kotlin-fullstack-sample

Kotlin Full-stack Application Example
Kotlin
1,218
star
13

kotlinx-kover

Kotlin
1,185
star
14

kotlin-jupyter

Kotlin kernel for Jupyter/IPython
Kotlin
1,037
star
15

kotlinx.collections.immutable

Immutable persistent collections for Kotlin
Kotlin
1,005
star
16

kmm-basic-sample

Example of Kotlin multiplatform project
Kotlin
887
star
17

kotlinx-cli

Pure Kotlin implementation of a generic CLI parser.
Kotlin
886
star
18

kotlinx-io

Kotlin multiplatform I/O library
Kotlin
817
star
19

kotlinx-atomicfu

The idiomatic way to use atomic operations in Kotlin
Kotlin
716
star
20

dataframe

Structured data processing in Kotlin
Kotlin
688
star
21

binary-compatibility-validator

Public API management tool
Kotlin
657
star
22

kotlinconf-spinner

Kotlin
603
star
23

workshop

JetBrains Kotlin Workshop Material
Kotlin
594
star
24

kotlin-interactive-shell

Kotlin Language Interactive Shell
Java
574
star
25

kotlin-frontend-plugin

Gradle Kotlin (http://kotlinlang.org) plugin for frontend development
Kotlin
570
star
26

multik

Kotlin
550
star
27

dukat

Converter of <any kind of declarations> to Kotlin external declarations
Kotlin
535
star
28

kdoctor

Environment analysis tool
Kotlin
509
star
29

kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Kotlin
435
star
30

kotlin-by-example

The sources of Kotlin by Example.
396
star
31

kotlin-wasm-examples

Examples with Kotlin/Wasm
Kotlin
395
star
32

kotlin-spec

Kotlin Language Specification:
Kotlin
358
star
33

kotlin-in-action

Code samples from the "Kotlin in Action" book
Kotlin
343
star
34

ts2kt

ts2kt is officially deprecated, please use https://github.com/Kotlin/dukat instead. // Converter of TypeScript definition files to Kotlin external declarations
Kotlin
320
star
35

kotlin-numpy

Kotlin bindings for NumPy
Kotlin
312
star
36

kotlin-style-guide

Work-in-progress notes for the Kotlin style guide
289
star
37

kotlinx-knit

Kotlin source code documentation management tool
Kotlin
287
star
38

anko-example

A small application built with Anko DSL
Kotlin
285
star
39

full-stack-web-jetbrains-night-sample

Full-stack demo application written with Kotlin MPP
Kotlin
271
star
40

kotlin-script-examples

Examples of Kotlin Scripts and usages of the Kotlin Scripting API
Kotlin
262
star
41

kotlinx-nodejs

Kotlin external declarations for using the Node.js API from Kotlin code targeting JavaScript
Kotlin
212
star
42

kotlin-eclipse

Kotlin Plugin for Eclipse
Kotlin
185
star
43

kotlinx.reflect.lite

Lightweight library allowing to introspect basic stuff about Kotlin symbols
Kotlin
150
star
44

api-guidelines

Best practices to consider when writing an API for your library
142
star
45

kotlin-benchmarks

This is the project to verify and investigate performance issues in Kotlin and standard library.
Kotlin
136
star
46

kotlin-libs-publisher

Gradle plugin for publishing of Kotlin libs
Kotlin
114
star
47

kandy

Kotlin plotting library.
Kotlin
107
star
48

KMP-App-Template

Kotlin Multiplatform app template with shared UI
Kotlin
100
star
49

kotlinx-browser

Kotlin browser API
Kotlin
100
star
50

kotlindl-app-sample

This repo demonstrates how to use KotlinDL for neural network inference on Android devices.
Kotlin
96
star
51

kotlin-koans-edu

Kotlin Koans for Educational Plugin and play.kotl.in
Kotlin
93
star
52

grammar-tools

Tokenization and parsing Kotlin code using the ANTLR Kotlin grammar
Kotlin
83
star
53

kmm-integration-sample

Kotlin
76
star
54

kmm-with-cocoapods-sample

This project represents the case when Cocoapods dependencies are added in Kotlin and there is no existing Xcode project
Kotlin
57
star
55

kotlin-koans-edu-obsolete

Obsolete: check https://github.com/Kotlin/kotlin-koans-edu for the latest version.
Kotlin
55
star
56

kotlin-native-calculator-sample

55
star
57

kotlinx.support

Extension and top-level functions to use JDK7/JDK8 features in Kotlin 1.0
Kotlin
54
star
58

js-externals

External declarations for Kotlin/JS
53
star
59

coroutines-workshop

Materials for a full-day workshop on Kotlin Coroutines
Kotlin
42
star
60

kotlin-playground-wp-plugin

WordPress plugin which allows to embed interactive Kotlin playground to any post via [kotlin] shortcode
PHP
35
star
61

io2019-serverside-demo

E2E Sample
Kotlin
31
star
62

kotlin-jupyter-libraries

Library descriptors for Kotlin kernel for Jupyter
23
star
63

kmm-with-cocoapods-multitarget-xcode-sample

This project is intended to demonstrate how to connect Kotlin library to Xcode project with several targets: iOS, macOS, tvOS, watchOS
Swift
23
star
64

KMP-App-Template-Native

Kotlin Multiplatform app template with native UI
Kotlin
22
star
65

kmp-native-wizard

A mostly-empty template to get started creating a Kotlin/Native project.
Kotlin
22
star
66

xcode-compat

AppCode helper for Kotlin/Native and Xcode
Kotlin
21
star
67

kotlin-in-action-2e

Code samples for the second edition of "Kotlin in Action".
Kotlin
20
star
68

kotlinx.dom

Kotlin
17
star
69

dokka-plugin-template

Dokka plugin quickstart template with pre-configured dependencies and publishing
Kotlin
13
star
70

kotlin-wasm-benchmarks

Kotlin Multiplatform Collection of Benchmarks focused on Kotlin/Wasm performance
Kotlin
12
star
71

kotlinx.team.infra

Kotlin
10
star
72

multiplatform-library-template

Kotlin
10
star
73

kotlin-js-inspection-pack-plugin

Adds useful inspections, intentions, and quick-fixes for working with Kotlin/JS projects.
Kotlin
10
star
74

kotlin-wasm-compose-template

A template repository for Compose Multiplatform with Kotlin/Wasm target
Kotlin
9
star
75

obsolete-kotlin-swing

Experimental library providing some helper functions and extensions for creating Swing user interfaces.
Kotlin
8
star
76

kotlin-in-action-2e-jkid

Sample project accompanying the second edition of "Kotlin in Action". JSON serialization/deserialization library for Kotlin data classes
Kotlin
8
star
77

obsolete-kotlin-jdbc

Experimental library providing some helper functions and extensions for working with JDBC in Kotlin.
Kotlin
7
star
78

kotlin-grammar-gpl2

Kotlin grammar ANTLR sources (under GPLv2)
ANTLR
6
star
79

community-project-gradle-plugin

Kotlin
6
star
80

website-grammar-generator

Kotlin ANTLR grammar converter to XML for the Kotlin website or text file
Kotlin
6
star
81

kotlin-spark-shell

Kotlin Language support for Apache Spark
Kotlin
5
star
82

web-site-samples

Examples repository for kotlinlang.org
4
star
83

kotlin-wasm-browser-template

A template repository for Kotlin/Wasm on browser
HTML
4
star
84

kotlin-cocoapods-spec

Ruby
3
star
85

kotlin-build-report-sample

Kotlin
3
star
86

spec-tests-relinking-recommender

Tool for relinking recommendation of the Kotlin compiler spec tests, that are inconsistent to the latest Kotlin specification
Python
1
star
87

kmm-with-cocoapods-xcode-two-kotlin-libraries-sample

This project is intended to demonstrate the connection of two Kotlin libraries to existing Xcode project through Cocoapods
Kotlin
1
star
88

kotlin-wasm-wasi-template

A template repository for Kotlin/Wasm with WASI
Kotlin
1
star