• Stars
    star
    528
  • Rank 83,941 (Top 2 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 9 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Low level access to native memory, JVM and OS.

Chronicle-Core: An Advanced Low-Level Library

About Chronicle-Core

Chronicle-Core is an advanced low-level library that equips developers with powerful tools to interact with the operating system, manage memory, handle resources, and more. However, it should be used with caution due to its low-level operations which, if misused, can lead to complex issues.

Here is a summary of the libraryโ€™s key features:

Operating System Calls: Chronicle-Core provides access to various system calls such as retrieving the process ID, checking the operating system, and obtaining the hostname, among others.

int processId = OS.getProcessId();
boolean isWindows = OS.isWindows();
String hostname = OS.getHostName();

See the section on OS Calls

JVM Access Methods: To access platform specific features of the JVM.

See the section on JVM Access Methods

Memory Mapped Files: The library offers an interface for managing memory mapped files, which is useful for high-performance I/O operations.

FileChannel fc = new CleaningRandomAccessFile(fileName, "rw").getChannel();
long address = OS.map(fc, MapMode.READ_WRITE, 0, 64 << 10);
OS.memory().writeLong(1024L, 0x1234567890ABCDEFL);
OS.unmap(address, 64 << 10);

Deterministic Resource Management: Chronicle-Core features components that can be closed or reference-counted, and released deterministically without waiting for garbage collection.

Closeable Resources: Chronicle-Core provides an interface for managing closeable resources, which are open when created and canโ€™t be used once closed. This helps in preventing resource leaks.

public class AbstractCloseableTest {
    @Test
    public void close() {
        MyCloseable mc = new MyCloseable();
        assertFalse(mc.isClosed());
        assertEquals(0, mc.performClose);
        mc.throwExceptionIfClosed();
        mc.close();
        assertTrue(mc.isClosed());
        assertEquals(1, mc.performClose);
    }
}

Resource Reference Counting: The library enables the use of reference counting for deterministic resource release. A reference-counted resource can add reservations until itโ€™s closed.

public class AbstractReferenceCountedTest {
    @Test
    public void reserve() {
        assertTrue(Jvm.isResourceTracing());
        MyReferenceCounted rc = new MyReferenceCounted();
        assertEquals(1, rc.refCount());
        ReferenceOwner a = ReferenceOwner.temporary("a");
        rc.reserve(a);
        assertEquals(2, rc.refCount());
        //...
    }
}

See the section on Resource Reference Counting

This library also wraps up low level access to

System properties from file

Chronicle-Coreโ€™s Jvm class automatically loads system properties from a system.properties file if found in the current directory or parent directory. This feature aids in streamlining your command line. You can specify a different properties file with the -Dsystem.properties=my.properties command.

static {
    Jvm.init();
}

The choice of file to load can be overridden on the command line with -Dsystem.properties=my.properties

In Jvm.java it can be seen how to guarantee that JVM class is initialized before the system property is read. For example with Jvm.getInteger or Jvm.getLong.

A number of relevant system properties are listed in systemProperties.adoc.

Note
Command line-specified system properties override those in the system.properties file.

Chronicle-Core Initialization

Chronicle-Core offers an initialization class, ChronicleInit, that enables developers to run their own code at startup. This code can be executed before and/or after the execution of Chronicleโ€™s static initializers, which perform tasks such as system property loading.

ChronicleInit allows the developer to hook in their own code to be run at startup before and/or after the Chronicle static initialisers are run. Chronicle static initialisers perform tasks such as loading system properties, so it is possible, for example, to override system properties using ChronicleInit. To this end, ChronicleInit introduces the following system properties:

  1. "chronicle.init.runnable"

    This system property specifies a fully qualified class name that will be run before any system property is read by Chronicle code, allowing the class to set them to the desired values. The class should contain an empty static init() method that is called to trigger class load.

  2. "chronicle.postinit.runnable"

    This system property specifies a fully qualified class name that will run only once after the Jvm initialisation static class. The class should contain an empty static postInit() method that is called to trigger class load.

The alternative way to using the above system properties is to implement the ChronicleInitRunnable interface whose implementing classes may be listed in the META-INF/services/net.openhft.chronicle.core.ChronicleInitRunnable file in any JAR in classpath to be discovered via ServiceLoader JVM facility. It can provide both init and post-init functionalities by implementing the ChronicleInitRunnableRunnable.run() and ChronicleInitRunnable.postInit() methods.

Off Heap Memory Access

This allows you to access native memory using primitives and some thread safe operations.

Memory memory = OS.memory();
long address = memory.allocate(1024);
try {
    memory.writeInt(address, 1);
    assert memory.readInt(address) == 1;
    final boolean swapped = memory.compareAndSwapInt(address, 1, 2);
    assert swapped;
    assert memory.readInt(address) == 2;
} finally {
    memory.freeMemory(address, 1024);
}

JVM Access Methods

Check the JVM is running in debug mode

if (Jvm.isDebug()) {
   // running in debug.

Rethrow a checked exception as an unchecked one.

try {
    // IO operation
} catch (IOException ioe) {
    throw Jvm.rethrow(ioe);
}

Get a Field for a Class by name

Field theUnsafe = Jvm.getField(Unsafe.class, "theUnsafe");
Unsafe unsafe = (Unsafe) theUnsafe.get(null);

OS Calls

Access to system calls

int processId = OS.getProcessId();
int maxProcessId = OS.getMaxProcessId();
int pageSize = OS.getPageSize();
boolean isWindows = OS.isWindows();
boolean is64bit = OS.is64Bit();
String hostname = OS.getHostName();
String username = OS.getUserName();
String targetDir = OS.getTarget(); // where is the target directory during builds.

Memory mapped files

FileChannel fc = new CleaningRandomAccessFile(fileName, "rw").getChannel();
// map in 64 KiB
long address = OS.map(fc, MapMode.READ_WRITE, 0, 64 << 10);
// use address
OS.memory().writeLong(1024L, 0x1234567890ABCDEFL);
// unmap memory region
OS.unmap(address, 64 << 10);

Deterministic Resource Management

Component which are closeable or reference counted can be released deterministically without waiting for a GC.

Closeable Resources

A Closeable resources has a simple lifecycle. It is open when created, and cannot be used once closed.

public class AbstractCloseableTest {

    @Test
    public void close() {
        MyCloseable mc = new MyCloseable();
        assertFalse(mc.isClosed());
        assertEquals(0, mc.performClose);

        mc.throwExceptionIfClosed();

        mc.close();
        assertTrue(mc.isClosed());
        assertEquals(1, mc.performClose);

        mc.close();
        assertTrue(mc.isClosed());
        assertEquals(1, mc.performClose);
    }

    @Test(expected = IllegalStateException.class)
    public void throwExceptionIfClosed() {
        MyCloseable mc = new MyCloseable();
        mc.close();
        mc.throwExceptionIfClosed();

 }

    @Test
    public void warnAndCloseIfNotClosed() {
        Map<ExceptionKey, Integer> map = Jvm.recordExceptions();
        MyCloseable mc = new MyCloseable();
        mc.warnAndCloseIfNotClosed();
        Jvm.resetExceptionHandlers();
        assertEquals("Discarded without closing\n" +
                        "java.lang.IllegalStateException: net.openhft.chronicle.core.StackTrace: Created Here",
                map.keySet().stream()
                        .map(e -> e.message + "\n" + e.throwable)
                        .collect(Collectors.joining(", ")));
    }

    static class MyCloseable extends AbstractCloseable {
        int performClose;

        @Override
        protected void performClose() {
            performClose++;
        }
    }
}

Resource Reference Counting

Use reference counting to deterministically release resources.

A reference counted resource can add reservations until closed.

public class AbstractReferenceCountedTest {

    @Test
    public void reserve() {
        assertTrue(Jvm.isResourceTracing());
        MyReferenceCounted rc = new MyReferenceCounted();
        assertEquals(1, rc.refCount());

        ReferenceOwner a = ReferenceOwner.temporary("a");
        rc.reserve(a);
        assertEquals(2, rc.refCount());

        ReferenceOwner b = ReferenceOwner.temporary("b");
        rc.reserve(b);
        assertEquals(3, rc.refCount());

        try {
            rc.reserve(a);
            fail();
        } catch (IllegalStateException ignored) {
        }
        assertEquals(3, rc.refCount());

        rc.release(b);
        assertEquals(2, rc.refCount());

        rc.release(a);
        assertEquals(1, rc.refCount());
        assertEquals(0, rc.performRelease);

        rc.releaseLast();
        assertEquals(0, rc.refCount());
        assertEquals(1, rc.performRelease);
    }

    @Test
    public void reserveWhenClosed() {
        MyReferenceCounted rc = new MyReferenceCounted();
        assertEquals(1, rc.refCount());

        ReferenceOwner a = ReferenceOwner.temporary("a");
        rc.reserve(a);
        assertEquals(2, rc.refCount());

        assertFalse(rc.isClosed());

        rc.closeable.close();

        assertEquals(2, rc.refCount());
        assertTrue(rc.isClosed());

        ReferenceOwner b = ReferenceOwner.temporary("b");
        try {
            rc.reserve(b);
            fail();
        } catch (IllegalStateException ignored) {
        }
        assertEquals(2, rc.refCount());

        assertFalse(rc.tryReserve(b));
        assertEquals(2, rc.refCount());

        rc.release(a);
        assertEquals(1, rc.refCount());
        assertEquals(0, rc.performRelease);

        rc.throwExceptionIfReleased();

        rc.releaseLast();
        assertEquals(0, rc.refCount());
        assertEquals(1, rc.performRelease);

        rc.throwExceptionBadResourceOwner();
        try {
            rc.throwExceptionIfClosed();

 fail();
        } catch (IllegalStateException ignored) {

        }
        try {
            rc.throwExceptionIfReleased();
            fail();
        } catch (IllegalStateException ignored) {

        }
    }

    @Test
    public void throwExceptionBadResourceOwner() {
        MyReferenceCounted rc = new MyReferenceCounted();
        MyReferenceCounted rc2 = new MyReferenceCounted();
        rc.reserve(rc2);
        rc.throwExceptionBadResourceOwner();

        rc2.closeable.close();
        try {
            rc.throwExceptionBadResourceOwner();
            fail();
        } catch (IllegalStateException ignored) {
        }
        rc.release(rc2);
        rc.releaseLast();
    }

    @Test
    public void throwExceptionIfClosed() {
        MyReferenceCounted rc = new MyReferenceCounted();
        rc.throwExceptionIfClosed();

        rc.closeable.close();
        try {
            rc.throwExceptionIfClosed();

           fail();
        } catch (IllegalStateException ignored) {

        }
    }

    static class MyReferenceCounted extends AbstractReferenceCounted {
        final AbstractCloseable closeable;
        int performRelease;

        public MyReferenceCounted() {
            this(new AbstractCloseableTest.MyCloseable());
        }

        public MyReferenceCounted(AbstractCloseable abstractCloseable) {
            super(abstractCloseable);
            closeable = abstractCloseable;
        }

        @Override
        protected void performRelease() {
            performRelease++;
        }
    }
}
MappedFile mf = MappedFile.mappedFile(tmp, chunkSize, 0);
MappedBytesStore bs = mf.acquireByteStore(chunkSize + (1 << 10));

assertEquals(2, mf.refCount());
assertEquals(3, bs.refCount());
assertEquals("refCount: 2, 0, 3", mf.referenceCounts());

mf.close();
assertEquals(2, bs.refCount());
assertEquals("refCount: 1, 0, 2", mf.referenceCounts());
bs2.releaseLast();
assertEquals(1, mf.refCount());
assertEquals(1, bs.refCount());
bs.releaseLast();
assertEquals(0, bs.refCount());
assertEquals(0, mf.refCount());
assertEquals("refCount: 0, 0, 0", mf.referenceCounts());

Releasing Resources

Releasing resources can be managed by starting the BACKGROUND_RESOURCE_RELEASER thread or alternatively it can be managed in a user defined thread. To start the BACKGROUND_RESOURCE_RELEASER thread, both system properties background.releaser and background.releaser.thread should be set to true. In this condition, the thread starts as a daemon thread and invokes BackgroundResourceReleaser.runReleaseResources().

If only background.releaser.thread is set to false, resources will still be queued for releasing, but they need to be released explicitly by calling BackgroundResourceReleaser.releasePendingResources().

If background.releaser is set to false regardless of background.releaser.thread, resources are not queued for release and release will be done synchronously (by calling the relevant close() function).

Calling BackgroundResourceReleaser.stop() releases pending resources and then stops the BACKGROUND_RESOURCE_RELEASER thread. To make sure the shutdown hook does not prevent classes from unloading, deregister the shutdown hook by calling PriorityHook.clear().

Table 1. Resource Release Configurations
background.releaser.thread background.releaser Release Behaviour

true

true

Resources are queued and then released in the BACKGROUND_RESOURCE_RELEASER thread.

false

true

Rresources are queued but should be released in a user thread by calling BackgroundResourceReleaser.releasePendingResources().

X

false

Resources are not queued and are released synchronously.

Object Pooling

Chronicle-Core provides object pooling for strings and enums, allowing you to convert a CharSequence into a String of a specific Enum type efficiently.

Bytes<?> b = Bytes.from("Hello World");
b.readSkip(6);

StringInterner si = new StringInterner(128);
String s = si.intern(b);
String s2 = si.intern(b);
assertEquals("World", s);
assertSame(s, s2);

Class Local Caching

Add caching of a data structure for each class using a lambda

public static final ClassLocal<EnumInterner> ENUM_INTERNER =
        ClassLocal.withInitial(c -> new EnumInterner<>(c));

E enumValue = ENUM_INTERNER.get(enumClass).intern(stringBuilder);

Maths Functions

Maths functions to support rounds

double a = 0.1;
double b = 0.3;
double c= Maths.round2(b - a); // 0.2 rounded to 2 decimal places

Checking type conversions

int i = Maths.toInt32(longValue);

Serializable Lambdas

There is a number of FunctionalInterfaces you can utilise as method arguments. This allows implicitly making a lambda Serializable.

// in KeyedVisitable
default <R> R applyToKey(K key, @NotNull SerializableFunction<E, R> function) {

// in code

String fullename = map.applyToKey("u:123223", u -> u.getFullName());

Histogram

A high dynamic range histogram with tunable accuracy.

Histogram h = new Histogram(32, 4);
long start = instance.ticks(), prev = start;
for (int i = 0; i <= 1000_000_000; i++) {
    long now = instance.ticks();
    long time = now - prev;
    h.sample(time);
    prev = now;
}
System.out.println(h.toLongMicrosFormat(instance::toMicros));

JLBH

JLBH has moved home and now lives in its own project, see JLBH.

Loop Block Monitor tool

The tool to summarise the thread stack traces is here.

net.openhft.chronicle.core.threads.MonitorProfileAnalyserMain

More Repositories

1

Chronicle-Queue

Micro second messaging that stores everything to disk
Java
3,130
star
2

Chronicle-Map

Replicate your Key Value Store across your network, with consistency, persistance and performance.
Java
2,669
star
3

Java-Thread-Affinity

Bind a java thread to a given core
Java
1,724
star
4

Zero-Allocation-Hashing

Zero-allocation hashing for Java
Java
762
star
5

Java-Runtime-Compiler

Java Runtime Compiler
Java
617
star
6

OpenHFT

Parent module to include active modules
Shell
609
star
7

Chronicle-Wire

A Low Garbage Java Serialisation Library that supports multiple formats
Java
464
star
8

Chronicle-Bytes

Chronicle Bytes has a similar purpose to Java NIO's ByteBuffer with many extensions
Java
382
star
9

Chronicle-Engine

A high performance, low latency, reactive processing framework
338
star
10

Java-Lang

Java Language support
Java
286
star
11

Chronicle-Network

A High Performance Network ( TCP/IP ) Library
Java
243
star
12

Chronicle-Logger

A sub microsecond java logger, supporting standard logging APIs such as Slf & Log4J
Java
220
star
13

Chronicle-Threads

Java
167
star
14

Chronicle-Values

Java
102
star
15

Chronicle-Algorithms

Java
77
star
16

JLBH

JLBH
Java
68
star
17

Chronicle-Queue-Demo

Sample programs for Chronicle Queue
Java
65
star
18

Chronicle-Accelerate

HFT meets Blockchain in Java platform
Java
59
star
19

Chronicle-TimeSeries

Multi-Threaded Time Series library
Java
59
star
20

Chronicle-Decentred

Framework for building Secure Scalable Microservices on Distributed Ledger Technology
Java
50
star
21

Chronicle-Salt

Chronicle wrapper for the NaCl library
Java
30
star
22

Chronicle-Test-Framework

Java
24
star
23

Chronicle-Ticker

A time ticker as a light weight clock
Java
23
star
24

RFC

RFC's used by OpenHFT
21
star
25

Chronicle-Websocket-Jetty

Provide Websocket access via Jetty
Java
17
star
26

Puzzles

OpenHFT Puzzles
Java
15
star
27

jvm-micro-benchmarks

Microbenchmarks for JVM code.
Java
14
star
28

Spoon

Java
11
star
29

Chronicle-Engine-GUI

CSS
11
star
30

Exception-Handler

Open a browser with a message about an Exception.
Java
10
star
31

Stage-Compiler

Java
8
star
32

Binary-Compatibility-Enforcer-Plugin

Wraps the Java API ComplianceChecker into a maven plugin
Java
7
star
33

Chronicle-Common

Java
6
star
34

Chronicle-Coder

Encode and decode data as symbols and readable words
Java
5
star
35

Posix

Java
5
star
36

Microservice-Benchmark

Open Microservices Benchmark
Java
5
star
37

Chronicle-Analytics

Support for Analytics
Java
4
star