• Stars
    star
    5,559
  • Rank 6,576 (Top 0.2 %)
  • Language
    Java
  • License
    Other
  • Created over 11 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

Hazelcast

Slack javadoc Docker pulls Quality Gate Status


What is Hazelcast

The world’s leading companies trust Hazelcast to modernize applications and take instant action on data in motion to create new revenue streams, mitigate risk, and operate more efficiently. Businesses use Hazelcast’s unified real-time data platform to process streaming data, enrich it with historical context and take instant action with standard or ML/AI-driven automation - before it is stored in a database or data lake.

Hazelcast is named in the Gartner Market Guide to Event Stream Processing and a leader in the GigaOm Radar Report for Streaming Data Platforms. To join our community of CXOs, architects and developers at brands such as Lowe’s, HSBC, JPMorgan Chase, Volvo, New York Life, and others, visit hazelcast.com.

When to use Hazelcast

Hazelcast provides a platform that can handle multiple types of workloads for building real-time applications.

  • Stateful data processing over streaming data or data at rest
  • Querying streaming and batch data sources directly using SQL
  • Ingesting data through a library of connectors and serving it using low-latency SQL queries
  • Pushing updates to applications on events
  • Low-latency queue-based or pub-sub messaging
  • Fast access to contextual and transactional data via caching patterns such as read/write-through and write-behind
  • Distributed coordination for microservices
  • Replicating data from one region to another or between data centers in the same region

Key Features

  • Stateful and fault-tolerant data processing and querying over data streams and data at rest using SQL or dataflow API
  • A comprehensive library of connectors such as Kafka, Hadoop, S3, RDBMS, JMS and many more
  • Distributed messaging using pub-sub and queues
  • Distributed, partitioned, queryable key-value store with event listeners, which can also be used to store contextual data for enriching event streams with low latency
  • A production-ready Raft-implementation which allows lineralizable (CP) concurrency primitives such as distributed locks.
  • Tight integration for deploying machine learning models with Python to a data processing pipeline
  • Cloud-native, run everywhere architecture
  • Zero-downtime operations with rolling upgrades
  • At-least-once and exactly-once processing guarantees for stream processing pipelines
  • Data replication between data centers and geographic regions using WAN
  • Microsecond performance for key-value point lookups and pub-sub
  • Unique data processing architecture results in 99.99% latency of under 10ms for streaming queries with millions of events per second.
  • Client libraries in Java, Python, Node.js, .NET, C++ and Go

Operational Data Store

Hazelcast provides distributed in-memory data structures which are partitioned, replicated and queryable. One of the main use cases for Hazelcast is for storing a working set of data for fast querying and access.

The main data structure underlying Hazelcast, called IMap, is a key-value store which has a rich set of features, including:

Hazelcast stores data in partitions, which are distributed to all the nodes. You can increase the storage capacity by adding additional nodes, and if one of the nodes go down, the data is restored automatically from the backup replicas.

You can interact with maps using SQL or a programming language client of your choice. You can create and interact with a map as follows:

CREATE MAPPING myMap (name varchar EXTERNAL NAME "__key", age INT EXTERNAL NAME "this") 
TYPE IMap
OPTIONS ('keyFormat'='varchar','valueFormat'='int');
INSERT INTO myMap VALUES('Jake', 29);
SELECT * FROM myMap;

The same can be done programmatically as follows using one of the supported programming languages. Here are some exmaples in Java and Python:

var hz = HazelcastClient.newHazelcastClient();
IMap<String, Integer> map = hz.getMap("myMap");
map.set(Alice, 25);
import hazelcast

client = hazelcast.HazelcastClient()
my_map = client.get_map("myMap")
age = my_map.get("Alice").result()

Other programming languages supported are C#, C++, Node.js and Go.

Alternatively, you can ingest data directly from the many sources supported using SQL:

CREATE MAPPING csv_ages (name VARCHAR, age INT)
TYPE File
OPTIONS ('format'='csv',
    'path'='/data', 'glob'='data.csv');
SINK INTO myMap
SELECT name, age FROM csv_ages;

Hazelcast also provides additional data structures such as ReplicatedMap, Set, MultiMap and List. For a full list, refer to the distributed data structures section of the docs.

Stateful Data Processing

Hazelcast has a built-in data processing engine called Jet. Jet can be used to build both streaming and batch data pipelines that are elastic. You can use it to process large volumes of real-time events or huge batches of static datasets. To give a sense of scale, a single node of Hazelcast has been proven to aggregate 10 million events per second with latency under 10 milliseconds. A cluster of Hazelcast nodes can process billion events per second.

An application which aggregates millions of sensor readings per second with 10-millisecond resolution from Kafka looks like the following:

var hz = Hazelcast.bootstrappedInstance();

var p = Pipeline.create();

p.readFrom(KafkaSources.<String, Reading>kafka(kafkaProperties, "sensors"))
 .withTimestamps(event -> event.getValue().timestamp(), 10) // use event timestamp, allowed lag in ms
 .groupingKey(reading -> reading.sensorId())
 .window(sliding(1_000, 10)) // sliding window of 1s by 10ms
 .aggregate(averagingDouble(reading -> reading.temperature()))
 .writeTo(Sinks.logger());

hz.getJet().newJob(p).join();

Use the following command to deploy the application to the server:

bin/hazelcast submit analyze-sensors.jar

Jet supports advanced streaming features such as exactly-once processing and watermarks.

Data Processing using SQL

Jet also powers the SQL engine in Hazelcast which can execute both streaming and batch queries. Internally, all SQL queries are converted to Jet jobs.

CREATE MAPPING trades (
    id BIGINT,
    ticker VARCHAR,
    price DECIMAL,
    amount BIGINT)
TYPE Kafka
OPTIONS (
    'valueFormat' = 'json',
    'bootstrap.servers' = 'kafka:9092'
);
SELECT ticker, ROUND(price * 100) AS price_cents, amount
  FROM trades
  WHERE price * amount > 100;
+------------+----------------------+-------------------+
|ticker      |           price_cents|             amount|
+------------+----------------------+-------------------+
|EFGH        |                  1400|                 20|

Messaging

Hazelcast provides lightweight options for adding messaging to your application. The two main constructs for messaging are topics and queues.

Topics

Topics provide a publish-subscribe pattern where each message is fanned out to multiple subscribers. See the examples below in Java and Python:

var hz = Hazelcast.bootstrappedInstance();
ITopic<String> topic = hz.getTopic("my_topic");
topic.addMessageListener(msg -> System.out.println(msg));
topic.publish("message");
topic = client.get_topic("my_topic")

def handle_message(msg):
    print("Received message %s"  % msg.message)
topic.add_listener(on_message=handle_message)
topic.publish("my-message")

For examples in other languages, please refer to the docs.

Queues

Queues provide FIFO-semantics and you can add items from one client and remove from another. See the examples below in Java and Python:

var client = Hazelcast.newHazelcastClient();
IQueue<String> queue = client.getQueue("my_queue");
queue.put("new-item")
import hazelcast

client = hazelcast.HazelcastClient()
q = client.get_queue("my_queue")
my_item = q.take().result()
print("Received item %s" % my_item)

For examples in other languages, please refer to the docs.

Get Started

Follow the Getting Started Guide to install and start using Hazelcast.

Documentation

Read the documentation for in-depth details about how to install Hazelcast and an overview of the features.

Get Help

You can use the following channels for getting help with Hazelcast:

How to Contribute

Thanks for your interest in contributing! The easiest way is to just send a pull request. Have a look at the issues marked as good first issue for some guidance.

Building From Source

Building Hazelcast requires at minimum JDK 11. Pull the latest source from the repository and use Maven install (or package) to build:

$ git pull origin master
$ ./mvnw clean package -DskipTests

It is recommended to use the included Maven wrapper script. It is also possible to use local Maven distribution with the same version that is used in the Maven wrapper script.

Additionally, there is a quick build activated by setting the -Dquick system property that skips tests, checkstyle validation, javadoc and source plugins and does not build extensions and distribution modules.

Testing

Take into account that the default build executes thousands of tests which may take a considerable amount of time. Hazelcast has 3 testing profiles:

  • Default: Type ./mvnw test to run quick/integration tests (those can be run in parallel without using network by using -P parallelTest profile).
  • Slow Tests: Type ./mvnw test -P nightly-build to run tests that are either slow or cannot be run in parallel.
  • All Tests: Type ./mvnw test -P all-tests to run all tests serially using network.

Some tests require Docker to run. Set -Dhazelcast.disable.docker.tests system property to ignore them.

When developing a PR it is sufficient to run your new tests and some related subset of tests locally. Our PR builder will take care of running the full test suite.

Trigger Phrases in the Pull Request Conversation

When you create a pull request (PR), it must pass a build-and-test procedure. Maintainers will be notified about your PR, and they can trigger the build using special comments. These are the phrases you may see used in the comments on your PR:

  • run-lab-run - run the default PR builder
  • run-lts-compilers - compiles the sources with JDK 11 and JDK 17 (without running tests)
  • run-ee-compile - compile hazelcast-enterprise with this PR
  • run-ee-tests - run tests from hazelcast-enterprise with this PR
  • run-windows - run the tests on a Windows machine (HighFive is not supported here)
  • run-with-jdk17 - run the tests with JDK 17
  • run-with-ibm-jdk-8 - run the tests with IBM JDK 8
  • run-cdc-debezium-tests - run all tests in the extensions/cdc-debezium module
  • run-cdc-mysql-tests - run all tests in the extensions/cdc-mysql module
  • run-cdc-postgres-tests - run all tests in the extensions/cdc-postgres module
  • run-mongodb-tests - run all tests in the extensions/mongodb module
  • run-s3-tests - run all tests in the extensions/s3 module
  • run-nightly-tests - run nightly (slow) tests. WARNING: Use with care as this is a resource consuming task.
  • run-ee-nightly-tests - run nightly (slow) tests from hazelcast-enterprise. WARNING: Use with care as this is a resource consuming task.
  • run-sql-only - run default tests in hazelcast-sql, hazelcast-distribution, and extensions/mapstore modules
  • run-docs-only - do not run any tests, check that only files with .md, .adoc or .txt suffix are added in the PR
  • run-sonar - run SonarCloud analysis
  • run-arm64 - run the tests on arm64 machine

Where not indicated, the builds run on a Linux machine with Oracle JDK 11.

Creating PRs for Hazelcast SQL

When creating a PR with changes located in the hazelcast-sql module and nowhere else, you can label your PR with SQL-only. This will change the standard PR builder to one that will only run tests related to SQL (see run-sql-only above), which will significantly shorten the build time vs. the default PR builder. NOTE: this job will fail if you've made changes anywhere other than hazelcast-sql.

Creating PRs which contain only documentation

When creating a PR which changes only documentation (files with suffix .md or .adoc) it makes no sense to run tests. For that case the label docs-only can be used. The job will fail in case you've made other changes than in .md, .adoc or .txt files.

License

Source code in this repository is covered by one of two licenses:

  1. Apache License 2.0
  2. Hazelcast Community License

The default license throughout the repository is Apache License 2.0 unless the header specifies another license.

Acknowledgments

Thanks to YourKit for supporting open source software by providing us a free license for their Java profiler.

We owe (the good parts of) our CLI tool's user experience to picocli.

Copyright

Copyright (c) 2008-2023, Hazelcast, Inc. All Rights Reserved.

Visit www.hazelcast.com for more info.

More Repositories

1

hazelcast-jet

Distributed Stream and Batch Processing
Java
1,079
star
2

hazelcast-code-samples

Hazelcast Code Samples
Java
536
star
3

hazelcast-go-client

Hazelcast Go Client
Go
178
star
4

hazelcast-kubernetes

Kubernetes Discovery for Hazelcast
Java
173
star
5

hazelcast-nodejs-client

Hazelcast Node.js Client
TypeScript
149
star
6

hazelcast-jet-demos

Demonstration applications using Hazelcast Jet
Java
136
star
7

hazelcast-python-client

Hazelcast Python Client
Python
112
star
8

hazelcast-csharp-client

Hazelcast .NET Client
C#
100
star
9

hazelcast-docker

This repository contains docker image for Hazelcast open-source in-memory data-grid.
Dockerfile
89
star
10

hazelcast-jet-code-samples

83
star
11

hazelcast-simulator

A tool for stress testing Hazelcast
Java
83
star
12

spring-data-hazelcast

Hazelcast Spring Data integration Project http://projects.spring.io/spring-data/
Java
82
star
13

hazelcast-cpp-client

Hazelcast IMDG C++ Client
C++
78
star
14

quarkus-hazelcast-client

Quarkus Hazelcast Client Extension
Java
43
star
15

hazelcast-aws

AWS EC2 discovery plugin for hazelcast
Java
40
star
16

hazelcast-hibernate

A distributed second-level cache for Hibernate
Java
38
star
17

charts

Hazelcast Official Helm Chart Repository
Smarty
35
star
18

hazelcast-platform-demos

hazelcast-platform-demos
Java
34
star
19

hazelcast-tomcat-sessionmanager

Tomcat Based Web Session Replication
Java
33
star
20

hazelcast-eureka

Hazelcast Discovery SPI Plugin for Netflix' Eureka Service Discovery V1
Java
30
star
21

hazelcast-scala

Scala language support for Hazelcast
Scala
26
star
22

hazelcast-commandline-client

Home of Hazelcast CLC
Go
25
star
23

hazelcast-spark

Spark Connector for Hazelcast
Scala
23
star
24

hazelcast-zookeeper

Hazelcast Discovery Plugin for Apache ZooKeeper
Java
23
star
25

training

Java
22
star
26

hazelcast-jet-contrib

Extension modules for Hazelcast Jet
Java
21
star
27

hazelcast-platform-operator

Easily deploy Hazelcast clusters and Management Center into Kubernetes/OpenShift environments and manage their lifecycles.
Go
21
star
28

hazelcast-reference-manual

Hazelcast Reference Manual
Java
20
star
29

hazelcast-azure

Azure discovery plugin for hazelcast
Java
18
star
30

hazelcast-hibernate-3-and-4

distributed second level cache for your Hibernate
Java
18
star
31

hazelcast-platform-training

Java
18
star
32

hive

Design system built with A11Y in mind
TypeScript
18
star
33

trading-demo

A trading demo application
Java
16
star
34

management-center-docker

This repository contains Docker image for Hazelcast Management Center.
Dockerfile
15
star
35

jet-train

Kotlin
15
star
36

hazelcast-wm

Hazelcast filter-based Web Session Manager
Java
13
star
37

hazelcast-mesos

Hazelcast on Mesos
Java
12
star
38

hazelcast-client-protocol

Hazelcast Open Binary Client Protocol
Jinja
11
star
39

big-data-benchmark

Java
11
star
40

hazelcast-operator

Shell
10
star
41

hazelcast-openshift

Hazelcast Openshift
Dockerfile
10
star
42

docker-grafana-graphite

Docker image with StatsD, Graphite and Grafana 2
Dockerfile
10
star
43

hazelcast-packaging

Shell
9
star
44

homebrew-hz

Homebrew Repository for Hazelcast Command Line
Ruby
8
star
45

hazelcast-jetty-sessionmanager

Jetty Based Web Session Replication
Java
7
star
46

hazelcast-remote-controller

Java
7
star
47

hz-docs

Source content for the Hazelcast Platform documentation
JavaScript
7
star
48

hazelcast-hibernate4

!!!This repo is outdated. Kept alive only for backward compatibility reasons.!!! Beta implementation of hazelcast-hibernate module for Hibernate 4.0.
Java
7
star
49

imdg-docs

Source content for the Hazelcast IMDG documentation
JavaScript
6
star
50

betleopard

Java
6
star
51

hazelcast-jmh

JMH Benchmarks for Hazelcast
Java
6
star
52

hazelcast-gcp

Google Cloud Platform
Java
5
star
53

hazelcast-docs

Hazelcast documentation playbook
JavaScript
5
star
54

hazelcast-jet-beam-runner

Hazelcast Jet Runner for Apache Beam
Java
5
star
55

fraud-detection-onnx

Hazelcast and Onnx for Low Latency Fraud Detection
Java
4
star
56

hazelcast-demos

Java
4
star
57

hazelcast-docs-ui

User interface styles for the Hazelcast documentation playbook.
Handlebars
4
star
58

hazelcast-qa

Collection of Hazelcast QA tools.
Java
4
star
59

hazelcast-jet-reference-manual

Reference Manual for Hazelcast Jet
4
star
60

hazelcast-jdbc

Hazelcast JDBC Driver allows Java applications to connect to Hazelcast using the standard JDBC API
Java
4
star
61

management-center-docs

Source content for the Hazelcast Management Center documentation
JavaScript
4
star
62

hazelcast-docker-samples

Java
4
star
63

hazelcast-platform-operator-agent

Go
3
star
64

hazelcast-ra

Hazelcast JCA Resource Adapter
Java
3
star
65

hazelcast-gradle-starter

Java
3
star
66

hazelcast-dissector-for-wireshark

Hazelcast 4+ member protocol dissector
C
3
star
67

performancetop5

The benchmarks for the Hazelcast Performance Top 5 blog series
Java
3
star
68

rel-scripts

3
star
69

client-compatibility-suites

JavaScript
3
star
70

training-courses

Java
3
star
71

hazelcast-python-client-kerberos

Kerberos authentication support for Hazelcast Python Client
Python
3
star
72

hazelcast-hadoop

Hadoop Integration for Hazelcast IMDG
Java
2
star
73

hazelcast-cloud-maven-plugin

Maven Plugin for Hazelcast Cloud
Java
2
star
74

hazelcast-jet-ansible-tests

Set of tests for Hazelcast Jet Soak Testing Environment.
Java
2
star
75

hazelcast-cloud-cli

CLI for Hazelcast Cloud
Go
2
star
76

hazelcast-dynacache

This repository contains Hazelcast DynaCache feature for Liberty Profile.
Java
2
star
77

hazelcast-cloud-go-sample-client

Sample Go Client For Hazelcast Cloud Community
Go
2
star
78

hazelcast-cloud-code-samples

Hazelcast Cloud Code Samples
Java
2
star
79

hazelcast-jet-docker

This repository contains Docker image for Hazelcast Jet open-source distributed computing platform built for high-performance stream processing and fast batch processing.
Dockerfile
2
star
80

cloud-docs

Source content for the Hazelcast Cloud documentation
JavaScript
1
star
81

fraud-detection-python

Python
1
star
82

clc-kafka-jet-demo

Kotlin
1
star
83

management-center-openshift

Dockerfile
1
star
84

hazelcast-grails

hazelcast grails plugin
Groovy
1
star
85

hazelcast-cloud-python-sample-client

Sample Python Client For Hazelcast Cloud Community
Python
1
star
86

hazelcast-jet-management-center-docker

This repository contains Docker image for Hazelcast Jet Management Center
Dockerfile
1
star
87

hazelcast-cloud-community

Hazelcast Cloud Community Feature,Feedback,Issue Repository
1
star
88

hazelcast-platform-operator-docs

Source content for the Kubernetes Operator of Hazelcast Platform (Enterprise)
JavaScript
1
star
89

hazelcast-jclouds

jclouds discovery plugin for hazelcast
Java
1
star
90

hazelcast-diagnostics

Java
1
star
91

sample-worker

Sample worker
Java
1
star
92

hazelcast-tpm

Repository of Hazelcast Technical Program Management Team for storing scripts & tools that helps gathering & processing information.
Python
1
star
93

cfsummit17-wednesday

Hazelcast Technology for PCF Demo presented at Cloud Foundry Summit Silicon Valley 2017
Java
1
star
94

java-client-reference-manual

Java
1
star