This repo is no longer actively maintained. While it should continue to work and there are no major known bugs, we will not be improving Heroic or releasing new versions.
A scalable time series database based on Bigtable, Cassandra, and Elasticsearch. Go to https://spotify.github.io/heroic/ for documentation.
This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.
Docker images are available on Docker Hub.
$ docker run -p 8080:8080 -p 9091:9091 spotify/heroic
Heroic will now be reachable at http://localhost:8080/status.
In production it's advised to use a tagged version.
For help on how to write a configuration file, see the Configuration Section of the official documentation.
Heroic has been tested with the following services:
- Cassandra (
2.1.x
,3.5
) when using metric/datastax. - Cloud Bigtable when using metric/bigtable.
- Elasticsearch (
7.x
) when using metadata/elasticsearch or suggest/elasticsearch. - Kafka (
0.8.x
) when using consumer/kafka.
In order to compile Heroic, you'll need:
- A Java 11 JDK
- Maven 3
- Gradle
The project is built using Gradle:
# full build, runs all tests and builds the shaded jar
./gradlew build
# only compile
./gradlew assemble
# build a single module
./gradlew heroic-metric-bigtable:build
The heroic-dist
module can be used to produce a shaded jar that contains all required dependencies:
./gradlew heroic-dist:shadowJar
After building, the entry point of the service is
com.spotify.heroic.HeroicService
.
The following is an example of how this can be run:
./gradlew heroic-dist:runShadow <config>
which is the equivalent of doing:
java -jar $PWD/heroic-dist/build/libs/heroic-dist-0.0.1-SNAPSHOT-shaded.jar <config>
$ docker build -t heroic:latest .
This is a multi-stage build and will first build Heroic via a ./gradlew clean build
and then copy the resulting shaded jar into the runtime container.
Running heroic via docker can be done:
$ docker run -d -p 8080:8080 -p 9091:9091 -v /path/to/config.yml:/heroic.yml spotify/heroic:latest
Logging is captured using SLF4J, and forwarded to Log4j.
To configure logging, define the -Dlog4j.configurationFile=<path>
parameter. You can use docs/log4j2-file.xml as a base.
We run tests with Gradle:
# run unit tests
./gradlew test
# run integration tests
./gradlew integrationTest
or to run a more comprehensive set of checks:
./gradlew check
This will run:
- unit tests
- integration tests
- Checkstyle
- Coverage Reporting with Jacoco
It is strongly recommended that you run the full test suite before setting up a pull request, otherwise it will be rejected by Travis.
Full cluster tests are defined in heroic-dist/src/test/java.
This way, they have access to all the modules and parts of Heroic.
The JVM RPC module is specifically designed to allow for rapid execution of integration tests. It allows multiple cores to be defined and communicate with each other in the same JVM instance.
- See AbstractClusterQueryIT
- JVM-based JvmClusterQueryIT
- gRPC-based GrpcClusterQueryIT
There's an ongoing project to improve test coverage. Clicking the above graph will bring you to codecov.io, where you can find areas to focus on.
To bypass automatic formatting and checkstyle validation you can use the following stanza:
// @formatter:off
final List<String> list = ImmutableList.of(
"Welcome to...",
"... The Wild West"
);
// @formatter:on
To bypass a FindBugs error, you should use the @SupressFBWarnings
annotation.
@SupressFBWarnings(value="FINDBUGS_ERROR_CODE", justification="I Know Better Than FindBugs")
public class IKnowBetterThanFindbugs() {
// ...
}
The Heroic project is split into a couple of modules.
The most critical one is heroic-component
. It contains
interfaces, value objects, and the basic set of dependencies necessary to glue
different components together.
Submodules include metric
, suggest
,
metadata
, and aggregation
. The first three
contain various implementations of the given backend type, while the latter
provides aggregation methods.
heroic-core
contains the
com.spotify.heroic.HeroicCore
class which is the central building block for setting up a Heroic instance.
heroic-elasticsearch-utils
is a collection of
utilities for interacting with Elasticsearch. This is separate since we have
more than one backend that needs to talk with elasticsearch.
Finally there is heroic-dist
, a small project that depends on all module. Here is where everything is bound together into a distribution
โ a shaded jar. It also provides the entry-point for services, namely
com.spotify.heroic.HeroicService
or through an interactive shell com.spotify.heroic.HeroicShell
.
The shell can either be run standalone or connected to an existing Heroic instance for administration.
Guidelines for contributing can be found here.