• This repository has been archived on 18/May/2020
  • Stars
    star
    269
  • Rank 152,662 (Top 4 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Tool that ease up creation of integration tests with Elasticsearch

This project is no longer maintained. We recommend using Testcontainers instead: https://www.testcontainers.org/modules/elasticsearch/

embedded-elasticsearch

Build Status Maven Central

Small utility for creating integration tests that use Elasticsearch. Instead of using Node it downloads Elasticsearch in specified version and starts it in a separate process. It also allows you to install required plugins which is not possible when using NodeBuilder. Utility was tested with 1.x, 2.x, 5.x and 6.x versions of Elasticsearch.

Introduction

All you need to do to use this tool is create EmbeddedElastic instance. To do so, use provided builder:

final embeddedElastic = EmbeddedElastic.builder()
        .withElasticVersion("5.0.0")
        .withSetting(PopularProperties.TRANSPORT_TCP_PORT, 9350)
        .withSetting(PopularProperties.CLUSTER_NAME, "my_cluster")
        .withPlugin("analysis-stempel")
        .withIndex("cars", IndexSettings.builder()
            .withType("car", getSystemResourceAsStream("car-mapping.json"))
            .build())
        .withIndex("books", IndexSettings.builder()
            .withType(PAPER_BOOK_INDEX_TYPE, getSystemResourceAsStream("paper-book-mapping.json"))
            .withType("audio_book", getSystemResourceAsStream("audio-book-mapping.json"))
            .withSettings(getSystemResourceAsStream("elastic-settings.json"))
            .build())
        .build()
        .start()

When you are done with creating it, you can start it real simple:

embeddedElastic.start()

And that's all, you can connect to your embedded-elastic instance on specified port and use it in your tests.

Available builder options

Method Description
withElasticVersion(String version) version of Elasticsearch; based on that version download url to official Elasticsearch repository will be created
withDownloadUrl(URL downloadUrl) if you prefer to download Elasticsearch from a different location than official repositories you can do that using this method
withInResourceLocation(String inResourcePath) you can also have your Elasticsearch package inside resource directory, specify it's location with this option
withSetting(String key, Object value) setting name and value as in elasticsearch.yml file
withPlugin(String expression) plugin that should be installed into Elasticsearch; treat expression as argument to ./elasticsearch-plugin install <expression> command; use multiple times for multiple plugins
withIndex(String indexName, IndexSettings indexSettings) specify index that should be created and managed by EmbeddedElastic
withTemplate(String templateName, String templateBody) specify a template that should be created and managed by EmbeddedElastic
withStartTimeout(long value, TimeUnit unit) specify timeout you give Elasticsearch to start
withInstallationDirectory(File installationDirectory) specify custom installation directory
withDownloadDirectory(File downloadDirectory) specify custom download directory where downloaded distribution packages will be saved
withCleanInstallationDirectoryOnStop(boolean cleanInstallationDirectoryOnStop) specify whether clean the installation directory after Elasticsearch stop
withEsJavaOpts(String javaOpts) value of ES_JAVA_OPTS variable to be set for Elasticsearch process
withJavaHome(JavaHomeOption javaHomeOption) select java environment to run in. For available options see below
getTransportTcpPort() get transport tcp port number used by Elasticsearch instance
getHttpPort() get http port number used by Elasticsearch instance
withDownloadProxy(Proxy proxy) proxy that should be used for downloading Elasticsearch package
withDownloaderConnectionTimeout(long value, TimeUnit unit) connection timeout that should be used by downloader
withDownloaderReadTimeout(long value, TimeUnit unit) socket timeout that should be used by downloader

Available IndexSettings.Builder options

Method Description
withType(String type, String mapping) specify type and it's mappings
withSettings(String settings) specify index settings

Availabe JavaHomeOption options

Method Description
useSystem() default behavior, lets elasticsearch startup script determine the JRE to run in
inheritTestSuite() use the same JRE as the process starting it
path(String path) manually set the path of the JRE to execute the embedded elastic

Available operations

EmbeddedElastic provides following operations:

Method Description
start() downloads Elasticsearch and specified plugins, setups everything and finally starts your Elasticsearch instance
stop() stops your Elasticsearch instance and removes all data
index index your document, comes with variants that take only document, or document and it's id
deleteIndex(String indexName), deleteIndices() deletes index with name specified during EmbeddedElastic creation
createIndex(String indexName), createIndices() creates index with name specified during EmbeddedElastic creation; note that this index is created during EmbeddedElastic startup, you will need this method only if you deleted your index using deleteIndex method
recreateIndex(String indexName), recreateIndices() combination of deleteIndex and createIndex
refreshIndices() refresh index; useful when you make changes in different thread, and want to check results instantly in tests
deleteTemplate(String templateName), deleteTemplates() deletes a template of the specified name/all templates during EmbeddedElastic creation
createTemplate(String templateName), createTemplates() creates a template with the specified name/all templates during EmbeddedElastic creation; note that this template is created during EmbeddedElastic startup, you will need this method only if you deleted your template using deleteTemplate method
recreateTemplate(String templateName), recreateTemplates() combination of deleteTemplate and createTemplate

Example

If you want to see example, look at this spec: pl.allegro.tech.search.embeddedelasticsearch.EmbeddedElasticSpec

Dependency

To start using embedded-elasticsearch in your project add it as a test dependency:

Gradle:

testCompile 'pl.allegro.tech:embedded-elasticsearch:2.7.0'

Maven:

<dependency>
    <groupId>pl.allegro.tech</groupId>
    <artifactId>embedded-elasticsearch</artifactId>
    <version>2.7.0</version>
    <scope>test</scope>
</dependency>

SBT:

libraryDependencies ++= Seq("pl.allegro.tech" % "embedded-elasticsearch" % "2.7.0" % "test")

Known problems

If you build your project on Travis, you may have problems with OOM errors when using default settings. You can change Elasticsearch memory settings using withEsJavaOpts method. Example (from spec pl.allegro.tech.embeddedelasticsearch.EmbeddedElasticSpec):

    static EmbeddedElastic embeddedElastic = EmbeddedElastic.builder()
            .withElasticVersion(ELASTIC_VERSION)
            .withSetting(TRANSPORT_TCP_PORT, TRANSPORT_TCP_PORT_VALUE)
            .withSetting(CLUSTER_NAME, CLUSTER_NAME_VALUE)
            .withEsJavaOpts("-Xms128m -Xmx512m")
            .withIndex(CARS_INDEX_NAME, CARS_INDEX)
            .withIndex(BOOKS_INDEX_NAME, BOOKS_INDEX)
            .withStartTimeout(1, MINUTES)
            .build()
            .start()

Running more then one Elasticsearch instance

There are cases where you might want to run more than one Elasticsearch instance e.g.:

  • running tests of one project in parallel (e.g. using gradle --parallel or mvn -T1C)
  • running tests of different projects on the same physical mashine (e.g. Jenkins jobs running on the same server)
  • running integration tests which require more than one Elasticsearch instance

In such situations you should use distinct values for following settings for each instance:

  • withSetting(PopularProperties.TRANSPORT_TCP_PORT, ...)
  • withSetting(PopularProperties.HTTP_PORT, ...)
  • withInstallationDirectory(...)

With such configuration embedded-elasticsearch will redownload elasticsearch installation package for every distinct installation directory. To avoid this behavior and reuse downloaded installation package you should set common location of downloaded files with withDownloadDirectory(...) for every embedded-elasticsearch configuration.

License

embedded-elasticsearch is published under Apache License 2.0.

More Repositories

1

bigcache

Efficient cache for gigabytes of data written in Go.
Go
7,508
star
2

ralph

Ralph is the CMDB / Asset Management system for data center and back office hardware.
Python
2,200
star
3

tipboard

Tipboard - in-house, tasty, local dashboarding system
JavaScript
1,106
star
4

php-protobuf

PHP Protobuf - Google's Protocol Buffers for PHP
PHP
911
star
5

allRank

allRank is a framework for training learning-to-rank neural models based on PyTorch.
Python
842
star
6

hermes

Fast and reliable message broker built on top of Kafka.
Java
808
star
7

turnilo

Business intelligence, data exploration and visualization web application for Druid, formerly known as Swiv and Pivot
TypeScript
725
star
8

axion-release-plugin

Gradle release & version management plugin.
Groovy
556
star
9

node-worker-nodes

A node.js library to run cpu-intensive tasks in a separate processes and not block the event loop.
JavaScript
489
star
10

typescript-strict-plugin

Typescript plugin that allows turning on strict mode in specific files or directories.
TypeScript
337
star
11

json-avro-converter

JSON to Avro conversion tool designed to make migration to Avro easier.
Groovy
275
star
12

vaas

VaaS
Python
229
star
13

grunt-maven-plugin

Grunt + Maven integration done right
Java
213
star
14

allegro-api

Issue tracker and wiki for Allegro REST API
212
star
15

tradukisto

A Java i18n library created to convert numbers to their word representations.
Groovy
203
star
16

marathon-consul

Integrates Marathon apps with Consul service discovery.
Go
190
star
17

restapi-guideline

Allegro REST API Guideline.
CSS
183
star
18

bigflow

A Python framework for data processing on GCP.
Python
116
star
19

handlebars-spring-boot-starter

Spring Boot auto-configuration for Handlebars
Groovy
109
star
20

envoy-control

Envoy Control is a platform-agnostic, production-ready Control Plane for Service Mesh based on Envoy Proxy.
Kotlin
98
star
21

akubra

Simple solution to keep a independent S3 storages in sync
Go
86
star
22

elasticsearch-analysis-morfologik

Morfologik Polish Lemmatizer plugin for Elasticsearch
Java
83
star
23

ecto-cursor-based-stream

Elixir library that allows for cursor-based streaming of Ecto records, that does not require database transaction.
Elixir
81
star
24

allms

A versatile and powerful library designed to streamline the process of querying LLMs
Python
70
star
25

opel

OPEL - asynchronous expression language
Java
68
star
26

HerBERT

HerBERT is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic masking of whole words.
64
star
27

fogger

Fogger - a library to create blurred background under Android's UI elements
Java
60
star
28

mesos-executor

Customizable Apache Mesos task executor
Go
49
star
29

json-logic-kmp

Kotlin multiplatform JsonLogic expressions evaluation engine. Targets iOS and JVM (also Android).
Kotlin
49
star
30

elasticsearch-reindex-tool

Elasticsearch reindexing tool.
Java
45
star
31

selena

SELENA is a tool used to test website performance by measuring response times or verifying the content of replies.
JavaScript
45
star
32

kafka-offset-monitor-graphite

Graphite reporter for Kafka Offset Monitor.
Scala
44
star
33

dotnet-utils

C#
37
star
34

cassandra-modeling-kata

Cassandra Modeling Kata
Java
36
star
35

mongo-migration-stream

Tool for online migrations of MongoDB databases.
Kotlin
33
star
36

swift-junit

A Swift library for creating JUnit XML test results that can be interpreted by tools such as Bamboo or Jenkins. Macos and Linux ready.
Swift
29
star
37

slinger

Slinger - deep linking library for Android
Java
29
star
38

django-powerdns-dnssec

Django application managing PowerDNS database
Python
27
star
39

ralph-cli

Command-line interface for the Ralph system.
Go
26
star
40

allegro.tech

TypeScript
23
star
41

hacktoberfest-dashboard

Allegro Hactoberfest activity dashboard
TypeScript
23
star
42

klejbenchmark-baselines

Fine-tuning scripts for evaluating transformer-based models on KLEJ benchmark.
Python
23
star
43

quanta

Fast image optimization as a service, based on mozjpeg, written in Swift
C
21
star
44

bigcache-bench

Benchmarks for BigCache project
Go
20
star
45

pyhermes

The Python interface to the Hermes message broker.
Python
19
star
46

swiftbox

SwiftBox is a package that helps building Swift/Vapor microservices.
Swift
19
star
47

consul-registration-hook

Hook that can be used for synchronous registration and deregistration in Consul discovery service on Kubernetes or Mesos cluster with Allegro executor
Go
18
star
48

marathon-appcop

Marathon applications law enforcement
Go
18
star
49

leader-only-spring-boot-starter

Java
17
star
50

newrelic-gradle-plugin

Newrelic Gradle plugin.
Groovy
17
star
51

map-with-indifferent-access

Elixir
17
star
52

grunt-maven-npm

npm tasks for grunt-maven-plugin 1.2+
JavaScript
16
star
53

redux-storage-decorator-engines

Composing decorator for redux-storage to use different storage types
JavaScript
15
star
54

spunit

Spunit – Spock elegance in Kotlin JUnit 5 tests
Kotlin
14
star
55

cosmosdb-utils

A collection of useful Azure CosmosDb SDK v3 extensions and utilities, developed as part of Allegro Pay product.
C#
13
star
56

envoy-perf-pprof

Convenient Envoy on-CPU performance analysis with perf and pprof.
Dockerfile
13
star
57

dotnet-sdk

C#
12
star
58

prometheus-net-metrics

C#
12
star
59

camus-compressor

Camus Compressor merges files created by Camus and saves them in a compressed format.
Java
12
star
60

blog

HTML
11
star
61

consul-recipes

Java library for interacting with Consul.
Java
11
star
62

ralph_pricing

A pricing module for Ralph
Python
11
star
63

solr-fast-collapsing-query-parser

Java
9
star
64

klejbenchmark-allegroreviews

Allegro Reviews is a sentiment analysis dataset, consisting of 11,588 product reviews written in Polish and extracted from Allegro.pl - a popular e-commerce marketplace.
9
star
65

inkpy-jinja

Generate PDF documents from ODT templates.
Python
8
star
66

allegro-tech-labs-microservices

Allegro Tech Labs Microservices workshop materials
Java
8
star
67

TypedListAdapter

Kotlin
7
star
68

logextractx

Python
7
star
69

django-bob

Django bob is a set of django helpers, widgets and form filters for Ralph DCIM/CMDB project .
JavaScript
7
star
70

solr-ids-export-plugin

Java
6
star
71

toper

PHP Rest client based on popular Guzzle Rest Client.
PHP
6
star
72

atm-event-app

ATM event application
JavaScript
5
star
73

application-insights

5
star
74

graphql-extended-audit-intstrumentation

Java
5
star
75

votakvot

Python
5
star
76

banana-split

JavaScript
5
star
77

conformal-prediction-wut

Jupyter Notebook
5
star
78

AlleNoise

Python
4
star
79

warsztaty-podstawy-ml-03-2019

Machine learning basics workshop
Jupyter Notebook
4
star
80

selena-agent

Agent for Selena
Python
4
star
81

json-cache

Java
3
star
82

swiftbox-config

Swift
3
star
83

vaas-registration-hook

Go
3
star
84

braincode

HTML
3
star
85

oauth-mock

Kotlin
3
star
86

parallel-test-execution-workshop

Resources for Parallel test execution workshop
Groovy
3
star
87

swiftbox-metrics-statsd

Swift
2
star
88

atm-hero-generator

JavaScript
2
star
89

client-side-logic-dsl

Kotlin
2
star
90

eslint-plugin-test-comments

TypeScript
2
star
91

couchbase-commons

Kotlin
1
star
92

hermes-page

Hermes OpenSource page.
HTML
1
star
93

ml

TypeScript
1
star
94

swiftbox-logging

Swift
1
star
95

podcast.allegro.tech

CSS
1
star
96

axion-release-example

Kotlin
1
star
97

jobs-conf

allegro.tech jobs postings
HTML
1
star
98

allegro-tech-labs-iot

Allegro Tech Labs IoT workshop materials
Python
1
star
99

versionlens.nvim

1
star