skratchdot/elasticsearch-tools skratchdot/elasticsearch-tools

  • Stars
    star
    189
  • Rank 204,649 (Top 5 %)
  • Language
    JavaScript
  • License
    MIT License
  • Created over 10 years ago
  • Updated almost 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
skratchdot/elasticsearch-tools
github

skratchdot

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A collection of elasticsearch command line tools for doing things like bulk importing/exporting and exporting/importing mappings.

elasticsearch-tools

A collection of elasticsearch command line tools for doing things like bulk importing/exporting and exporting/importing mappings.

It was created because some of the existing import/export tools ran too slow on my machine. Using the new bulk API seemed to speed things up dramatically. The other tools I used also weren't exporting _parent and _routing fields.

Installation

npm install -g elasticsearch-tools

After installing, you will have access to the following command line tools:

Exporting

Importing

Usage: es-export-bulk

Options

es-export-bulk --help

Usage: es-export-bulk [options]

  Options:

    -h, --help                           output usage information
    -v, --version                        output the version number
    -u, --url <url>                      comma-separated elasticsearch urls to connect to
    -f, --file <file>                    the file to write data to
    -m, --max <number>                   the maximum number of items to export. different than the scroll size
    --transformMeta <js>                 a javascript function that returns an object that is the transformed meta object
    --transformSource <js>               a javascript function that returns an object that is the transformed source object
    --transformMetaInit <js>             a javascript function that returns an init object that contains helpers for the transform function
    --transformSourceInit <js>           a javascript function that returns an init object that contains helpers for the transform function
    --index <index>                      ES OPTION: a comma-separated list of index names to search; use _all or empty string to perform the operation on all indices
    --type <type>                        ES OPTION: a comma-separated list of document types to search; leave empty to perform the operation on all types
    --body <body>                        ES OPTION: the body to send along with this request.
    --analyzer <analyzer>                ES OPTION: The analyzer to use for the query string
    --analyzeWildcard <analyzeWildcard>  ES OPTION: specify whether wildcard and prefix queries should be analyzed (default: false)
    --fields <fields>                    ES OPTION: a comma-separated list of fields to return as part of a hit (default: "*")
    --from <from>                        ES OPTION: starting offset (default: 0)
    --q <q>                              ES OPTION: query in the Lucene query string syntax
    --routing <routing>                  ES OPTION: a comma-separated list of specific routing values
    --scroll <scroll>                    ES OPTION: specify how long a consistent view of the index should be maintained for scrolled search (default: 1m)
    --size <size>                        ES OPTION: number of hits to return during each scan
    --sort <sort>                        ES OPTION: a comma-separated list of <field>:<direction> pairs
    --timeout <timeout>                  ES OPTION: explicit operation timeout
    --apiVersion <apiVersion>            ES CLIENT OPTION: the major version of the Elasticsearch nodes you will be connecting to (default: 2.3)
    --maxRetries <maxRetries>            ES CLIENT OPTION: how many times should the client try to connect to other nodes before returning a ConnectionFault error (default: 3)
    --requestTimeout <requestTimeout>    ES CLIENT OPTION: milliseconds before an HTTP request will be aborted and retried. This can also be set per request (default: 30000)
    --deadTimeout <deadTimeout>          ES CLIENT OPTION: milliseconds that a dead connection will wait before attempting to revive itself (default: 60000)
    --pingTimeout <pingTimeout>          ES CLIENT OPTION: milliseconds that a ping request can take before timing out (default: 3000)
    --maxSockets <maxSockets>            ES CLIENT OPTION: maximum number of concurrent requests that can be made to any node (default: 10)
    --minSockets <minSockets>            ES CLIENT OPTION: minimum number of sockets to keep connected to a node (default: 10)
    --selector <selector>                ES CLIENT OPTION: select a connection from the ConnectionPool using roundRobin (default) or random

Examples

export 1 hour of data from local db

es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --body '
{"query":{"range":{"timestamp":{"gte":"2014-08-13T11:00:00.000Z","lte":"2014-08-13T12:00:00.000Z"}}}}
'

export "myIndex" from local db

es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --index myIndex

add a key/value to all exported documents

es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'data.foo = "neat"'
# the return statement is optional
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'data.foo = "neat";return data;'

delete the key "foo" from all exported documents

es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'delete data.foo'

don't include _parent in meta data

es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformMeta 'delete data.index._parent'

change the index name that we export

es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformMeta 'data.index._index = "newIndex"'

Usage: es-export-mappings

Options

es-export-mappings --help

Usage: es-export-mappings [options]

  Options:

    -h, --help                               output usage information
    -v, --version                            output the version number
    -u, --url <url>                          the elasticsearch url to connect to
    -f, --file <file>                        the file to write data to
    --index <index>                          ES OPTION: String, String[], Boolean — A comma-separated list of index names
    --type <type>                            ES OPTION: String, String[], Boolean — A comma-separated list of document types
    --ignoreUnavailable <ignoreUnavailable>  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)
    --allowNoIndices <allowNoIndices>        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)
    --expandWildcards <expandWildcards>      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.
    --local <local>                          ES OPTION: Boolean — Return local information, do not retrieve the state from master node (default: false)

Examples

export mappings from local db

es-export-mappings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.mappings.json

Usage: es-export-settings

Options

es-export-settings --help

Usage: es-export-settings [options]

  Options:

    -h, --help                               output usage information
    -v, --version                            output the version number
    -u, --url <url>                          the elasticsearch url to connect to
    -f, --file <file>                        the file to write data to
    --index <index>                          ES OPTION: String, String[], Boolean — A comma-separated list of index names
    --ignoreUnavailable <ignoreUnavailable>  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)
    --allowNoIndices <allowNoIndices>        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)
    --expandWildcards <expandWildcards>      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.
    --local <local>                          ES OPTION: Boolean — Return local information, do not retrieve the state from master node (default: false)
    --name <name>                            ES OPTION: String, String[], Boolean — The name of the settings that should be included

Examples

export settings from local db

es-export-settings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.settings.json

Usage: es-export-aliases

Options

es-export-aliases --help

Usage: es-export-aliases [options]

  Options:

    -h, --help         output usage information
    -v, --version      output the version number
    -u, --url <url>    the elasticsearch url to connect to
    -f, --file <file>  the file to write data to
    --index <index>    ES OPTION: String, String[], Boolean — A comma-separated list of index names
    --local <local>    ES OPTION: Boolean — Return local information, do not retrieve the state from master node (default: false)
    --name <name>      ES OPTION: String, String[], Boolean — The name of the settings that should be included

Examples

export aliases from local db

es-export-aliases --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.aliases.json

Usage: es-import-bulk

Options

es-import-bulk --help

Usage: es-import-bulk [options]

  Options:

    -v, --version          output the version number
    -u, --url <url>        the elasticsearch url to connect to
    -f, --file <file>      the file to read data from
    -m, --max <items>      the max number of lines to process per batch (default: 20,000) (default: 20000)
    --requestTimeout <ms>  ES CLIENT OPTION: milliseconds before an HTTP request will be aborted and retried. This can also be set per request (default: 30000) (default: 30000)
    -h, --help             output usage information

Examples

import data to local db from file

es-import-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/rafflev1.json

Usage: es-import-mappings

Options

es-import-mappings --help

Usage: es-import-mappings [options]

  Options:

    -v, --version                            output the version number
    -u, --url <url>                          the elasticsearch url to connect to
    -f, --file <file>                        the file to read data from
    --ignoreConflicts <ignoreConflicts>      ES OPTION: Boolean — Specify whether to ignore conflicts while updating the mapping (default: false)
    --timeout <timeout>                      ES OPTION: Date, Number — Explicit operation timeout
    --masterTimeout <masterTimeout>          ES OPTION: Date, Number — Specify timeout for connection to master
    --ignoreUnavailable <ignoreUnavailable>  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)
    --allowNoIndices <allowNoIndices>        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)
    --expandWildcards <expandWildcards>      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.
    -h, --help                               output usage information

Examples

import mappings to local db

es-import-mappings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.mappings.json

Usage: es-import-settings

Options

es-import-settings --help

Usage: es-import-settings [options]

  Options:

    -v, --version                            output the version number
    -u, --url <url>                          the elasticsearch url to connect to
    -f, --file <file>                        the file to read data from
    --masterTimeout <masterTimeout>          ES OPTION: Date, Number — Specify timeout for connection to master
    --ignoreUnavailable <ignoreUnavailable>  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)
    --allowNoIndices <allowNoIndices>        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)
    --expandWildcards <expandWildcards>      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.
    -h, --help                               output usage information

Examples

import settings to local db

es-import-settings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.settings.json

Usage: es-import-aliases

Options

es-import-aliases --help

Usage: es-import-aliases [options]

  Options:

    -v, --version                    output the version number
    -u, --url <url>                  the elasticsearch url to connect to
    -f, --file <file>                the file to read data from
    --timeout <timeout>              ES OPTION: Date, Number — Explicit operation timeout
    --masterTimeout <masterTimeout>  ES OPTION: Date, Number — Specify timeout for connection to master
    -h, --help                       output usage information

Examples

import aliases to local db

es-import-aliases --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.aliases.json

Other Elasticsearch Tools

Imports / Exports

Running tests

Unit tests can be ran via:

npm run test

The integration tests hit an elasticsearch server at: localhost:20202. To start the server, you can install docker, then run:

docker-compose up

One the server is running, you can run the integration tests via:

npm run test:integration

License

Copyright (c) 2014 skratchdot Licensed under the MIT license.

More Repositories

1

open-golang

Open a file, directory, or URI using the OS's default application for that object type. Optionally, you can specify an application to use.
Go
757
star
2

react-bootstrap-daterangepicker

A date/time picker for react (using bootstrap). This is a react wrapper around the bootstrap-daterangepicker project.
JavaScript
477
star
3

random-useragent

Get a random user agent (with an optional filter to select from a specific set of user agents).
JavaScript
233
star
4

react-github-corner

Add a Github banner to your project page. A React version of: https://github.com/tholman/github-corners / See also: http://projects.skratchdot.com/react-github-corner/
JavaScript
178
star
5

react-bootstrap-multiselect

A multiselect component for react (with bootstrap). This is a react port of bootstrap-multiselect.
JavaScript
119
star
6

color-blind

Simulate color blindness by converting RGB hex codes.
JavaScript
94
star
7

random-seed

GRC's UHE PRNG in node (Ultra-High Entropy Pseudo-Random Number Generator by Gibson Research Corporation)
JavaScript
91
star
8

color-harmony

Create color scales by rotating hue
JavaScript
80
star
9

github-code-search.user.js

A user script that adds a search box to repository pages which allows you to search the code in that repository.
JavaScript
53
star
10

github-enhancement-suite

A collection of userscripts specific to Github
JavaScript
43
star
11

open-electribe-editor

Open Electribe Editor (formerly known as ESX Wave Organizer) is a program that allows you to organize, edit, and create .esx files for the Korg ESX-1.
43
star
12

ble-midi

A Bluetooth Low Energy MIDI library written in javascript
JavaScript
39
star
13

audio-sort

A webpage to visualize and audibilize sorting algorithms using javascript.
JavaScript
39
star
14

mesh

An extended shell for MongoDB
JavaScript
37
star
15

audio-editor

An audio editor built with React
JavaScript
35
star
16

npm-dview

Compare current package.json dependency version numbers with latest remote version number.
JavaScript
30
star
17

mongodb-schema

A schema analysis tool for MongoDB
JavaScript
28
star
18

color-matrix

Apply filters to colors to simulate things like color blindness and night vision.
JavaScript
22
star
19

soundfont2mp3

soundfont2mp3 is a command line utility that lets you extract single note mp3s from soundfont files.
JavaScript
15
star
20

infinite-gradients

An HTML experiment for generating randomly created CSS3 gradients (both linear and radial)
JavaScript
12
star
21

colorify

a collection of color tools
JavaScript
12
star
22

audio-grid

A javascript port of the great Tone Matrix tool by Andre Michelle
JavaScript
10
star
23

color-palette

Get a color palette from a uri using phantomjs and color-thief
JavaScript
10
star
24

github-repo-filter-info.user.js

A user script to display some additional info below the repository filter on a user's main GitHub page.
JavaScript
9
star
25

github-get-missing-descriptions.user.js

If there are missing descriptions on a Github profile page, a button will be added. When clicked, ajax requests will be made to grab the descriptions.
JavaScript
8
star
26

github-repo-counts.user.js

A user script to display repo counts when browsing Github repository pages.
JavaScript
7
star
27

color-stats

Generate color information based off of the hash of any object (or a valid color string)
JavaScript
7
star
28

github-pull-request-links.user.js

A user script to "linkify" the to/from branches on Pull Request pages.
JavaScript
6
star
29

amd-to-commonjs-codemod

A codemod to transform amd style includes into commonjs includes
JavaScript
6
star
30

object-path-set

set values in javascript objects by specifying a path
JavaScript
5
star
31

skratchdot.com

source code for skratchdot.com
HTML
5
star
32

sox.js

An emscripten port of sox for use in the browser and node
JavaScript
5
star
33

Git-Diff-Build-Script

A shell script to create "deploy" and "restore" folders based off of a git diff of 2 branches.
Shell
5
star
34

binary-format

a typescript library for reading and writing binary files to and from plain javascript objects
TypeScript
5
star
35

vagrant-open-electribe-editor

a vagrant box with open electribe editor installed on it
Shell
5
star
36

object-path-get

get values from javascript objects by specifying a path
JavaScript
4
star
37

color-quantize

Convert colors to websafe / websmart values
JavaScript
4
star
38

random-picker

Randomly pick an item from a set (using the specified probabilities).
JavaScript
4
star
39

react-file-processor

A React component for choosing files
JavaScript
4
star
40

github-fork-count.user.js

A user script to display the fork count underneath the "public" repo count on a user's main GitHub page.
JavaScript
3
star
41

philharmonia-samples

A mirror of the samples from: http://www.philharmonia.co.uk/explore/make_music
3
star
42

esx-files

a collection of esx files
HTML
3
star
43

browsercheck

a tool to check your scripts for browser compatibility
TypeScript
3
star
44

get-object-type

get the type of a javascript object
JavaScript
3
star
45

riff-wav-for-java

This EMF based project was created to allow the reading and writing of .wav files in Java. It attempts to read in well-known RIFF chunks, and provide an API for reading/manipulating those chunks.
Java
3
star
46

sox

A mirror of the sox codebase: https://sourceforge.net/p/sox/code/ci/master/tree/
C
3
star
47

js-playground

benchmark js snippets in a web worker
JavaScript
2
star
48

mongodb-flatten

The flatten() function provides a way to flatten documents into id/key/value pairs.
JavaScript
2
star
49

timbre.soundfont.js

play soundfont urls using timbre.js
JavaScript
2
star
50

github-gh-pages-link.user.js

If a repository has a gh-pages branch, then this will add links to the Github Page, as well as the gh-page source code.
JavaScript
2
star
51

audio-automator

A better way to change AudioParam values
JavaScript
2
star
52

github-editor-theme.user.js

Add a "theme" dropdown when editing files in the Github UI.
JavaScript
2
star
53

svg-react-experiments

A collection of a svg experiments using React
HTML
2
star
54

stats-collector

Collect stats about numbers
JavaScript
2
star
55

mongodb-wild

Adds a wildcard search to the mongodb shell
JavaScript
2
star
56

vagrant-elasticsearch

a vagrant box with elasticsearch tools installed
HTML
2
star
57

framecapture

Take a screenshot of a portion of your screen, and add a border to it in one swoop.
Shell
1
star
58

help.github.com

GitHub help guides
JavaScript
1
star
59

free-midi

a collection of mp3s generated by soundfont2mp3 (see: https://code.google.com/archive/p/free-midi/)
JavaScript
1
star
60

code

a dumping ground for random code snippets and miscellaneous projects
JavaScript
1
star
61

w3c-html-elements.user.js

A user script to display additional information for each html element listed on the W3C elements page.
JavaScript
1
star
62

storybook-7-issue

a reproduction of a potential storybook v7 issue
TypeScript
1
star
63

github-twitter-link.user.js

Adds a twitter link on Github profile pages if a corresponding user name exists at Twitter.
JavaScript
1
star
64

list-tool

a simple web application to compare lists
JavaScript
1
star
65

domFormat

domFormat is a simple javascript library to get DOM nodes as formatted/pretty strings.
JavaScript
1
star
66

grunt-init-react

grunt-init template for a very basic react project
JavaScript
1
star
67

basecamp-search-all.user.js

A user script that displays a "search all projects" form in the header of BaseCamp pages. It only displays when logged in.
JavaScript
1
star
68

skratchdot-codemods

a collection of utility codemods
JavaScript
1
star
69

audio-generation-loss

Evaluating lossless/lossy audio compression and generation loss
JavaScript
1
star
70

bytebeat

a site for live coding bytebeat
TypeScript
1
star
71

rockbox-theme-classic-v2

A Rockbox Theme for the Sansa Clip+
1
star
72

audio-files

a collection of test audio files for my web audio projects
HTML
1
star
73

mongodb-distinct-types

Similar to the db.myCollection.distinct() function, distinctTypes() will return "types" rather than "values"
JavaScript
1
star
74

array-any-index

get values from an array using any index: negative or float
JavaScript
1
star
75

greasemonkey-membase-view-top-key-values

A GreaseMonkey script that linkifies the list of top keys in the Membase admin console.
JavaScript
1
star