es2unix
Deprecated since elasticsearch 1.0. Use the cat API instead. Also check out https://github.com/drewr/copycat.
Elasticsearch API consumable by the command line.
JSON isn't always the most convenient output, particularly on a
terminal. The tabular format has stuck around for good reason. It's
compact. It's line-oriented. es2unix strives to keep spaces
significant so all output works with existing *NIX tools. grep,
sort, & awk are first-class citizens here.
Install
es2unix's only dependency is Java (Oracle or OpenJDK). Version 7 should be preferred, but there is no functional difference with 6. Earlier versions aren't supported.
curl -s download.elasticsearch.org/es2unix/es >~/bin/es
chmod +x ~/bin/es
You can also pin to a particular version from your provisioning tools.
curl -s download.elasticsearch.org/es2unix/es-20130503595fce2 >~/bin/es
Usage
The es command takes subcommands and a few options. It assumes it's
talking to ES at its default HTTP port using http://localhost:9200
but accepts -u to change that. It must be a fully qualifed URL with
scheme, host, & port.
You can also supply -v, for most commands, to print a column header.
Version
% es version
es 1.0.0
elasticsearch 0.21.0.Beta1
Health
% es health -v
cluster status nodes data pri shards relo init unassign
kluster green 2 2 3 6 0 0 0
Count
Sometimes you need a quick count to tell whether a cluster has any data and whether it's indexing. You can also supply a query.
% es count
2,319,799
% es count elasticsearch
3 "q=elasticsearch"
% es count "john deacon"
225,839 "q=john deacon"
% es count "\"saved by the bell\""
220 "q="saved by the bell""
Search
Not exhaustive access to the query API by any stretch, but it suffices
when you need to get a glance of the data in your cluster. Searches
across indices with a default query of *:*.
% es search
1.0 wiki page 1228929
1.0 wiki page 1229142
1.0 wiki page 1229146
1.0 wiki page 1229153
1.0 wiki page 1228943
1.0 wiki page 1229155
1.0 wiki page 1228950
1.0 wiki page 1229159
1.0 wiki page 1228956
1.0 wiki page 1229160
Total: 2319799
Can also specify a query, like es search \"george costanza\", and,
possibly more interestingly, a list of fields to return.
% es search -v "george costanza" title
score index type id title
5.78647 wiki page 660183 George Costansa
5.78647 wiki page 273868 George Constanza
5.63803 wiki page 865781 Vandelay Industries
4.69835 wiki page 932333 Art Vandelay
4.69835 wiki page 2147975 Can't Stand Ya
4.67351 wiki page 2486208 Art vandelay
4.07630 wiki page 2147959 Costanza
3.23200 wiki page 2147971 The Costanza family
3.21007 wiki page 2147972 Costanza family
2.94863 wiki page 4946953 Santa costanza
Total: 118186
Master
% es master
J-erllamTOiW5WoGVUd04A 127.0.0.1 Slade, Frederick
Indices
% es indices -v
status name pri rep docs size
green _river 0 1 4 8068
green wiki 1 1 1104894 13805525784
Maybe your cluster is red and you need to know which indices are affected:
% es indices | grep \^red
red bb 5 0
red test 4 1 218b 218 0
red enron 5 0
red uno 1 0
Allocation
Displays shard allocation counts across nodes
% es allocation -v
count ip name
12 192.168.0.24 Sage
30 x.x.x.x UNASSIGNED
Nodes
What HTTP port is Cannonball I listening on? Who's the master?
Who's master-eligible? Who's got data=true?
% es nodes
Uv1Iy8FvR0y6_RzPXKBolg 127.0.0.1 9201 127.0.0.1 9300 d Cannonball I
J-erllamTOiW5WoGVUd04A 127.0.0.1 9200 127.0.0.1 9301 * d Slade, Frederick
j27iagsmQQaeIpl6yU6mCg 127.0.0.1 9203 127.0.0.1 9303 - c Georgianna Castleberry
T1aFDU2BSUm748gYxjEN9w 127.0.0.1 9202 127.0.0.1 9302 d Living Tribunal
If you have access to logs from all the nodes, you can run lifecycle
to get a playback of all the node joinings and leavings with their
timestamps ordered sequentially. This is much faster than combing the
logs and piecing together the sequence manually.
% es lifecycle /tmp/es-*/logs/elasticsearch.log
2013-02-08 13:47:15,516 Lurking Unknown INIT 0.21.0.Beta1-SNAPSHOT
2013-02-08 13:47:20,413 Lurking Unknown MASTER Lurking Unknown
2013-02-08 13:47:20,467 Lurking Unknown START
2013-02-08 13:47:36,319 Cameron Hodge INIT 0.21.0.Beta1-SNAPSHOT
2013-02-08 13:47:41,211 Lurking Unknown ADD Cameron Hodge
2013-02-08 13:47:41,223 Cameron Hodge MASTER Lurking Unknown
2013-02-08 13:47:41,278 Cameron Hodge START
2013-02-08 13:47:59,426 Armageddon INIT 0.21.0.Beta1-SNAPSHOT
2013-02-08 13:48:04,279 Lurking Unknown ADD Armageddon
2013-02-08 13:48:04,280 Cameron Hodge ADD Armageddon
2013-02-08 13:48:04,287 Armageddon MASTER Lurking Unknown
2013-02-08 13:48:04,340 Armageddon START
2013-02-08 13:48:30,333 Lurking Unknown REMOVE Armageddon
2013-02-08 13:48:30,339 Cameron Hodge REMOVE Armageddon
2013-02-08 13:48:30,362 Armageddon STOP
Heap
Heap across the cluster.
% es heap | sort -rnk6
XO6c2A1D 23.9mb 25138608 123.7mb 129761280 19.4% 127.0.0.1 Junkpile
uVP8g9_l 94.6mb 99257976 990.7mb 1038876672 9.6% 127.0.0.1 Hammond, Jim
pjbeg_k8 76.9mb 80730208 990.7mb 1038876672 7.8% 127.0.0.1 Scarlet Centurion
For some quick and dirty monitoring, I like to put this in a loop.
% while true; do es heap | sort -rnk6 | head -1; sleep 60; done
XO6c2A1D 57.3mb 60157200 123.7mb 129761280 46.4% 127.0.0.1 Junkpile
XO6c2A1D 54.7mb 57405904 123.7mb 129761280 44.2% 127.0.0.1 Junkpile
XO6c2A1D 62.7mb 65834752 123.7mb 129761280 50.7% 127.0.0.1 Junkpile
XO6c2A1D 56.9mb 59743504 123.7mb 129761280 46.0% 127.0.0.1 Junkpile
XO6c2A1D 52.1mb 54676216 123.7mb 129761280 42.1% 127.0.0.1 Junkpile
XO6c2A1D 37.1mb 38971744 123.7mb 129761280 30.0% 127.0.0.1 Junkpile
XO6c2A1D 52mb 54528424 123.7mb 129761280 42.0% 127.0.0.1 Junkpile
XO6c2A1D 46.5mb 48787064 123.7mb 129761280 37.6% 127.0.0.1 Junkpile
This can be extremely helpful during indexing, for example. If you see a single node showing up a lot, you might have hot shard(s) there. If you see all the nodes regularly showing up with varying heap usage percentage, it's likely a healthy cluster with good shard distribution.
Searching has slightly different characteristics, but you can make similarly helpful inferences.
Ids
Sometimes it's helpful to retrieve the ids of all documents in an ES index.
% es ids test -v
index type id
test doc 1
test doc 2
test doc 3
test doc 4
Shards
Node startup
We've started up three nodes where we had two before. ES decided to move one shard to the third node.
% es shards
wiki 0 p STARTED 1160290 7.2gb 7776371641 127.0.0.1 Feline
wiki 0 r STARTED 1160290 7.2gb 7776371602 127.0.0.1 Jenkins, Abner
wiki 1 p RELOCATING 1159509 7.5gb 8116295811 127.0.0.1 Feline -> 127.0.0.1 Amphibius
wiki 1 r STARTED 1159509 7.5gb 8116295811 127.0.0.1 Jenkins, Abner
After turning on more replicas
We set index.number_of_replicas to 2, so ES is creating another
copy of each primary shard.
% es shards
wiki 0 p STARTED 1160290 7.2gb 7776371641 127.0.0.1 Feline
wiki 0 r INITIALIZING 0 100.2mb 105077522 127.0.0.1 Amphibius
wiki 0 r STARTED 1160290 7.2gb 7776371602 127.0.0.1 Jenkins, Abner
wiki 1 r INITIALIZING 0 120.3mb 126157581 127.0.0.1 Feline
wiki 1 p STARTED 1159509 7.5gb 8116295811 127.0.0.1 Amphibius
wiki 1 r STARTED 1159509 7.5gb 8116295811 127.0.0.1 Jenkins, Abner
Single node filter by index, sort reverse by bytes
You can limit the results to a substring match of an index. This filters that output's sixth column through a descending sort.
% es shards wik | sort -rnk6
wiki 1 r STARTED 2.7gb 2980767835 276016 127.0.0.1 Namora
wiki 0 r STARTED 2.7gb 2953985585 276441 127.0.0.1 Namora
wiki 1 p STARTED 2.7gb 2909784771 276016 127.0.0.1 Android Man
wiki 0 p STARTED 2.6gb 2846741702 276441 127.0.0.1 Android Man
Normal three-node cluster operation
Add column names.
% es shards -v
index shard pri/rep state docs size bytes ip node
wiki 0 p STARTED 1160290 7.2gb 7776371641 127.0.0.1 Feline
wiki 0 r INITIALIZING 0 3.1gb 3384641066 127.0.0.1 Amphibius
wiki 0 r STARTED 1160290 7.2gb 7776371602 127.0.0.1 Jenkins, Abner
wiki 1 r INITIALIZING 0 3.7gb 4029041251 127.0.0.1 Feline
wiki 1 p STARTED 1159509 7.5gb 8116295811 127.0.0.1 Amphibius
wiki 1 r STARTED 1159509 7.5gb 8116295811 127.0.0.1 Jenkins, Abner
Contributing
es2unix is written in Clojure. You'll need leiningen 2.0+ to build.
% make package
License
This software is licensed under the Apache 2 license, quoted below.
Copyright 2012-2013 ElasticSearch <http://www.elasticsearch.org>
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.