• Stars
    star
    1,480
  • Rank 31,765 (Top 0.7 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created almost 8 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Plugin to integrate Learning to Rank (aka machine learning for better relevance) with Elasticsearch

Build Status

The Elasticsearch Learning to Rank plugin uses machine learning to improve search relevance ranking. It's powering search at places like Wikimedia Foundation and Snagajob!

What this plugin does...

This plugin:

  • Allows you to store features (Elasticsearch query templates) in Elasticsearch
  • Logs features scores (relevance scores) to create a training set for offline model development
  • Stores linear, xgboost, or ranklib ranking models in Elasticsearch that use features you've stored
  • Ranks search results using a stored model

Where's the docs?

We recommend taking time to read the docs. There's quite a bit of detailed information about learning to rank basics and how this plugin can ease learning to rank development.

You can also participate in regular trainings on Elasticsearch Learning to Rank, which support the free work done on this plugin.

I want to jump in!

The demo lives in another repo now, Hello LTR and it has both ES and Solr example. Follow the directions for Elasticsearch in the README to set up the environment and start with the notebooks/elasticsearch/tmdb/hello-ltr.ipynb. Have fun!

Installing

See the full list of prebuilt versions and select the version that matches your Elasticsearch version. If you don't see a version available, see the link below for building or file a request via issues.

To install, you'd run a command like this but replacing with the appropriate prebuilt version zip:

./bin/elasticsearch-plugin install https://github.com/o19s/elasticsearch-learning-to-rank/releases/download/v1.5.4-es7.11.2/ltr-plugin-v1.5.4-es7.11.2.zip

(It's expected you'll confirm some security exceptions, you can pass -b to elasticsearch-plugin to automatically install)

If you already are running Elasticsearch, don't forget to restart!

Known issues

As any other piece of software, this plugin is not exempt from issues. Please read the known issues to learn about the current issues that we are aware of. This file might include workarounds to mitigate them when possible.

Build and Deploy Locally

Notes if you want to dig into the code or build for a version there's no build for, please feel free to run the build and installation process yourself:

./gradlew clean check
./bin/elasticsearch-plugin install file:///path/to/elasticsearch-learning-to-rank/build/distributions/ltr-<LTR-VER>-es<ES-VER>.zip

How to Contribute

For more information on helping us out (we need your help!), developing with the plugin, creating docs, etc please read CONTRIBUTING.md.

Elastic Release Support

We do our best to officially support *.*.1 releases of Elasticsearch. If you have a need for "dot-oh" compatibility or a version we don't support please consider submitting a PR.

Who built this?

Other Acknowledgments & Stuff To Read

More Repositories

1

relevant-search-book

Code and Examples for Relevant Search
Jupyter Notebook
300
star
2

quepid

Improve your Elasticsearch, OpenSearch, Solr, Vectara, Algolia and Custom Search search quality.
Ruby
284
star
3

hello-ltr

Set of Jupyter notebooks demonstrating Learning to Rank integrated with Solr and Elasticsearch
Jupyter Notebook
165
star
4

elyzer

"Stop worrying about Elasticsearch analyzers", my therapist says
Python
153
star
5

splainer

Elasticsearch/Solr Sandbox for exploring explain information and tweaking
JavaScript
135
star
6

hello-nlp

A natural language search microservice
Python
96
star
7

awesome-search-relevance

Tools and other things for people who work on search relevance & information retrieval
82
star
8

Spyglass

Simple search results with Solr and EmberJS
JavaScript
58
star
9

solr-to-es

Migrate a Solr node to an Elasticsearch index.
Python
54
star
10

lucene-query-example

Educational Examle of a custom Lucene Query & Scorer
Java
48
star
11

solr_nginx

Starter Reverse Proxy Configuration for Solr
47
star
12

RankyMcRankFace

Hardened Fork of Ranklib learning to rank library
Java
44
star
13

SemanticSearchInNumpy

XSLT
44
star
14

hangry

Vector search in Lucene based search attempting to use just the existing Lucene data structures (experimental)
Java
43
star
15

trireme

Migration tool providing support for Apache Cassandra, DataStax Enterprise Cassandra, & DataStax Enterprise Solr.
Python
37
star
16

elastic-graph-recommender

Building recommenders with Elastic Graph!
JavaScript
37
star
17

elasticsearch-ltr-demo

This demo uses data from TheMovieDB (TMDB) to demonstrate using Ranklib learning to rank models with Elasticsearch.
HTML
36
star
18

lazy-semantic-indexing

Elasticsearch Latent Semantic Indexing experimentation
Python
33
star
19

pdf-discovery-demo

Demonstration of searching PDF document with Solr, Tika, and Tesseract
JavaScript
30
star
20

match-query-parser

Search a single field with different query time analyzers in Solr
Java
25
star
21

splainer-search

Angular JS Solr and Elasticsearch and OpenSearch Diagnostic Search Services
JavaScript
25
star
22

tmdb_dump

Dump TheMovieDB
Python
23
star
23

es-tmdb

Elasticsearch TMDB examples
Python
21
star
24

solr-tmdb

TheMovieDB in Solr
Python
19
star
25

StackExchangeSolrIndexing

AutoTaxonomyExtractionAndTagging
XML
18
star
26

skipchunk

Extracts a latent knowledge graph from text and index/query it in elasticsearch or solr
Python
18
star
27

cfn-solr

Cloud formation script for solr servers
Shell
16
star
28

search-metrics

Python functions for popular relevance metrics (ndcg, err, etc)
Python
15
star
29

solr_angular_demo

A little search widget for instant Solr search with angular
JavaScript
15
star
30

lucene-bm25f

BM25F demo with lucene using BlendedTermQuery and a custom similarity
Java
15
star
31

bearded-wookie

An experiment in visualizing your Solr index via term counts, document counts, and memory usage per field and data type.
CSS
15
star
32

elasticsearch-image-search

Stupid Experiments in Elasticsearch Image Search
Jupyter Notebook
14
star
33

ubi

User Behavior Insights standard schema
13
star
34

solr-movielens-recommender

Movielens collaborative filtering with Solr streaming expression
Python
11
star
35

grand_central

Docker & Kubernetes deployment system for dynamic environments.
Java
11
star
36

agent_q

Headless agent for test driven relevancy with Quepid.com
Ruby
10
star
37

ltr-synth-judg

Experiments in creating synthetic training data for learning to rank
Python
9
star
38

payload-component

Solr component that surfaces payloads for matching terms
Java
9
star
39

goRank

click tracking for creating judgement lists for search-y stuff
Go
8
star
40

puppet-solr

Puppet module for installing solr with a stand alone jetty server
Shell
7
star
41

SolrSwan

SolrSwan is a query parser and highlighter for Solr that accepts proximity and Boolean queries.
Java
7
star
42

semantic-search-course

Semantic Search Course, Originally delivered at Code4Lib
Python
7
star
43

Sample-Spark-Project

Sample Spark project with Scala and SBT
Scala
7
star
44

solr_dump

Dump Solr docs to file; Write dumped docs to a Solr
Python
7
star
45

lucene_codec_hello_world

Starting point and instructions on developing a Lucene Codec
Java
7
star
46

solr-docker

Sample Dockerfiles for running Solr in a container
6
star
47

o19s-lambda

AWS Lambda Functions to make your life easier.
JavaScript
6
star
48

StackExchangeElasticSearch

Playing with ElasticSearch and the SciFi Stackexchange Dataset
Python
6
star
49

highlighting-pdf-viewer

A component (written in Vue) that supports highlighting of words in the PDF document.
Vue
6
star
50

opensearch-ubi

OpenSearch plugin for User Behavior Insights
Java
6
star
51

elasticsearch-vagrant

An ubuntu 14.04 vagrant box running Elasticsearch
Shell
5
star
52

jackhanna

Simple CLI for Zookeeper
Java
5
star
53

tlre-nlp

Materials for "Think Like A Relevance Engineer - NLP" Training
Jupyter Notebook
5
star
54

keel

This gem provides a few easy to run rake tasks to deploy your Rails application to a Kubernetes cluster.
Ruby
5
star
55

bad-libs

📝 Automatically converts any book into a Mad-Libs style game of silliness using spaCy. Free Charles Dickens included!
Jupyter Notebook
4
star
56

elasticsearch-query-builder-example

Basic Elasticsearch Query Builder Plugin
Java
4
star
57

natural-language-search

Colaboratory notebooks for OSC's Natural Language Search training
Jupyter Notebook
4
star
58

word2vec-experiments

Some experimentation with word2vec
Jupyter Notebook
3
star
59

trec-news-index

Index for the TREC Washington Post corpus
Jupyter Notebook
3
star
60

twittalytics

Twitter Analytics with Cassandra
Python
3
star
61

solr-monitor

Java
2
star
62

search-viz

Various experiments demonstrating pairing realtime visualizations with search results.
JavaScript
2
star
63

tm-import

Importing public domain Trademark XML from Google
Go
2
star
64

elasticsearch-heatmap

Java
2
star
65

o19s-blog-ltr

Using the Elasticsearch LTR demo w/ some hand-created judgments
Python
2
star
66

JodaTimeCodecs

A collection of Cassandra TypeCodecs for serializing and deserializing Joda Time objects.
Java
2
star
67

Spark-Cassandra-Demo

Demo code for loading data into Cassandra and Solr with Spark.
Java
2
star
68

trec-podcasts-index

Index Spotify's 100k podcasts dataset into Elasticsearch
Python
2
star
69

ispy_component

Relevance debugging component for Solr
Java
2
star
70

quepid-jupyterlite

Jupyter notebooks to help with search relevancy measurements, optimized for Quepid.
Jupyter Notebook
2
star
71

clustering-lowes-grouts

Code to support a blog post about extracting tags from Lowes.com for clustering unsanded grout search results
JavaScript
2
star
72

visualizing-signals

A Practical Introduction to Exploring and Visualizing E-Commerce Search Signal Data
Shell
2
star
73

solr-query-parser-demo

A "surround"-like and capitalization custom query parsers demo
Java
2
star
74

user-behavior-insights-elasticsearch

User Behavior Insights (UBI) plugin for Elasticsearch
Java
2
star
75

metric-plots

Plots for search metrics nDCG and ERR
JavaScript
1
star
76

jupyter-blogs

Drafts of Doug's Jupyter Notebook Blog Posts
Python
1
star
77

os-tmdb

TLRE OpenSearch
Python
1
star
78

ndoch-trademark-challenge

Applications built for the National Day of Civic Hacking's USPTO Trademarks Challenge
Ruby
1
star
79

movielens-judgments

experiments using movielens genome tags as an experimental ltr training set
Python
1
star
80

training_coms

R scripts to manage bulk training communications and certificate generation
R
1
star
81

jarjar

Joint Analysis Review of Judgements And Raters
Jupyter Notebook
1
star
82

puppet-modules

puppet modules for o19s
Puppet
1
star
83

thats-trackable

Running app for XC team.
Ruby
1
star
84

ggoodggraphics

The grammer of graphics is powerful and now in Python thanks for `plotnine`!
Jupyter Notebook
1
star