• Stars
    star
    657
  • Rank 65,947 (Top 2 %)
  • Language
    JavaScript
  • Created almost 11 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A search server that can be installed with npm

NPM version NPM downloads MIT License Build Status Join the chat at https://gitter.im/fergiemcdowall/search-index

      ___           ___           ___           ___           ___      
     /\__\         /\  \         /\  \         /\  \         /\__\     
    /::|  |       /::\  \       /::\  \       /::\  \       /:/  /     
   /:|:|  |      /:/\:\  \     /:/\:\  \     /:/\:\  \     /:/__/      
  /:/|:|  |__   /:/  \:\  \   /::\~\:\  \   /:/  \:\  \   /::\  \ ___  
 /:/ |:| /\__\ /:/__/ \:\__\ /:/\:\ \:\__\ /:/__/ \:\__\ /:/\:\  /\__\ 
 \/__|:|/:/  / \:\  \ /:/  / \/_|::\/:/  / \:\  \  \/__/ \/__\:\/:/  / 
     |:/:/  /   \:\  /:/  /     |:|::/  /   \:\  \            \::/  /  
     |::/  /     \:\/:/  /      |:|\/__/     \:\  \           /:/  /   
     /:/  /       \::/  /       |:|  |        \:\__\         /:/  /    
     \/__/         \/__/         \|__|         \/__/         \/__/     

Deploy

npm install norch and then start with norch

or programatically:

require('norch')(options, function(err, norch) {
  // Norch server started on http://localhost:3030 (or the specified host/port)
})

Put stuff in

curl -X POST -d @myData.json http://localhost:3030/add (where myData.json is a newline separated file of JSON objects)

Search for hits (uses search-index's API)

http://localhost:3030/search?q={"query":[{"AND":{"*":["usa"]}}]}

(http://localhost:3030/search returns everything)

Make autosuggest

http://localhost:3030/matcher?q=usa

Export, import, and replicate an index

# create a snapshot on the server (available under /latestSnapshot)
curl -X POST http://localhost:3030/snapshot
# get latest snapshot
curl -X GET http://anotherIndex:3030/latestSnapshot > export.json
# replicate an export file into a new index on another server
curl -X POST -d @export.json http://someOtherServer:3030/import

API

Endpoint Method Response Typical Use Case
/add POST status code Add documents to the index
/availableFields GET stream Discover the name of fields which can be searched in
/buckets GET stream Aggregate documents on ranges of metadata
/categorize GET stream Aggregate documents on single metadata values
/concurrentAdd POST status code For when more than one source is adding documents to the index at the same time
/createSnapshot POST status code Create a snapshot of the index
/delete DELETE status code Remove documents from index
/docCount GET object Counts total document in index
/flush DELETE status code Remove all documents from index
/get GET stream Get documents by ID
/import POST file Import/merge an existing index into this one
/latestSnapshot GET file Download the latest index snapshot
/listSnapshots GET file See list of snapshots
/match GET stream Match by linguistic similarity- autosuggest, autocomplete
/search GET stream Search in the index
/totalHits GET object Show number of hits that a given query returns

The Norch API docs are here. Norch is essentially an http wrapper around search-index.

About Norch

Norch.js is an experimental search engine built with Node.js and search-index featuring, Full text search, Stopword removal, aggregation Matching (Autosuggest), Phrase search, Fielded search, Field weighting, Relevance weighting (tf-idf), Paging (offset and resultset length)

Logging

On Linux and OSX. Install bunyan, tail the log-file and pipe to bunyan.

Install bunyan:

npm install -g bunyan

Tail log-file:

tail -f log-info.log |bunyan

Mailing list: [email protected] - subscribe by sending an email to [email protected]

License

MIT, Copyright (c) 2013-16 Fergus McDowall

More Repositories

1

search-index

A persistent, network resilient, full text search library for the browser and Node.js
JavaScript
1,367
star
2

stopword

A module for node.js and the browser that takes in text and strips it of stopwords
JavaScript
211
star
3

solrstrap

Solrstrap is a Query-Result interface for Solr written in JavaScript, HTML and CSS
JavaScript
86
star
4

reuters-21578-json

An terse and JSONified version of the Reuters 21578 dataset
29
star
5

pumbledb

PumbleDB is a Node.js key-value server that uses LevelDB
JavaScript
21
star
6

term-vector

A node.js module that creates a term vector from a mixed text input. Supports stopword removal and customisable separators.
JavaScript
19
star
7

fergies-inverted-index

Throw JavaScript objects at the index and they will become retrievable by their properties using promises and map-reduce
JavaScript
18
star
8

term-frequency

A simple term frequency library (see https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency_2 )
JavaScript
10
star
9

docproc

A document processing pipeline
JavaScript
9
star
10

norch-document-processor

A library for parsing HTML into any sort of JSON you want
JavaScript
9
star
11

norch-fetch

Fetch pure HTML from a webserver and save it to disk
JavaScript
8
star
12

norch-indexer

An indexing tool for Forage.js
JavaScript
7
star
13

search-index-simple-node-js-example

This is a really basic example of how you get search-index to work in a node.js application
JavaScript
6
star
14

norch-bootstrap

A pluggable front-end for Forage based on Angular and Bootstrap
HTML
6
star
15

search-index-adder

The indexing module for search-index
JavaScript
5
star
16

search-index-indexing-from-pouchdb-example

This is a quick and dirty demo that shows how to feed some documents into a pouchdb instance, and then index the entire database into a search-index.
JavaScript
5
star
17

search-index-indexing-in-browser

This is a super simple demo for indexing documents to a search-index in the browser
JavaScript
5
star
18

search-index-searcher

search in a search-index
JavaScript
4
star
19

term-cluster

return clusters of terms in a given text
JavaScript
4
star
20

virtual-forage

A virtualisation package for Forage. Allows a host machine to run Norch on a guest OS
Ruby
4
star
21

level-out

levelup to stdout
JavaScript
4
star
22

norch-cluster

Norch-cluster is a distributed search engine
JavaScript
3
star
23

search-index-replicate-index-to-browser-example

JavaScript
3
star
24

world-bank-dataset

A JSONified dataset of projects funded by the World Bank.
JavaScript
3
star
25

level-in

A simple utility for writing to a leveldb
Shell
3
star
26

ngraminator

A really small ngram generator
JavaScript
2
star
27

search-index-replicator

The replication module for search-index
JavaScript
2
star
28

norch-client

JavaScript
2
star
29

norch-angular-app

A basic Norch-powered angular app. Use as an example or starting point.
JavaScript
2
star
30

search-index-issue-537

as per https://github.com/fergiemcdowall/search-index/issues/537
JavaScript
1
star
31

webpack-module-example

Example of javascript module compiled with webpack that can then be used with node, frontend bundles, and <script> tags
JavaScript
1
star
32

Oslo-bysykkel

Show the location and availability of Oslo city bikes
Vue
1
star
33

gatsby-starter-netlify-cms

JavaScript
1
star
34

intersect-arrays-to-stream

Takes an arbitrary amount of sorted arrays and returns the intersect as a stream
JavaScript
1
star
35

distribute-array

Distributes one array evenly between a given number of smaller arrays
JavaScript
1
star
36

search-index-deleter

The deletion module for search-index
JavaScript
1
star
37

search-index-matcher

A matcher that generates its content from a search-index
JavaScript
1
star
38

fergies-packager-test

A repo for testing packaging strategies
JavaScript
1
star
39

search-index-getter

A lib for retrieving individual documents from a search-index corpus
JavaScript
1
star