• Stars
    star
    611
  • Rank 70,499 (Top 2 %)
  • Language
    TypeScript
  • License
    Other
  • Created over 2 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Lightweight and performant Lucene-like parser, serializer and search engine.

liqe

Travis build status Coveralls NPM version Canonical Code Style Twitter Follow

Lightweight and performant Lucene-like parser, serializer and search engine.

Motivation

Originally built Liqe to enable Roarr log filtering via cli. I have since been polishing this project as a hobby/intellectual exercise. I've seen it being adopted by various CLI and web applications that require advanced search. To my knowledge, it is currently the most complete Lucene-like syntax parser and serializer in JavaScript, as well as a compatible in-memory search engine.

Liqe use cases include:

  • parsing search queries
  • serializing parsed queries
  • searching JSON documents using the Liqe query language (LQL)

Note that the Liqe AST is treated as a public API, i.e., one could implement their own search mechanism that uses Liqe query language (LQL).

Usage

import {
  filter,
  highlight,
  parse,
  test,
} from 'liqe';

const persons = [
  {
    height: 180,
    name: 'John Morton',
  },
  {
    height: 175,
    name: 'David Barker',
  },
  {
    height: 170,
    name: 'Thomas Castro',
  },
];

Filter a collection:

filter(parse('height:>170'), persons);
// [
//   {
//     height: 180,
//     name: 'John Morton',
//   },
//   {
//     height: 175,
//     name: 'David Barker',
//   },
// ]

Test a single object:

test(parse('name:John'), persons[0]);
// true
test(parse('name:David'), persons[0]);
// false

Highlight matching fields and substrings:

test(highlight('name:john'), persons[0]);
// [
//   {
//     path: 'name',
//     query: /(John)/,
//   }
// ]
test(highlight('height:180'), persons[0]);
// [
//   {
//     path: 'height',
//   }
// ]

Query Syntax

Liqe uses Liqe Query Language (LQL), which is heavily inspired by Lucene but extends it in various ways that allow a more powerful search experience.

Liqe syntax cheat sheet

# search for "foo" term anywhere in the document (case insensitive)
foo

# search for "foo" term anywhere in the document (case sensitive)
'foo'
"foo"

# search for "foo" term in `name` field
name:foo

# search for "foo" term in `full name` field
'full name':foo
"full name":foo

# search for "foo" term in `first` field, member of `name`, i.e.
# matches {name: {first: 'foo'}}
name.first:foo

# search using regex
name:/foo/
name:/foo/o

# search using wildcard
name:foo*bar

# boolean search
member:true
member:false

# null search
member:null

# search for age =, >, >=, <, <=
height:=100
height:>100
height:>=100
height:<100
height:<=100

# search for height in range (inclusive, exclusive)
height:[100 TO 200]
height:{100 TO 200}

# boolean operators
name:foo AND height:=100
name:foo OR name:bar

# unary operators
NOT foo
-foo
NOT foo:bar
-foo:bar
name:foo AND NOT (bio:bar OR bio:baz)

# implicit AND boolean operator
name:foo height:=100

# grouping
name:foo AND (bio:bar OR bio:baz)

Keyword matching

Search for word "foo" in any field (case insensitive).

foo

Search for word "foo" in the name field.

name:foo

Search for name field values matching /foo/i regex.

name:/foo/i

Search for name field values matching f*o wildcard pattern.

name:f*o

Search for phrase "foo bar" in the name field (case sensitive).

name:"foo bar"

Number matching

Search for value equal to 100 in the height field.

height:=100

Search for value greater than 100 in the height field.

height:>100

Search for value greater than or equal to 100 in the height field.

height:>=100

Range matching

Search for value greater or equal to 100 and lower or equal to 200 in the height field.

height:[100 TO 200]

Search for value greater than 100 and lower than 200 in the height field.

height:{100 TO 200}

Wildcard matching

Search for any word that starts with "foo" in the name field.

name:foo*

Search for any word that starts with "foo" and ends with bar in the name field.

name:foo*bar

Boolean operators

Search for phrase "foo bar" in the name field AND the phrase "quick fox" in the bio field.

name:"foo bar" AND bio:"quick fox"

Search for either the phrase "foo bar" in the name field AND the phrase "quick fox" in the bio field, or the word "fox" in the name field.

(name:"foo bar" AND bio:"quick fox") OR name:fox

Serializer

Serializer allows to convert Liqe tokens back to the original search query.

import {
  parse,
  serialize,
} from 'liqe';

const tokens = parse('foo:bar');

// {
//   expression: {
//     location: {
//       start: 4,
//     },
//     quoted: false,
//     type: 'LiteralExpression',
//     value: 'bar',
//   },
//   field: {
//     location: {
//       start: 0,
//     },
//     name: 'foo',
//     path: ['foo'],
//     quoted: false,
//     type: 'Field',
//   },
//   location: {
//     start: 0,
//   },
//   operator: {
//     location: {
//       start: 3,
//     },
//     operator: ':',
//     type: 'ComparisonOperator',
//   },
//   type: 'Tag',
// }

serialize(tokens);
// 'foo:bar'

AST

import {
  type BooleanOperatorToken,
  type ComparisonOperatorToken,
  type EmptyExpression,
  type FieldToken,
  type ImplicitBooleanOperatorToken,
  type ImplicitFieldToken,
  type LiteralExpressionToken,
  type LogicalExpressionToken,
  type RangeExpressionToken,
  type RegexExpressionToken,
  type TagToken,
  type UnaryOperatorToken,
} from 'liqe';

There are 11 AST tokens that describe a parsed Liqe query.

If you are building a serializer, then you must implement all of them for the complete coverage of all possible query inputs. Refer to the built-in serializer for an example.

Utilities

import {
  isSafeUnquotedExpression,
} from 'liqe';

/**
 * Determines if an expression requires quotes.
 * Use this if you need to programmatically manipulate the AST
 * before using a serializer to convert the query back to text.
 */
isSafeUnquotedExpression(expression: string): boolean;

Compatibility with Lucene

The following Lucene abilities are not supported:

Recipes

Handling syntax errors

In case of a syntax error, Liqe throws SyntaxError.

import {
  parse,
  SyntaxError,
} from 'liqe';

try {
  parse('foo bar');
} catch (error) {
  if (error instanceof SyntaxError) {
    console.error({
      // Syntax error at line 1 column 5
      message: error.message,
      // 4
      offset: error.offset,
      // 1
      offset: error.line,
      // 5
      offset: error.column,
    });
  } else {
    throw error;
  }
}

Highlighting matches

Consider using highlight-words package to highlight Liqe matches.

Development

Compiling Parser

If you are going to modify parser, then use npm run watch to run compiler in watch mode.

Benchmarking Changes

Before making any changes, capture the current benchmark on your machine using npm run benchmark. Run benchmark again after making any changes. Before committing changes, ensure that performance is not negatively impacted.

Tutorials

More Repositories

1

react-css-modules

Seamless mapping of class names to CSS modules inside of React components.
JavaScript
5,232
star
2

slonik

A Node.js PostgreSQL client with runtime and build time type safety, and composable SQL.
TypeScript
4,356
star
3

swing

A swipeable cards interface. The swipe-left/swipe-right for yes/no input. As seen in apps like Jelly and Tinder.
JavaScript
2,618
star
4

babel-plugin-react-css-modules

Transforms styleName to className using compile time CSS module resolution.
JavaScript
2,044
star
5

redux-immutable

redux-immutable is used to create an equivalent function of Redux combineReducers that works with Immutable.js state.
TypeScript
1,880
star
6

eslint-plugin-flowtype

Flow type linting rules for ESLint.
JavaScript
1,078
star
7

prepack-webpack-plugin

A webpack plugin for prepack.
JavaScript
1,041
star
8

eslint-plugin-jsdoc

JSDoc specific linting rules for ESLint.
JavaScript
1,024
star
9

roarr

JSON logger for Node.js and browser.
TypeScript
1,024
star
10

turbowatch

Extremely fast file change detector and task orchestrator for Node.js.
TypeScript
901
star
11

table

Formats data into a string table.
TypeScript
871
star
12

usus

Webpage pre-rendering service. ⚑️
JavaScript
805
star
13

flow-runtime

A runtime type system for JavaScript with full Flow compatibility.
JavaScript
802
star
14

surgeon

Declarative DOM extraction expression evaluator. πŸ‘¨β€βš•οΈ
JavaScript
693
star
15

eslint-config-canonical

The most comprehensive ES code style guide.
JavaScript
536
star
16

write-file-webpack-plugin

Forces webpack-dev-server to write bundle files to the file system.
JavaScript
528
star
17

lightship

Abstracts readiness, liveness and startup checks and graceful shutdown of Node.js services running in Kubernetes.
TypeScript
514
star
18

gitdown

GitHub markdown preprocessor.
JavaScript
445
star
19

xhprof.io

GUI to analyze the profiling data collected using XHProf – A Hierarchical Profiler for PHP.
PHP
429
star
20

contents

Table of contents generator.
JavaScript
416
star
21

brim

View (minimal-ui) manager for iOS 8.
JavaScript
391
star
22

global-agent

Global HTTP/HTTPS proxy agent configurable using environment variables.
TypeScript
341
star
23

youtube-player

YouTube iframe API abstraction.
JavaScript
340
star
24

react-aux

A self-eradicating component for rendering multiple elements.
JavaScript
328
star
25

http-terminator

Gracefully terminates HTTP(S) server.
TypeScript
318
star
26

isomorphic-webpack

Abstracts universal consumption of application code base using webpack.
JavaScript
291
star
27

scream

Dynamic viewport management for mobile. Manage viewport in different states of device orientation. Scale document to fit viewport. Calculate the dimensions of the "minimal" iOS 8 view relative to your viewport width.
JavaScript
289
star
28

create-index

Creates ES6 ./index.js file in target directories that imports and exports all sibling files and directories.
JavaScript
279
star
29

graphql-deduplicator

A GraphQL response deduplicator. Removes duplicate entities from the GraphQL response.
JavaScript
278
star
30

gajus.com-blog

The contents of the http://gajus.com/blog/.
JavaScript
226
star
31

wholly

jQuery plugin used to select the entire table row and column in response to mouseenter and mouseleave events. Wholly supports table layouts that utilize colspan and rowspan.
JavaScript
204
star
32

puppeteer-proxy

Proxies Puppeteer Page requests.
JavaScript
195
star
33

canonical-reducer-composition

Spec for Canonical Reducer Composition design pattern.
188
star
34

angular-swing

AngularJS directive for Swing: A swipeable cards interface. The swipe-left/swipe-right for yes/no input. As seen in apps like Jelly and Tinder, and many others.
JavaScript
182
star
35

dindent

HTML indentation library for development and testing.
PHP
177
star
36

babel-plugin-graphql-tag

Compiles GraphQL tagged template strings using graphql-tag.
JavaScript
172
star
37

vlad

Input validation library promoting succinct syntax with extendable validators and multilingual support.
PHP
104
star
38

babel-plugin-log-deprecated

Adds a console.warn statement to the functions annotated with @deprecated tag.
JavaScript
103
star
39

redux-immutable-examples

A complete application showing use of redux-immutable.
JavaScript
103
star
40

eslint-plugin-canonical

ESLint rules for Canonical ruleset.
TypeScript
98
star
41

babel-preset-es2015-webpack

Babel preset for all es2015 plugins except babel-plugin-transform-es2015-modules-commonjs.
JavaScript
97
star
42

scalpel

A CSS selector parser.
JavaScript
95
star
43

eslint-plugin-sql

SQL linting rules for ESLint.
TypeScript
88
star
44

graphql-lazyloader

GraphQL directive that adds Object-level data resolvers.
TypeScript
88
star
45

orientationchangeend

The orientationchangeend event is fired when the orientation of the device has changed and the associated rotation animation has been complete.
JavaScript
78
star
46

bugger

Bugger is a collection of functions for debugging PHP code.
CSS
77
star
47

planton

Database-agnostic task scheduler.
TypeScript
77
star
48

pg-formatter

A PostgreSQL SQL syntax beautifier.
TypeScript
76
star
49

dora

Input generation library for value resolution, data persistence, templates, CSRF and protection from XSS.
CSS
73
star
50

react-css-modules-examples

Usage examples for react-css-modules.
JavaScript
72
star
51

format-graphql

Formats GraphQL schema definition language (SDL) document.
JavaScript
70
star
52

to-string-loader

to-string loader for webpack
JavaScript
64
star
53

interdependent-interactive-histograms

This is a helper function that utilises d3.js and Crossfilter to create interdependent interactive histograms.
JavaScript
60
star
54

extract-email-address

Extracts email address from an arbitrary text input.
JavaScript
59
star
55

babel-plugin-transform-function-composition

Syntactic sugar 🍧🍨🍦 for easy to read function composition. πŸ¦„
JavaScript
58
star
56

preoom

Retrieves & observes Kubernetes Pod resource (CPU, memory) utilisation.
JavaScript
55
star
57

fuss

The Facebook SDK for PHP provides an interface to the Graph API.
PHP
53
star
58

postloader

A scaffolding tool for projects using DataLoader, Flow and PostgreSQL.
JavaScript
51
star
59

moa

MOA implements dynamically generated Active Record database abstraction.
PHP
50
star
60

extract-date

Extracts date from an arbitrary text input.
JavaScript
49
star
61

gitinfo

Gets information about a Git repository.
JavaScript
47
star
62

sister

Foundation for your emitter implementation. 202 reasons to not write your own implementation of event emitter.
JavaScript
45
star
63

react-outside-event

A higher order React component that attaches an event listener for events that occur outside of the component element.
JavaScript
44
star
64

babel-plugin-annotate-console-log

Annotates console.log call expression with information about the invocation context.
JavaScript
42
star
65

react-youtube-player

React component that encapsulates YouTube IFrame Player API and exposes player controls using the component properties.
JavaScript
40
star
66

bundle-dependencies

Generates bundledDependencies package.json value using values of the dependencies property. Updates package.json definition using the generated bundledDependencies value.
JavaScript
39
star
67

facebook-friend-rank

PHP class that can calculate who are the best user's friends. Data accuracy depends on the user activity and granted permissions.
PHP
39
star
68

waitehr

Waits for HTTP response and retries request until the expected response is received.
TypeScript
36
star
69

doll

Extended PDO with inline type hinting, deferred connection support, logging and benchmarking.
PHP
36
star
70

pie-chart

This is a helper function that utilises d3.js to create pie charts.
JavaScript
36
star
71

sguid

Signed Globally Unique Identifier (SGUID) generator.
JavaScript
34
star
72

prepack-loader

A webpack loader for prepack.
JavaScript
31
star
73

slonik-utilities

Utilities for manipulating data in PostgreSQL database using Slonik.
TypeScript
31
star
74

seeql

Real-time SQL profiler.
JavaScript
31
star
75

react-strict-prop-types

A higher order component that raises an error if component is used with an unknown property.
JavaScript
29
star
76

crack-json

Extracts all JSON objects from an arbitrary text document.
JavaScript
29
star
77

paggern

Pattern interpreter for generating random strings.
PHP
28
star
78

cluster-map

Abstracts execution of tasks in parallel using Node.js cluster.
JavaScript
27
star
79

roarr-cli

A CLI program for processing Roarr logs.
TypeScript
26
star
80

database-types

A generic type generator for various databases.
JavaScript
26
star
81

iapetus

Prometheus metrics server.
TypeScript
25
star
82

postgres-bridge

postgres/pg compatibility layer
TypeScript
25
star
83

roarr-browser-log-writer

Roarr log writer for use in a web browser.
TypeScript
23
star
84

require-new

Requires a new module object.
JavaScript
23
star
85

pan

Touch enabled implementation of WHATWG drag and drop mechanism.
JavaScript
23
star
86

tmdb

The Movie Database (TMDb) SDK.
JavaScript
22
star
87

pragmatist

A collection of tasks to standardize builds.
JavaScript
21
star
88

pianola

A declarative function composition and evaluation engine.
JavaScript
20
star
89

eslint-plugin-zod

Zod linting rules for ESLint.
TypeScript
20
star
90

approximate-now

Approximate (fast) current UNIX time.
TypeScript
20
star
91

babel-plugin-transform-export-default-name

Babel plugin that transforms default exports to named exports.
JavaScript
20
star
92

xfetch

A light-weight HTTP client for Node.js.
JavaScript
19
star
93

extract-time

Extracts time from an arbitrary text input.
JavaScript
19
star
94

override-require

Overrides Node.js module resolution logic.
JavaScript
18
star
95

fastify-webpack-hot

A Fastify plugin for serving files emitted by Webpack with Hot Module Replacement (HMR).
TypeScript
18
star
96

babel-plugin-lodash-modularize

Babel plugin that replaces lodash library import statement to individual module imports.
JavaScript
18
star
97

canonical

Canonical code style linter and formatter for JavaScript, SCSS, CSS and JSON.
JavaScript
18
star
98

semantic-url-parser

Extracts content information from known URL patterns.
TypeScript
17
star
99

extract-price

Extracts prices from an arbitrary text input.
JavaScript
17
star
100

async-request

async-request is a wrapper for request that uses ES7 async functions.
JavaScript
16
star