• This repository has been archived on 12/Jan/2019
  • Stars
    star
    757
  • Rank 59,453 (Top 2 %)
  • Language
    JavaScript
  • Created over 10 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A pluggable integration with ElasticSearch to provide advanced content searches in Firebase.

Status: Archived

This repository has been archived and is no longer maintained.

status: inactive

Flashlight

A pluggable integration with ElasticSearch to provide advanced content searches in Firebase.

This script can:

  • monitor multiple Firebase paths and index data in real time
  • communicates with client completely via Firebase (client pushes search terms to search/request and reads results from search/result)
  • clean up old, outdated requests

Getting Started

  • Install and run ElasticSearch or add Bonsai service via Heroku
  • git clone https://github.com/firebase/flashlight
  • npm install
  • edit config.js (see comments at the top, you must set FB_URL and FB_SERVICEACCOUNT at a minimum)
  • node app.js (run the app)

Check out the recommended security rules in example/seed/security_rules.json. See example/README.md to seed and run an example client app.

If you experience errors like {"error":"IndexMissingException[[firebase] missing]","status":404}, you may need to manually create the index referenced in each path:

curl -X POST http://localhost:9200/firebase

To read more about setting up a Firebase service account and configuring FB_SERVICEACCOUNT, click here.

Client Implementations

Read example/index.html and example/example.js for a client implementation. It works like this:

  • Push an object to /search/request which has the following keys: index, type, and q (or body for advanced queries)
  • Listen on /search/response for the reply from the server

The body object can be any valid ElasticSearch DSL structure (see Building ElasticSearch Queries).

Deploy to Heroku

  • cd flashlight
  • heroku login
  • heroku create (add heroku to project)
  • heroku addons:add bonsai (install bonsai)
  • heroku config (check bonsai instance info and copy your new BONSAI_URL - you will need it later)
  • heroku config:set FB_NAME=<instance> FB_TOKEN="<token>" (declare environment variables)
  • git add config.js (update)
  • git commit -m "configure bonsai"
  • git push heroku master (deploy to heroku)
  • heroku ps:scale worker=1 (start dyno worker)

Setup Initial Index with Bonsai

After you've deployed to Heroku, you need to create your initial index name to prevent IndexMissingException error from Bonsai. Create an index called "firebase" via curl using the BONSAI_URL that you copied during Heroku deployment.

Migration

0.2.0 -> 0.3.0

Flashlight now returns the direct output of ElasticSearch, instead of just returning the hits part. This change is required to support aggregations and include richer information. You must change how you read the reponse accordingly. You can see example responses of Flashlight below:

Before, in 0.2.0

"total" : 1000,
"max_score" : null,
"hits" : [
  ..
]

After, in 0.3.0

{
  "took" : 63,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : null,
    "hits" : [
      ..
    ]
  },
  "aggregations" : {
    ..
  }
}

Advanced Topics

Parsing and filtering indexed data

The paths specified in config.js can include the special filter and parse functions to manipulate the contents of the index. For example, if I had a messaging app, but I didn't want to index any system-generated messages, I could add the following filter to my messages path:

filter: function(data) { return data.name !== 'system'; }

Here, data represents the JSON snapshot obtained from the database. If this method does not return true, that record will not be indexed. Note that the filter method is applied before parse.

If I want to remove or alter data getting indexed, that is done using the parse function. For example, assume I wanted to index user records, but remove any private information from the index. I could add a parse function to do this:

parse: function(data) {
   return {
      first_name: data.first_name,
      last_name: data.last_name,
      birthday: new Date(data.birthday_as_number).toISOString()
   };
}

Building ElasticSearch Queries

The full ElasticSearch API is supported. Check out this great tutorial on querying ElasticSearch. And be sure to read the ElasticSearch API Reference.

Example: Simple text search

 {
   "q": "foo*"
 }

Example: Paginate

You can control the number of matches (defaults to 10) and initial offset for paginating search results:

 {
   "from" : 0, 
   "size" : 50, 
   "body": {
     "query": {
        "match": {
           "_all": "foo"
        }
     }
   }
 }; 

Example: Search for multiple tags or categories

 {
   "body": {
     "query": {
       { "tag": [ "foo", "bar" ] }
     }
   }
 }

read more

Example: Search only specific fields

 {
   "body": {
     "query": {
       "match": {
         "field":  "foo",
       }
     }
   }
 }

Example: Give more weight to specific fields

 {
   "body": {
     "query": {
       "multi_match": {
         "query":  "foo",
         "type":   "most_fields", 
         "fields": [ 
            "important_field^10", // adding ^10 makes this field relatively more important 
            "trivial_field" 
         ]
       }
     }
   }
 }

read more

Helpful section of ES docs

Search lite (simple text searches with q) Finding exact values Sorting and relevance Partial matching Wildcards and regexp Proximity matching Dealing with human language

Operating at massive scale

Is Flashlight designed to work at millions or requests per second? No. It's designed to be a template for implementing your production services. Some assembly required.

Here are a couple quick optimizations you can make to improve scale:

  • Separate the indexing worker and the query worker (this could be as simple as creating two Flashlight workers, opening app.js in each, and commenting out SearchQueue.init() or PathMonitor.process() respectively.
  • When your service restarts, all data is re-indexed. To prevent this, you can use refBuilder as described in the next section.
  • With a bit of work, both PathMonitor and SearchQueue could be adapted to function as a Service Worker for firebase-queue,
    allowing multiple workers and potentially hundreds of thousands of writes per second (with minor degredation and no losses at even higher throughput).

Use refBuilder to improve indexing efficiency

In config.js, each entry in paths can be assigned a refBuilder function. This can construct a query for determining which records get indexed.

This can be utilized to improve efficiency by preventing all data from being re-indexed any time the Flashlight service is restarted, and generally by preventing a large backlog from being read into memory at once.

For example, if I were indexing chat messages, and they had a timestamp field, I could use the following to never look back more than a day during a server restart:

exports.paths = [
   {
      path  : "chat/messages",
      index : "firebase",
      type  : "message",
      fields: ['message_body', 'tags'],
      refBuilder: function(ref, path) {
         return ref.orderByChild('timestamp').startAt(Date.now());
      }
   }
];

Loading paths to index from the database instead of config file

Paths to be indexed can be loaded dynamically from the database by providing a path string instead of the paths array. For example, the paths given in config.example.js could be replaced with dynamic_paths and then those paths could be stored in the database, similar to this.

Any updates to the database paths are handled by Flashlight (new paths are indexed when they are added, old paths stop being indexed when they are removed).

Unfortunately, since JSON data stored in Firebase can't contain functions, the filter, parser, and refBuilder options can't be used with this approach.

Support

Submit questions or bugs using the issue tracker.

For Firebase-releated questions, try the mailing list.

License

MIT LICENSE Copyright © 2013 Firebase [email protected]

More Repositories

1

code-prettify

An embeddable script that makes source-code snippets in HTML prettier.
JavaScript
5,768
star
2

chromedeveditor

Chrome Dev Editor is a developer tool for building apps on the Chrome platform - Chrome Apps and Web Apps, in JavaScript or Dart. (NO LONGER IN ACTIVE DEVELOPMENT)
Dart
2,924
star
3

android-Camera2Basic

Migrated:
Java
2,863
star
4

android-ConstraintLayoutExamples

Migrated:
Java
2,571
star
5

firebase-jobdispatcher-android

DEPRECATED please see the README.md below for details.
Java
1,796
star
6

vrview

Library for embedding immersive media into traditional websites.
JavaScript
1,709
star
7

tiger

Java
1,645
star
8

flipjs

A helper library for doing FLIP animations.
JavaScript
1,409
star
9

android-PictureInPicture

Migrated:
Java
1,393
star
10

android-FingerprintDialog

Migrated:
Java
1,375
star
11

observe-js

A library for observing Arrays, Objects and PathValues
JavaScript
1,357
star
12

PyDrive

Google Drive API Python wrapper library
Python
1,304
star
13

js-marker-clusterer

A marker clustering library for the Google Maps JavaScript API v3.
JavaScript
1,279
star
14

android-RuntimePermissions

This sample has been deprecated/archived. Check this repo for related samples:
Java
1,263
star
15

android-Camera2Video

Migrated:
Java
1,201
star
16

chromium-webview-samples

Useful examples for Developing apps with the Chromium based WebView
Java
1,173
star
17

androidtv-Leanback

Migrated:
Java
1,148
star
18

caja

Caja is a tool for safely embedding third party HTML, CSS and JavaScript in your website.
Java
1,126
star
19

android-ui-toolkit-demos

Migrated:
Java
1,118
star
20

android-ScreenCapture

Migrated:
Java
1,021
star
21

android-BluetoothChat

Migrated:
Java
986
star
22

android-BluetoothLeGatt

Migrated:
Java
911
star
23

sample-media-pwa

A sample video-on-demand media Progressive Web App
JavaScript
889
star
24

ChromeWebLab

The Chrome Web Lab for Makers, Hackers and everyone
JavaScript
877
star
25

big-rig

A proof-of-concept Performance Dashboard, CLI and Node module
CSS
857
star
26

android-instant-apps

Migrated:
Java
847
star
27

cloud-functions-emulator

A local emulator for deploying, running, and debugging Google Cloud Functions.
JavaScript
827
star
28

guitar-tuner

A web-based guitar tuner
JavaScript
822
star
29

android-JobScheduler

This sample has been deprecated/archived. Check this repo for related samples:
Java
773
star
30

friendlypix

FriendlyPix is a cross-platform Firebase example app
726
star
31

voice-memos

A Progressive Web App for recording and playing back voice memos.
JavaScript
724
star
32

android-audio-high-performance

We now recommend you use the Oboe libraries:
C++
715
star
33

android-EmojiCompat

Migrated:
Java
707
star
34

android-RecyclerView

Migrated:
Java
675
star
35

ios-swift-chat-example

FireChat implemented in Swift!
Objective-C
673
star
36

leanback-showcase

Migrated:
Java
639
star
37

android-transition-examples

Migrated:
581
star
38

geofire

Realtime location queries with Firebase
569
star
39

drive-music-player

Fully client side Music Player for Google Drive
JavaScript
566
star
40

android-viewpager2

Migrated:
552
star
41

science-journal-ios

Use the sensors in your mobile devices to perform science experiments. Science doesn’t just happen in the classroom or lab—tools like Science Journal let you see how the world works with just your phone.
Swift
532
star
42

topeka

quiz app
HTML
532
star
43

android-PdfRendererBasic

Migrated:
Java
522
star
44

science-journal

Use the sensors in your mobile devices to perform science experiments. Science doesn’t just happen in the classroom or lab—tools like Science Journal let you see how the world works with just your phone.
Java
508
star
45

tango-examples-java

Example projects for Project Tango [deprecated] Java API
Java
500
star
46

AndroidChat

Demonstrates using the Firebase Android SDK to back a ListView.
Java
499
star
47

android-nearby

Migrated:
Java
494
star
48

android-play-places

Deprecated:
Java
479
star
49

tango-examples-unity

Project Tango [deprecated] UnitySDK Example Projects
C#
475
star
50

easygoogle

Simple wrapper library for Google APIs
Java
471
star
51

firefeed

JavaScript
461
star
52

android-dynamic-features

Migrated:
Kotlin
460
star
53

android-MediaBrowserService

This sample is deprecated.
Java
457
star
54

graphd

The Metaweb graph repository server
C
445
star
55

drive-zipextractor

Extract (decompress) ZIP files into Google Drive using the Google Drive API
JavaScript
435
star
56

android-AutofillFramework

Migrated:
Java
432
star
57

webplatform-samples

HTML5 Samples/Demos
430
star
58

cloud-functions-go

Unofficial Native Go Runtime for Google Cloud Functions
Go
427
star
59

android-unsplash

Deprecated:
424
star
60

android-MultiWindowPlayground

Migrated:
Java
418
star
61

appengine-flask-skeleton

A skeleton for creating Python applications using the Flask framework on App Engine
Python
416
star
62

angularfire-seed

Seed project for AngularFire apps
JavaScript
412
star
63

node-big-rig

A CLI version of Big Rig
JavaScript
410
star
64

firebase-dart

Dart wrapper for Firebase
Dart
409
star
65

android-Camera2Raw

Migrated:
Java
386
star
66

js-store-locator

A library for easily building store-locator-type applications using the Google Maps JavaScript API v3
JavaScript
380
star
67

ADBPlugin

Google Chrome Extension with ADB Daemon
C++
372
star
68

android-fit

Migrated:
Java
372
star
69

android-NotificationChannels

This sample has been deprecated/archived. Check this repo for related samples:
Java
369
star
70

appengine-php-wordpress-starter-project

Starter project for running WordPress on Google Cloud Platform
PHP
368
star
71

android-ActivitySceneTransitionBasic

Migrated:
368
star
72

android-text

Migrated:
Kotlin
363
star
73

android-XYZTouristAttractions

Migrated:
356
star
74

firebase-angular-starter-pack

A Firebase + AngularJS Starter Pack
JavaScript
348
star
75

android-OurStreets

Migrated:
345
star
76

tango-examples-c

JNI example projects for Project Tango [deprecated] C-API
C++
334
star
77

simian

Simian is an enterprise-class Mac OS X software deployment solution. Google App Engine hosted server, with a client powered by the Munki open-source project.
Python
334
star
78

OpenInChrome

Open in Chrome
Objective-C
333
star
79

android-DownloadableFonts

Migrated:
Java
319
star
80

friendlypix-web

FriendlyPix is a cross-platform Firebase example app - This is the web version
JavaScript
312
star
81

pywebsocket

WebSocket server and extension for Apache HTTP Server for testing
Python
310
star
82

android-DarkTheme

migrated:
Java
302
star
83

firereader

Firereader: A feed reader built with Firebase and AngularJS
JavaScript
301
star
84

android-WatchFace

Migrated:
300
star
85

android-MediaRecorder

Migrated:
Java
299
star
86

backbonefire

Backbone bindings for Firebase
JavaScript
291
star
87

TemplateBinding

TemplateBinding Prolyfill
JavaScript
291
star
88

firebase-login-demo-android

Java
280
star
89

Firebase-Unity

Objective-C
279
star
90

seed-element

Polymer element boilerplate
HTML
278
star
91

firebase-util

An experimental toolset for Firebase
JavaScript
276
star
92

android-NavigationDrawer

This sample has been deprecated/archived. Check this repo for related samples:
Java
276
star
93

android-DisplayingBitmaps

Migrated:
Java
272
star
94

wReader-app

RSS Reader written using AngularJS
JavaScript
271
star
95

gcm-playground

CSS
268
star
96

android-CardView

Migrated:
Java
265
star
97

soundstagevr

C#
263
star
98

ShadowDOM

ShadowDOM Polyfill
JavaScript
263
star
99

android-credentials

Migrated:
Java
260
star
100

gamebuilder

Game Builder is an application that allows users to create games with little or no coding experience.
C#
253
star