• Stars
    star
    105
  • Rank 328,196 (Top 7 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated about 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A simple proxy service to forward JSON events and transform or filter them along the way.

EEL - A simple Proxy Service for JSON Event Transformation and Forwarding

It's simple - a single JSON event comes in and one (or maybe a few) transformed events get out. Events are arbitrary JSON encoded payloads and they are typically sent around as HTTP POSTs. EEL is stateless and therefore scales easily.

In this example we removed the unwanted "detail" field from the event and adjusted the schema slightly.

Converting JSON payload A from service S1 into JSON payload B for service S2 is a very common operation we perform all the time in service oriented architectures. Here are some good reasons for event transformation:

  1. REDUCTION: Remove everything that's not needed and only pass the data actually required by the downstream service. This can provide performance benefits for the downstream service if processing is expensive and events are bloated.
  2. CANONICALIZATION / CLEANUP: Adjust the structure of the JSON event to validate against a given schema. Typically this can be achieved by rearranging elements in the JSON event, renaming keys and possibly filtering out unwanted subsets of data.
  3. META-TAGGING: Inject additional metadata into the JSON event. Examples are injecting UUIDs or timestamps (for example for log tracing), tagging or labeling events with type fields, calling external auxiliary services and injecting their output into the downstream payload (provided it is JSON).
  4. MAPPING / SUBSTITUTION: Map one ID type to another via look up using an external lookup service.
  5. FILTERING: Filter out unwanted events to reduce load for downstream services.
  6. FANOUT: Forward a single incoming JSON event to multiple downstream services, either by applying the same transformation for all services or by performing different transformations.

You could write each of these transformations in just a few lines of go code, and we often do. The downside is that transformations become engraved in code and cannot be changed easily.

EEL is offering a simple JSON template language combined with a proxy micro service to relay and transform events between upstream and downstream services. The goals of the EEL template language are to be simple and yet powerful. EEL can be the glue in a JSON based service oriented eco-system.

The syntax of the EEL transformation language is inspired by a few of the core concepts from XPath and XSLT applied to JSON.

Installation

go get -u github.com/comcast/eel

Usage

./eel [options...]

Options:

-config  path to config.json (default is ./config-eel/config.json)
-handlers  path to handlers (default is ./config-handlers)
-loglevel  log level (default is "info")
-env  environment name such as qa, prod for logging (default is "default")

No Nonsense

go build -o bin/eel
./bin/starteel.sh

Docker Alternative

Docker 1.13+ verified.

Build a dev image

docker build -t eel:dev .

Run an instance

docker run --rm -p 8080:8080 --name eel-dev eel:dev

The command above will utilize port 8080 of your host. You can change it to any other port via -p ANYOTHERPORT:8080

To pass parameters to eel you can use EEL_PARAMS env variable, e.g.

docker run --rm -e "EEL_PARAMS=-loglevel error" -p 8080:8080 --name eel-dev eel:dev

A Simple Example

Transformation handlers are used to tell EEL if and how to process JSON events it receives and where to forward them to. Each transformation handler is stored in a single JSON file and there may be several such handler files each of them taking care of a certain class of events. See here for more details.

Edit the default transformation handler in config-handlers/tenant1/default.json so it looks like this (there's only a small change needed in the Transformation section).

{
    "Version": "1.0",
    "Name": "Default",
    "Info": "",
    "Active": true,
    "Match": null,
    "IsMatchByExample": false,
    "TerminateOnMatch": true,
    "Transformation": {
        "{{/event}}": "{{/}}"
    },
    "IsTransformationByExample": false,
    "Path": "",
    "Verb": "POST",
    "Endpoint": "http://localhost:8082",
    "HttpHeaders": {
      "X-B3-TraceId": "{{traceid()}}",
      "X-Tenant-Id": "{{tenant()}}"
    }
}

The center piece of the handler configuration is the Transformation section which uses JPath expressions to describe the structural transformations to be performed on the incoming event before forwarding it to the downstream service. In this example we are telling EEL to take the entire payload unmodified and wrap it inside a new element called event.

Compile EEL.

EEL is implemented in go and you will need a golang environment to compile the code. However, you don't need any go coding skills to understand and author EEL templates!

The make file and shell scripts have been tested on Mac and Linux environments.

make all

Launch EEL as a service and start listening to incoming events.

./bin/starteel.sh

Send a test event to EEL.

curl -X POST --data '{ "message" : "hello world!!!" }' http://localhost:8080/v1/events

Output:

{"status":"processed"}

You can check EEL's log output eel.log to see if and how the event is processed. You can also ask for a detailed debug response using the X-Debug header.

curl -X POST -H 'X-Debug: true' --data '{ "message" : "hello world!!!" }' http://localhost:8080/v1/events

Output:

[
	{
		"api": "http",
		"handler": "Default",
		"tenant.id": "tenant1",
		"trace.in.data": {
			"message": "hello world!!!"
		},
		"trace.out.data": {
			"event": {
				"message": "hello world!!!"
			}
		},
		"trace.out.endpoint": "http://localhost:8088",
		"trace.out.headers": {
			"X-B3-TraceId": "20073ee4-d681-4ab5-a973-50c978cd1111",
			"X-Tenant-Id": "tenant1"
		},
		"trace.out.path": "",
		"trace.out.protocol": "http",
		"trace.out.url": "http://localhost:8088",
		"trace.out.verb": "POST",
		"tx.id": "20073ee4-d681-4ab5-a973-50c978cd1111",
		"tx.traceId": "20073ee4-d681-4ab5-a973-50c978cd1111"
	}
]

Or, you can use the /v1/sync/events instead of the /v1/events endpoint to get an immediate response containing the transformed event as it would be forwarded to the downstream service.

curl -X POST --data '{ "message" : "hello world!!!" }' http://localhost:8080/v1/sync/events

Output:

{
	"event": {
		"message": "hello world!!!"
	}
}

To review the current set of active transformation handlers call the health check API.

curl http://localhost:8080/v1/health

Stop EEL.

./bin/stopeel.sh

EEL as Command Line Tool

You can also start experimenting with EEL by using the command line parameters. Example:

./eel -in='{"foo":"bar"}' -tf='{"Foo":"{{/foo}}"}' -istbe=true

More examples can be found here .

Exploring EEL Features

The unit tests are a good starting point to learn more about EEL features and look at some examples in detail.

Each test is contained in its own configuration folder, for example eel/test/data/test01. Each test folder contains a complete set of handler configurations for EEL in the handlers subfolder (usually just one handler), an example input event in.json and one or more expected output events out.json.

The tests can be executed using go test:

cd eel/test
go test -v

Or, you can launch EEL with the handler configurations for a specific test and send the in.json event manually.

./eel -handlers=test/data/test01/handlers > eel.log &
curl -X POST --data @test/data/test01/in.json http://localhost:8080/v1/sync/events
No Name Test Name Description
0 Identity Transformation TestDontTouchEvent Doesn't apply any transformation and forwards everything unchanged.
1 Canonicalize TestCanonicalizeEvent Simple structural changes and array path selectors.
2 External Lookup TestInjectExternalServiceResponse Get JSON data from external service and inject into payload.
3 Transformation By Example TestTransformationByExample Describe transformation using by-example syntax rather than using by-path syntax.
4 Named Transformations TestNamedTransformations Choose from several named transformations based on input data.
5 Conditional Message Generation TestMessageGeneration Assemble message string with ifte() and equals().
6 Handler Matching 1 TestTerminateOnMatchTrue Pick best matching handler and forward single event.
7 Handler Matching 2 TestTerminateOnMatchFalse Pick all matching handlers and forward multiple events.
8 Multi Tenancy TestMultiTenancy Handlers for different tenants or apps.
9 Cascade TestSequentialHandlerCascade Cascade of multiple handlers which will be executed sequentially by using EEL recursively.
10 Java Script For Everything Else TestJavaScript If you really can't avoid it, resort to Java Script.
11 Handler Matching 3 TestMatchByExample Matching handlers using by-example syntax.
12 Convert Headers To Payload TestHeaders Inject HTTP headers from upstream service into JSON event for downstream service.
13 Custom Properties TestCustomProperties Custom properties in handlers for sharing data.
14 Fan Out TestFanOut Send incoming event to several downstream services.
15 Basic String Operations TestStringOps Uppercase, lowercase, substring.
16 Named Transformations 2 TestNamedTransformations2 Perform named transformation on external document.
17 Contains TestContains Check if one JSON event is contained in another.
18 Tenant Id Header TestMultiTenency2 Pass in tenant id as HTTP header.
19 Conditional Message Generation 2 TestMessageGeneration2 Use case() function to simplify conditional string generation.
20 Regex TestRegex Use regex() to evaluate regular expressions.
22 Filter By Path TestFilterByPath Filter event after transformation using by-path syntax.
23 Filter By Example TestFilterByExample Filter event after transformation using by-example syntax.
25 Array Path Selector TestArrayPathSelector Select elements from arrays by index or by path.
27 Iterate Over Array NamedTransformationsAndArrays Iterate over array and apply named transformation.
32 Simple Types NamedTransformationsAndSimpleTypes Apply named transformation to a simple type.
34 Join Join Merge two JSON documents.
41 Partial Matching MatchPartialArrays3 Match event against partial pattern.
46 Filter Cascade FilterCascade Apply multiple filters.
47 Choose From Array ChooseFromArray Choose elements from array by pattern.
48 Named Transformation With Array And Pattern NamedTransformationWithArrayAndPattern Apply named transformation by pattern.
49 Named Transformation With Array And Join NamedTransformationWithArrayAndJoin Apply named transformation with join.
51 Complex Example ComplexExample A real world example.
53 Crush Crush Flatten a deeply nested array.

Further Reading

More Repositories

1

FreeFlow

A layout engine for Android that decouples layouts from the View containers that manage scrolling and view recycling. FreeFlow makes it really easy to create custom layouts and beautiful transition animations as data and layouts change
Java
2,397
star
2

rulio

Rulio
Go
336
star
3

gots

MPEG Transport Stream handling in Go
Go
306
star
4

sirius

A distributed system library for managing application reference data
Scala
298
star
5

cmb

This project is no longer actively supported. It is made available as read-only. A highly available, horizontally scalable queuing and notification service compatible with AWS SQS and SNS
Java
278
star
6

jrugged

A Java libary of robustness design patterns
Java
266
star
7

ip4s

Defines immutable, safe data structures for describing IP addresses, multicast joins, socket addresses and similar IP & network related data types
Scala
224
star
8

mamba

Mamba is a Swift iOS, tvOS and macOS framework to parse, validate and write HTTP Live Streaming (HLS) data.
Swift
178
star
9

kube-yarn

Running YARN on Kubernetes with PetSet controller.
Makefile
166
star
10

k8sh

A simple, easily extensible shell for navigating your kubernetes clusters
Shell
155
star
11

patternlab-edition-node-webpack

The webpack wrapper around patternlab-node core, providing tasks to interact with the core library and move supporting frontend assets.
JavaScript
127
star
12

gaad

GAAD (Go Advanced Audio Decoder)
Go
126
star
13

sheens

Message Machines
Go
119
star
14

Speed-testJS

JavaScript
94
star
15

traffic_control

Traffic Control CDN
92
star
16

Surf-N-Perf

Micro-library for gathering web page performance data
JavaScript
90
star
17

pulsar-client-go

A Go client library for Apache Pulsar
Go
78
star
18

graphvinci

A better schema visualizer for GraphQL APIs
JavaScript
74
star
19

MirrorTool-for-Kafka-Connect

A Kafka Source connector for Kafka Connect
Java
72
star
20

php-legal-licenses

A utility to help generate a file containing information about dependencies including the full license text.
PHP
70
star
21

caption-inspector

Caption Inspector is a reference decoder for Closed Captions (CEA-608 and CEA-708).
C
69
star
22

money

Dapper Style Distributed Tracing Instrumentation Libraries
Scala
67
star
23

compass-csslint

Easily integrate CSS Lint into your projects that use the Compass CSS Framework
Ruby
65
star
24

snowdrift

App to perform testing and validation of firewall rules
Shell
63
star
25

dialyzex

A Mix task for type-checking your Elixir project with dialyzer
Elixir
61
star
26

ssh-to

Easily manage dozens or hundreds of machines via SSH
Shell
58
star
27

ansible-sdkman

An Ansible role that installs, configures, and manages SDKMAN
Python
58
star
28

Bynar

Server remediation as a service
Rust
57
star
29

xGitGuard

AI based Secrets Detection Python Framework
Python
54
star
30

Canticle

Go
50
star
31

scte35-js

A SCTE 35 Parser for JavaScript
HTML
47
star
32

go-leaderelection

GoLea is a Go library that provides the capability for a set of distributed processes to compete for leadership for a shared resource.
Go
45
star
33

Comcast.github.io-archive

The main Open Source portal for Comcast
HTML
38
star
34

scte35-go

Golang implementation of ANSI/SCTE-35
Go
36
star
35

sitemapper-for-js

Generate XML sitemaps for JS Websites (Supports Angular, React)
JavaScript
35
star
36

zucchini

Run your cucumber-jvm tests in parallel across all your devices
Java
34
star
37

Oscar

OSCAR - OpenSource Cablemodem file AssembleR - DOCSIS, PacketCable, DPoE Configuration Editor and API Framework
Java
33
star
38

hlsparserj

Java
32
star
39

weasel

Lightweight license checker.
Go
32
star
40

Infinite-File-Curtailer

Curtail is a utility program that reads stdin and writes to a file bound by size.
C
32
star
41

DahDit

Custom Views for drawing Dotted and Dashed Lines without jumping through hoops.
Kotlin
27
star
42

resourceprovider2

Resource Management API Generator for Android
Kotlin
22
star
43

python-batch-runner

A tiny framework for building batch applications as a collection of tasks in a workflow.
Python
22
star
44

Buildenv-Tool

Go
21
star
45

blueprint

Blueprint is a compact framework for constructing mvp architecture within a scrollable, multi-view-type list UI. It uses the Android RecyclerView library, and currently only supports LinearLayouts
Kotlin
21
star
46

comcast.github.io

The main Open Source portal for Comcast
JavaScript
20
star
47

ProjectGuardRail

AI/ML applications have unique security threats. Project GuardRail is a set of security and privacy requirements that AI/ML applications should meet during their design phase that serve as guardrails against these threats. These requirements help scope the threats such applications must be protected against.
20
star
48

resourceprovider-utils

Samples and Test Utilities Library for the ResourceProvider API Generator Gradle Plugin for Android.
Java
19
star
49

xCOMPASS

This repository hosts a persona based privacy threat modeling solution called Models of Applied Privacy or MAP.
17
star
50

compass-csscss

Easily integrate csscss into your projects that use the Compass CSS Framework
Ruby
16
star
51

go-edgegrid

A Golang Akamai API client and Akamai OPEN EdgeGrid Authentication scheme authentication handler.
Go
16
star
52

cea-extractor

Parsing and display logic for CEA-608 caption data in fragmented MP4 files.
TypeScript
16
star
53

all-digital

Comcast All Digital CSS
SCSS
15
star
54

littlesheens

A Sheens implementation that has a small footprint
C
15
star
55

rdk-on-raspberrypi

Documentation for running RDK profiles ( Video, broadband, Camera ) on Raspberrypi boards
15
star
56

terraform-provider-akamai

An Akamai GTM Terraform provider
Go
14
star
57

Xooie

Important note: this project is no longer actively maintained. Xfinity Xooie - Modular, Modifiable, Responsive, Accessible
JavaScript
14
star
58

Superior-Cache-ANalyzer

A tool for inspecting the contents of Apache Traffic Server caches
Python
14
star
59

ActorServiceRegistry

Scala
14
star
60

connvitals-monitor

A persistent monitor and aggregator of network statistics
Python
13
star
61

RestfulHttpsProxy

Go
13
star
62

Priority-Operation-Processing

A workflow orchestration system where the workflow is scheduled as a unit giving resource priority once selected. Priority queuing and customizable scheduling algorithms. Customer aware for multi-tenant. A JSON DAG based blueprint defines the execution flow. Executes operations within a workflow by spinning up on-demand Kubernetes Pods (dynamic resource allocation)
Java
13
star
63

connvitals

A network analysis and statistics aggregation tool
Python
13
star
64

xml-selector

A jQuery-like interface for working with XML using CSS-style selectors
JavaScript
12
star
65

prombox

Prombox creates a sandbox environment for editing and testing Prometheus configuration on the fly
JavaScript
12
star
66

svn-to-github

Comprehensive Tool for Converting SVN to Git in Bulk
Shell
12
star
67

watchmen-ping-nightmare

A plugin for watchmen that uses nightmare and an electron browser to monitor websites
JavaScript
12
star
68

tsb

A Transitive Source Builder for managing builds across multiple repositories
Go
11
star
69

scte224structs

Utilities for modeling SCTE-224 data in Go
Go
11
star
70

cf-recycle-plugin

Cloud Foundry cli plugin for rolling restart of application instances
Go
10
star
71

redirector

Java
10
star
72

discovery

The Comcast discovery services for the open source community
Java
10
star
73

fishymetrics

Redfish API Prometheus Exporter for monitoring large scale server deployments
Go
10
star
74

libcertifier

With small, space optimized, 90KB libCertifier(), IoT devices (cameras, toasters, sensors ….) can now request and receive unique, compact (650 bytes) digital certificates (x509 v3 compliant).
C
10
star
75

compare-ini-files

Compare an arbitrary number of .ini files based on logical sections and key/value pairs.
PHP
9
star
76

flume2storm

This project is no longer actively maintained. The Flume2Storm repository contains the source code, documentation (wiki) and issue tracking system
Java
9
star
77

cts-mpx

Ruby
8
star
78

ansible-role-pypi

An ansible role for configuring a pypi-server on a systemd centos system.
Python
8
star
79

akamai-slack-reporter

A Slack slash command integration for querying your team's Akamai configuration
JavaScript
8
star
80

rapid-ip-checker

A GPU accelerated tool to compare large lists of IPv4/IPv6 addresses.
Python
8
star
81

css_lint_ruby

Nicholas C. Zakas and Nicole Sullivan's CSS Lint made available as a Ruby gem.
Ruby
8
star
82

akamai-gtm

A CLI to Akamai GTM
Go
8
star
83

DNS-over-HTTPs-translator

Go
7
star
84

tlsrpt_processor

Simple python script to process TLSRPT reports
Python
7
star
85

libstorage

rust storage server helper utilities
Rust
7
star
86

EasyConnect

Reference Development Kit that will help partners and vendors implement EasyConnect in their devices.
Java
7
star
87

dmc-sim

ns3 simulator code for Distributed Monotonic Clocks
C++
7
star
88

jovo-plugin-resume

🔈 A plugin for resuming conversations in the Open Source Voice Layer, Jovo (https://www.jovo.tech)
TypeScript
7
star
89

Porrtal

This project was created to help developers. You can use the platform to build web applications. The project supports both React and Angular development.
TypeScript
7
star
90

cf-zdd-plugin

Go
7
star
91

vesper_legacy

Secure Telephone Identity Management in Session Initiation Protocol
Go
7
star
92

SyntaViz

A visualization interface for analyzing a (very large) corpus of natural-language queries.
Python
7
star
93

hypergard

A JavaScript client for HAL APIs with support for forms
JavaScript
7
star
94

ctex

A Mix task and helpers for running common_test suites
Elixir
7
star
95

plax

A test automation engine for messaging systems
Go
7
star
96

Ravel

Go
7
star
97

pipeclamp

Java
6
star
98

rally-rest-toolkit

Client API library for interacting with the Rally REST API
Go
6
star
99

rdkcryptoapi

Contains Cryptographic APIs used in the RDK Software Stack
C
6
star
100

polaris

Vanilla Web Components for global Header, Footer & Toast notifications for XFINITY Websites
JavaScript
6
star