• Stars
    star
    612
  • Rank 73,287 (Top 2 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 6 years ago
  • Updated 6 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

JSON query and transformation language

JSLT

JSLT is a complete query and transformation language for JSON. The language design is inspired by jq, XPath, and XQuery.

JSLT can be used as:

  • a query language to extract values from JSON (.foo.bar[0]),
  • a filter/check language to test JSON objects (starts-with(.foo.bar[0], "http://")) ,
  • a transformation language to convert between JSON formats.

An example transform:

{
    "time": round(parse-time(.published, "yyyy-MM-dd'T'HH:mm:ssX") * 1000),
    "device_manufacturer": .device.manufacturer,
    "device_model": .device.model,
    "language": .device.acceptLanguage,
    "os_name": .device.osType,
    "os_version": .device.osVersion,
    "platform": .device.platformType,
    "user_properties": {
        "is_logged_in" : boolean(.actor."spt:userId")
    }
}

Demo playground.

Language tutorial.

Function documentation.

More examples.

Javadoc.

API introduction.

Build Status

Quick reference

Operation Explanation
. The context node
.<name> Get value of key "<name>" inside an object
.[<index>] Get value <index> inside an array
.[<from> : <to>] Array slicing
if (<expr>) <expr> else <expr> If test to decide which value to return
let <name> = <expr> Define a variable
$<name> Refer to a variable
[for (<expr>) <expr>] Transform an array
{for (<expr>) <expr> : <expr>} Transform an object
def <name>(<name>, <name>...) <expr> Declare a function
// <anything up to end of line> Comment
{ <key> : <expr> } Object constructor
{ <key> : <expr>, * : . } Specify one key, copy rest of input
5 * 7 + 23.2 Arithmetic operations
7 < 5 Comparators
7 < 5 and .foo == "yes" Boolean operators

Using the library

To include JSLT in your project, depend on:

<dependency>
  <groupId>com.schibsted.spt.data</groupId>
  <artifactId>jslt</artifactId>
  <version>0.1.14</version>
</dependency>

At runtime JSLT depends on Jackson, and nothing else.

To transform one JsonNode into another, do:

import com.schibsted.spt.data.jslt.Parser;
import com.schibsted.spt.data.jslt.Expression;

JsonNode input = ...;
Expression jslt = Parser.compileString(transform);
JsonNode output = jslt.apply(input);

For more alternatives, see the javadoc.

Command-line

To run transforms on the command-line, first build with ./gradlew clean shadowJar. Then you can run with:

java -cp build/libs/*.jar com.schibsted.spt.data.jslt.cli.JSLT transform.jslt input.json

The result is written to standard out.

Extension functions

You can implement your own functions and add them to the language. See the extension function tutorial.

Feedback

If you have questions about how to use JSLT, please ask the question on StackOverflow, with the tag jslt.

If you have problems, feature requests, or think you found a bug, please open an issue.

Status

The language design is not finished, so features may be added. The language as it stands is not likely to change.

The entire language is implemented, and all of the function library. Functions may be added.

The language has been used in production at Schibsted since January 2018, performing about 9 billion transforms per day, and many times more queries.

Building JSLT

To build JSLT as a jar file, run ./gradlew jar.

To build a fat jar with all dependencies included, run ./gradlew shadowJar.

To run the tests: ./gradlew check.

There is a pom.xml file, but Maven is not used for building JSLT, and the file is not intended to work. It's only there to make Github dependency tracking work.

More information

Developing a language for JSON processing: video of talk, slides only.

Running the playground yourself

Anthony Sparks is working on a VM-based implementation in Java

A paper describing (among other things) some of the ways Schibsted uses JSLT.

Visual Studio syntax highlighter for JSLT.

IntelliJ JSLT plugin.

Apache Camel JSLT component.

JSLT is also integrated in Apache NiFi as a processor.

How Willhaben.at uses JSLT with Kafka Connect

IBM Cloud Pak for Business Automation.

Pincette event sourcing framework uses JSLT.

LICENSE

Copyright (c) 2018 Schibsted Marketplaces Products & Technology AS

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

What is missing

Things to be done:

  • Move the tests out into YAML files.
  • Write a proper spec with EBNF and everything.
  • Fix the syntax ambiguity problem with let and def.
  • Implement toString() throughout the object tree, so that it's possible to turn expressions back to strings.
  • Optimizer:
    • Optimize ... and boolean( ... ) by removing boolean().
    • Implement parse tree traversal API.
    • Make sure entire tree is traversed (inside function decls and variables, for example).
    • Complete constant folding. Particularly constant folding for variables would be valuable.
    • Inlining of functions.
    • Eliminate unused variables.
  • Use property-based testing and fuzz testing to harden the parser.

See also the list of ideas.

More Repositories

1

WAAS

Whisper as a Service (GUI and API with queuing for OpenAI Whisper)
JavaScript
1,802
star
2

strongbox

A secret manager for AWS
Java
239
star
3

jsx-pdf

Generate PDFs using JSX! 🎯
JavaScript
124
star
4

tweet_ui

Flutter package to display tweets from a Twitter API JSON (v1 or v2) on Android and iOS. Support for Tweets with 1-4 photos, Video, GIFs, hashtags, mentions, symbols, urls, quoted Tweets and retweets.
Dart
79
star
5

schibsted-grotesk

The Schibsted Grotesk font
Python
57
star
6

mesoscope

Local Mesos cluster running on top of docker for learning/testing purposes
Shell
34
star
7

meeting-concluder

Record audio from a meeting, then transcribe, conclude and send the conclusion and a piece of advice to Slack
Go
29
star
8

artishock

A tool to investigate Dependency Confusion in Artifactory
Java
23
star
9

triathlon

Java
20
star
10

github-actions-self-hosted-cdk

TypeScript
20
star
11

aws-cost-monitoring

AWS Cost Monitoring terraform module
HCL
17
star
12

ai-academy

A repository for the schibsted ai academy
16
star
13

account-sdk-browser

Schibsted Account SDK for browsers
JavaScript
16
star
14

account-sdk-android

⛔️ DEPRECATED Schibsted Account SDK for Android
Kotlin
14
star
15

node-token-introspection

Library to introspect token of service following RFC-7662
JavaScript
13
star
16

mesos2iam

Provide IAM credentials to containers running inside a mesos cluster based on environment variable of the container
Go
9
star
17

tests

The tests we give to people we want to work with
JavaScript
9
star
18

eslint-config-schibsted

eslint configurations from Schibsted engineers
JavaScript
9
star
19

deathstar

Because a little chaos is good for you.
TypeScript
8
star
20

sdk-js

SPiD SDK Client for Javascript
JavaScript
7
star
21

sdk-php

SPiD SDK Client for PHP
PHP
6
star
22

account-sdk-ios

⛔️ DEPRECATED SchibstedAccount SDK for iOS
Swift
6
star
23

spid-tech-docs

Technical documentation for the Schibsted account platform
Clojure
6
star
24

pr_changelog

A script to generate a nice list of changes given two git references.
Ruby
6
star
25

sdk-android

SPiD SDK Client for Android. DEPRECATED: This is deprecated and will not be supported after May 2018. Please refer to https://github.com/schibsted/account-sdk-android instead
Java
6
star
26

next-gen

Next Generation Products
5
star
27

middy-error-handler

Error handler middleware that returns JSON response
JavaScript
5
star
28

krakend-ratelimit

krakend rate limiter
Go
5
star
29

dockermaze-reloaded

It’s year BD5. The Dockerbot is your only hope of survival in an unknown world. Will you be able to fix it in time?
Go
4
star
30

bigip-to-terraform

Transform running f5 bigip configuration to terraform resources
Python
4
star
31

libvmod-hashids

https://hashids.org/ support for Varnish
C
4
star
32

sdk-example

An example demonstrating usage of SDKs
JavaScript
4
star
33

spid-client-java

Lightweight client for the SPiD API
Java
4
star
34

udp-pipe

Reads UDP packages and forwards to different targets
JavaScript
4
star
35

serverless-slack-deploy-notification

Plugin that sends a slack message on deployment start and finish
JavaScript
4
star
36

sdk-java-event-tracking

Java
3
star
37

SPiDSDK

SPiD SDK Client for iOS
Objective-C
3
star
38

sdk-js-event-tracking-contrib

Automatic tracking layer for the js sdk.
JavaScript
3
star
39

sum

Sum, a powerful tool for enhancing your articles with the help of ChatGPT.
TypeScript
3
star
40

depstrive

Multi-tool for missing CI/CD features
JavaScript
3
star
41

googoo

Deploy review apps for your project using Serverless
JavaScript
3
star
42

spid-php-examples

Using the SPiD API: Working examples for PHP
PHP
3
star
43

deathstar-middleware

Node.js middleware for Deathstar, to create simulated outages
TypeScript
3
star
44

spid-clj-examples

Using the SPiD API: Working examples for Clojure
Clojure
3
star
45

gocd-s3-poller

Poll AWS S3 bucket and folder for files, and trigger pipeline in GoCD
Java
3
star
46

overloadimpact

Command line tool and framework for writing and running tests suites for loadimpact.com, with support for custom lua libraries.
Python
3
star
47

schibsted-account-chrome-extension

Let's you transfer the Schibsted Account cookie to your local projects with ease!
JavaScript
3
star
48

pingdom-status-page

A status page to show off your current and historical uptime from Pingdom
JavaScript
2
star
49

krakend-eureka

krakend eureka integration
Go
2
star
50

discord-news-bot

Stay updated with the latest headlines from VG directly in your Discord server!
TypeScript
2
star
51

chef-rest

Chef resource provider to do REST queries in recipes
Ruby
2
star
52

core-sdk-node

Core SDK for Identity and Payment
JavaScript
2
star
53

VG-Nyhets-Quiz

WIP
TypeScript
2
star
54

problematic

Random problem generator
CSS
2
star
55

account-sdk-flutter

Flutter plugin for Schibsted Account SDK
Kotlin
2
star
56

account-sdk-ios-web

Schibsted account SDK for iOS
Swift
2
star
57

smaug

A simple credentials provider for mesos2iam
Go
2
star
58

slack-calendar-topic

Set the topic of a Slack channel to the currently ongoing event in a Google calendar
JavaScript
2
star
59

sls-oncall-twilio

Call routing with Twilio and PagerDuty
JavaScript
2
star
60

ratpack-datadog

Java
2
star
61

ratpack-eureka

Java
2
star
62

krakend-cbreaker

krakend circuit breaker hystrix style implemetation
Go
2
star
63

spid-client-clojure

A very thin Clojure wrapper for the SPID Java SDK
Clojure
2
star
64

copyfield

Update fields in a PostgreSQL database with values from another column
Go
2
star
65

spid-sdk-client-node

Lightweight Node.js client for the SPiD API
JavaScript
1
star
66

spid-java-examples

Examples using the spid-client-java
Java
1
star
67

sls-oncall

Slack bot that fetches user oncall given a schedule in PagerDuty
JavaScript
1
star
68

middy-caching-headers

Add caching headers to the response e.g. Cache-Control, Surrogate-Control
JavaScript
1
star
69

express-api-formatter

Decorates express response with success/error functions
JavaScript
1
star
70

uka-workshop-2019

Kubernetes Workshop at UKA 2019
1
star
71

homebrew-strongbox

Homebrew recipes
Ruby
1
star
72

docker-postgres-tests

Docker images specialised for testing purposes
Go
1
star
73

tjejer-kodar-schibsted

The project for Schibsted's workshop for Tjejer Kodar at October 2018
JavaScript
1
star
74

middy-cors

Adds CORS headers to the response (success and error)
JavaScript
1
star
75

express-json-error-handler

Error handler for body-parser
JavaScript
1
star
76

configure-convox-racks

JavaScript
1
star
77

sdk-java

SPiD SDK Client for JAVA
Java
1
star
78

eslint-config-schibsted-react

ESLint configs for React.js from Schibsted
JavaScript
1
star
79

strongbox-examples

Usage examples for Strongbox
Java
1
star
80

react-account

React Context Provider and Hook making it easier to use Schibsted Account in your React app
JavaScript
1
star
81

identity-sdk

The identity SDK for Node
JavaScript
1
star
82

cassandra-backup

Stress-free improved Cassandra backups
1
star
83

browser-sdk

[DEPRECATED — DO NOT USE] Schibsted Browser SDK for identity and payment
JavaScript
1
star
84

lambda-slack-app-template

Everything you need to get started on making a Slack App running on AWS Lambda
JavaScript
1
star
85

WAAS-web

HTML
1
star
86

account-sdk-android-web

Schibsted account SDK for Android
Kotlin
1
star