• Stars
    star
    2,031
  • Rank 22,790 (Top 0.5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 6 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

write APIs using direct SQL queries with no hassle, let's rethink about SQL

SQLer

SQL-er is a tiny portable server enables you to write APIs using SQL query to be executed when anyone hits it, also it enables you to define validation rules so you can validate the request body/query params, as well as data transformation using simple javascript syntax. sqler uses nginx style configuration language (HCL) and javascript engine for custom expressions.

Table Of Contents

Features

  • Standalone with no dependencies.
  • Works with most of SQL databases out there including (SQL Server, MYSQL, SQLITE, PostgreSQL, Cockroachdb)
  • Built-in RESTful server
  • Built-in RESP Redis Protocol, you connect to SQLer using any redis client
  • Built-in Javascript interpreter to easily transform the result
  • Built-in Validators
  • Automatically uses prepared statements
  • Uses (HCL) configuration language
  • You can load multiple configuration files not just one, based on unix glob style pattern
  • Each SQL query could be named as Macro
  • Uses Javascript custom expressions.
  • Each macro has its own Context (query params + body params) as .Input which is map[string]interface{}, and .Utils which is a list of helper functions, currently it contains only SQLEscape.
  • You can define authorizers, an authorizer is just a simple webhook that enables sqler to verify whether the request should be done or not.
  • Trigger a webhook or another macro when a specific macro get executed.
  • Schedule specific macros to run at specific time using simple cron syntax.

Quick Tour

  • You install sqler using the right binary for your os from the releases page.
  • Let's say that you downloaded sqler_darwin_amd64
  • Let's rename it to sqler, and copy it to /usr/local/bin
  • Now just run sqler -h, you will the next
                         ____   ___  _
                        / ___| / _ \| |    ___ _ __
                        \___ \| | | | |   / _ \ '__|
                         ___) | |_| | |__|  __/ |
                        |____/ \__\_\_____\___|_|

        turn your SQL queries into safe valid RESTful apis.


  -config string
        the config file(s) that contains your endpoints configs, it accepts comma seprated list of glob style pattern (default "./config.example.hcl")
  -driver string
        the sql driver to be used (default "mysql")
  -dsn string
        the data source name for the selected engine (default "root:root@tcp(127.0.0.1)/test?multiStatements=true")
  -resp string
        the resp (redis protocol) server listen address (default ":3678")
  -rest string
        the http restful api listen address (default ":8025")
  -workers int
        the maximum workers count (default 4)
  • you can specifiy multiple files for -config as configuration, i.e -config="/my/config/dir/*.hcl,/my/config/dir2/*.hcl"
  • you need specify which driver you need and its dsn from the following:
Driver DSN
mysql usrname:password@tcp(server:port)/dbname?option1=value1&...
postgres postgresql://username:password@server:port/dbname?option1=value1
sqlite3 /path/to/db.sqlite?option1=value1
sqlserver sqlserver://username:password@host/instance?param1=value&param2=value
sqlserver://username:password@host:port?param1=value&param2=value
sqlserver://sa@localhost/SQLExpress?database=master&connection+timeout=30
mssql server=localhost\\SQLExpress;user id=sa;database=master;app name=MyAppName
server=localhost;user id=sa;database=master;app name=MyAppName
odbc:server=localhost\\SQLExpress;user id=sa;database=master;app name=MyAppName
odbc:server=localhost;user id=sa;database=master;app name=MyAppName
hdb (SAP HANA) hdb://user:password@host:port
clickhouse (Yandex ClickHouse) tcp://host1:9000?username=user&password=qwerty&database=clicks&read_timeout=10&write_timeout=20&alt_hosts=host2:9000,host3:9000

Supported DBMSs

  • MYSQL, TiDB, MariaDB, Percona and any MYSQL compatible server uses mysql driver.
  • PostgreSQL, CockroachDB and any PostgreSQL compatible server uses postgres driver.
  • SQL Server, MSSQL, ADO, ODBC uses sqlserver or mssql driver.
  • SQLITE, uses sqlite3 driver.
  • HANA (SAP), uses hdb driver.
  • Clickhouse, uses clickhouse driver.

Docker

SQLer has a docker image called alash3al/sqler it is an automated build, you can use it like the following:

# run the help message
docker run --rm alash3al/sqler --help

# connect to a local mysql
docker run --network=host alash3al/sqler -driver=mysql -dsn=usr:pass@tcp(127.0.0.1:3306)/dbname

# connect to another mysql container
docker run -link mysql alash3al/sqler -driver=mysql -dsn=usr:pass@tcp(mysql:3306)/dbname

Configuration Overview

// create a macro/endpoint called "_boot",
// this macro is private "used within other macros" 
// because it starts with "_".
_boot {
    // the query we want to execute
    exec = <<SQL
        CREATE TABLE IF NOT EXISTS `users` (
            `ID` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
            `name` VARCHAR(30) DEFAULT "@anonymous",
            `email` VARCHAR(30) DEFAULT "@anonymous",
            `password` VARCHAR(200) DEFAULT "",
            `time` INT UNSIGNED
        );
    SQL
}

// adduser macro/endpoint, just hit `/adduser` with
// a `?user_name=&user_email=` or json `POST` request
// with the same fields.
adduser {
    validators {
        user_name_is_empty = "$input.user_name && $input.user_name.trim().length > 0"
        user_email_is_empty = "$input.user_email && $input.user_email.trim(' ').length > 0"
        user_password_is_not_ok = "$input.user_password && $input.user_password.trim(' ').length > 5"
    }

    bind {
        name = "$input.user_name"
        email = "$input.user_email"
        password = "$input.user_password"
    }

    methods = ["POST"]

    authorizer = <<JS
        (function(){
            log("use this for debugging")
            token = $input.http_authorization
            response = fetch("http://requestbin.fullcontact.com/zxpjigzx", {
                headers: {
                    "Authorization": token
                }
            })
            if ( response.statusCode != 200 ) {
                return false
            }
            return true
        })()
    JS

    // include some macros we declared before
    include = ["_boot"]

    exec = <<SQL
        INSERT INTO users(name, email, password, time) VALUES(:name, :email, :password, UNIX_TIMESTAMP());
        SELECT * FROM users WHERE id = LAST_INSERT_ID();
    SQL
}

// list all databases, and run a transformer function
databases {
    exec = "SHOW DATABASES"
    cron = "* * * * *"
    trigger {
        webhook = "http://some.url/hook"
    }
}

// list all tables from all databases
tables {
    exec = "SELECT `table_schema` as `database`, `table_name` as `table` FROM INFORMATION_SCHEMA.tables"
}

// a macro that aggregates `databases` macro and `tables` macro into one macro
databases_tables {
    aggregate = ["databases", "tables"]
}

REST vs RESP

RESTful server could be used to interact directly with i.e mobile, browser, ... etc, in this mode SQLer is protected by authorizers, which gives you the ability to check authorization against another 3rd-party api.
Each macro you add to the configuration file(s) you can access to it by issuing a http request to /<macro-name>, every query param and json body will be passed to the macro .Input.

RESP server is just a basic REDIS compatible server, you connect to it using any REDIS client out there, even redis-cli, just open redis-cli -p 3678 list to list all available macros (commands), you can execute any macro as a redis command and pass the arguments as a json encoded data, i.e redis-cli -p 3678 adduser "{\"user_name\": \"u6\", \"user_email\": \"[email protected]\", \"user_password\":\"pass@123\"}".

Sanitization

SQLer uses prepared statements, you can bind inputs like the following:

addpost {
    // $input is a global variable holds all request inputs,
    // including the http headers too (prefixed with `http_`)
    // all http header keys are normalized to be in this form 
    // `http_x_header_example`, `http_authorization` ... etc in lower case.
    bind {
        title = "$input.post_title"
        content = "$input.post_content"
        user_id = "$input.post_user"
    }

    exec = <<SQL
        INSERT INTO posts(user_id, title, content) VALUES(:user_id, :title, :content);
        SELECT * FROM posts WHERE id = LAST_INSERT_ID();
    SQL
}

Validation

Data validation is very easy in SQLer, it is all about simple javascript expression like this:

addpost {
    // if any rule returns false, 
    // SQLer will return 422 code, with invalid rules.
    // 
    // $input is a global variable holds all request inputs,
    // including the http headers too (prefixed with `http_`)
    // all http header keys are normalized to be in this form 
    // `http_x_header_example`, `http_authorization` ... etc in lower case.
    validators {
        post_title_length = "$input.post_title && $input.post_title.trim().length > 0"
        post_content_length = "$input.post_content && $input.post_content.length > 0"
        post_user = "$input.post_user"
    }

    bind {
        title = "$input.post_title"
        content = "$input.post_content"
        user_id = "$input.post_user"
    }

    exec = <<SQL
        INSERT INTO posts(user_id, title, content) VALUES(:user_id, :title, :content);
        SELECT * FROM posts WHERE id = LAST_INSERT_ID();
    SQL
}

Authorization

If you want to expose SQLer as a direct api to API consumers, you will need to add an authorization layer on top of it, let's see how to do that

addpost {
    authorizer = <<JS
        (function(){
            // $input is a global variable holds all request inputs,
            // including the http headers too (prefixed with `http_`)
            // all http header keys are normalized to be in this form 
            // `http_x_header_example`, `http_authorization` ... etc in lower case.
            token = $input.http_authorization
            response = fetch("http://requestbin.fullcontact.com/zxpjigzx", {
                headers: {
                    "Authorization": token
                }
            })
            if ( response.statusCode != 200 ) {
                return false
            }
            return true
        })()
    JS
}

using that trick, you can use any third-party Authentication service that will remove that hassle from your code.

Data Transformation

In some cases we need to transform the resulted data into something more friendly to our API consumers, so I added javascript interpreter to SQLer so we can transform our data, each js code has a global variable called $result, it holds the result of the exec section, you should write your code like the following:

// list all databases, and run a transformer function
databases {
    exec = "SHOW DATABASES"

    transformer = <<JS
        // there is a global variable called `$result`,
        // `$result` holds the result of the sql execution.
        (function(){
            newResult = []

            for ( i in $result ) {
                newResult.push($result[i].Database)
            }

            return newResult
        })()
    JS
}

Aggregators

SQLer helps you to merge multiple macros into one to minimize the API calls number, see the example bellow

databases {
    exec = "SHOW DATABASES"

    transformer = <<JS
        // there is a global variable called `$result`,
        // `$result` holds the result of the sql execution.
        (function(){
            newResult = []

            for ( i in $result ) {
                newResult.push($result[i].Database)
            }

            return newResult
        })()
    JS
}

tables {
    exec = "SELECT `table_schema` as `database`, `table_name` as `table` FROM INFORMATION_SCHEMA.tables"
    transformer = <<JS
        (function(){
            $ret = {}
            for (  i in $result ) {
                if ( ! $ret[$result[i].database] ) {
                    $ret[$result[i].database] = [];
                }
                $ret[$result[i].database].push($result[i].table)
            }
            return $ret
        })()
    JS
}

databasesAndTables {
    aggregate {
        databases = "current_databases"
        tables = "current_tables"
    }
}

Issue/Suggestion/Contribution ?

SQLer is your software, feel free to open an issue with your feature(s), suggestions, ... etc, also you can easily contribute even you aren't a Go developer, you can write wikis it is open for all, let's make SQLer more powerful.

Author

I'm Mohamed Al Ashaal, just a problem solver :), you can view more projects from me here, and here is my email [email protected]

License

Copyright 2019 The SQLer Authors. All rights reserved. Use of this source code is governed by a Apache 2.0 license that can be found in the LICENSE file.

More Repositories

1

redix

a very simple pure key => value storage system that speaks Redis protocol with Postgres as storage engine and more
Go
1,166
star
2

wsify

Just a tiny, simple and real-time self-hosted pub/sub messaging service
Go
504
star
3

smtp2http

A tiny software that receive a smtp request (email) and send it to the specified webhook as a http post request
Go
410
star
4

lightify

a reverse proxy that boosts the web app performance!
Go
191
star
5

srchx

A standalone lightweight full-text search engine built on top of blevesearch and Go with multiple storage (scorch, boltdb, leveldb, badger)
Go
151
star
6

scraply

Scraply a simple dom scraper to fetch information from any html based website
Go
122
star
7

httpsify

a transparent HTTPS termination proxy using letsencrypt with auto certification renewal
Go
109
star
8

go-pubsub

An in-process simple publish/subscribe library for golang
Go
103
star
9

go-smtpsrv

a tiny package that implements SMTP server for Go projects
Go
95
star
10

re-txt

converts text-formats from one to another, it is very useful if you want to re-format a json file to yaml, toml to yaml, csv to yaml, ... etc
Go
64
star
11

dockssh

Dockssh, ssh into any container from anywhere with the help of redis for password storage
Go
61
star
12

bbadger

adding badger support to blevesearch
Go
59
star
13

xyr

Query any data source using SQL, works with the local filesystem, s3, and more. It should be a very tiny and lightweight alternative to AWS Athena, Presto ... etc.
Go
58
star
14

phoo

a very simple high performance PHP application server and php-fpm supervisor
Go
58
star
15

Plus

Asynchronous I/O environment in pure PHP
PHP
57
star
16

Horus

a minimal event-driven, flexible, portable and micro PHP web application framework
PHP
56
star
17

goukv

a key-value store with multiple backends including leveldb, badgerdb, postgresql
Go
52
star
18

scrapyr

a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd
Go
50
star
19

droxy

a transparent standalone http reverse proxy for docker containers
Go
45
star
20

sql2slack

send sql results periodically to slack
Go
34
star
21

xerver

a tiny static and fastcgi reverse proxy written in golang
Go
34
star
22

pxml-tpl

PHTML a smart tiny oo PHP HTML style template system, no more regex and no more large code, packags, ...
PHP
33
star
23

arp2http

notify the specified webhook when a device is connected/disconnected
Go
30
star
24

mailux

Experimental Passwordless Login even without sending emails/sms to users
Shell
29
star
25

goemitter

an event-emitter for golang
Go
22
star
26

katch

headless chrome as a service for generating PDF, PNG, JPEG and HTML of any web page
Go
16
star
27

aggrex

a crazy API gateway aggregation using javascript as a language and go as a runtime
Go
15
star
28

exeq

painless task queue manager for shell commands with an intuitive cli interface (execute shell commands in distributed cloud-native queue manager).
Go
13
star
29

axync

a smart cooperative multitasking kernel for php7
PHP
13
star
30

olive-go

Just a lightweight golang web application middleware
Go
13
star
31

img2pdf

convert a directory of images to a PDF file
Go
11
star
32

vidutils

a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg software
Go
11
star
33

go-schemaless

a schemaless implementation based on postgres and jsonb
Go
10
star
34

tix

a super simple stupid event-loop kernel in pure PHP
PHP
9
star
35

libsrchx

a simple wrapper for blevesearch created for srchx the tiny full-text search engine
Go
9
star
36

xerve

fast, tiny static file server with auto gzip & minify
Go
8
star
37

mylitedb

a mysql server backed by sqlite!
Go
8
star
38

funker

a platform for function as a service based javascript written in Go.
Go
7
star
39

go-univush

a universal push notification library written in go based on apns2 and firebase
Go
7
star
40

tlx

a dummy tls reverse proxy written in go
Go
6
star
41

dir2webp

convert images in directory/directories to webp format using the power of concurrent programming
Go
6
star
42

dart-path-selector

extract values from Map/List using dot-seprated strings you don't have to cast multiple times to fetch a simple values, this is very useful while working with i.e json data
Dart
5
star
43

uwatch

notify email(s) that a url is now online
Go
5
star
44

mysql2sqlite

a command-line utility to move data from mysql to sqlite
Go
5
star
45

xonsole

A tiny cli framework based on laravel\console, so you can use it without full laravel installation
PHP
4
star
46

go-fastcgi-client

A FastCGI client library for Golang projects
Go
4
star
47

HorusJS

A lightweight NodeJS microframework
JavaScript
3
star
48

cgify

a for fun script server (custom cgi)
Go
3
star
49

xtore

a document store based on cockroachdb
Go
3
star
50

wp-allowed-hosts

a plugin that protects your wp site from the CVE-2017-8295 vulnerability
PHP
2
star
51

alash3al

WHO AM I
2
star
52

xmg

a tiny microservice for getting perceptual hash of an image or faces in the image
Go
2
star
53

rsty

a tiny RESTful service middleware for Golang applications
Go
2
star
54

googly

a simple command line tool to search google with simple options
Go
2
star
55

goring

A simple consistent hashing ring implementation in Golang
Go
2
star
56

aof

A simple concurrency safe Append-Only-File for storage purposes
Go
2
star
57

olive.js

a lightweight micro javascript web application framework
JavaScript
1
star
58

php-myRouter

A PHP Router (/class/method/parm1/parm2/parm3/parm4/.....................)
PHP
1
star
59

z-pharo

Z-Pharo PHP CMS Framework
1
star
60

isup

check whether a url is up or down, and notify you whenever it become available
Go
1
star
61

LiteJS

A very tiny full featured template engine for javascript for both client and server sides, you need no-learning curve
JavaScript
1
star
62

odb

just a for-fun only-value database
Go
1
star