• This repository has been archived on 04/Jun/2019
  • Stars
    star
    8
  • Rank 2,099,232 (Top 42 %)
  • Language
    Erlang
  • License
    Apache License 2.0
  • Created over 10 years ago
  • Updated almost 9 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Client Libraries for Erlang

Craterl

Join the chat at https://gitter.im/crate/craterl

Build Status Another Badge

Erlang client for crate.

This client is in an alpha state and, though tested very intensively, not yet verified to be rock-solid in a production environment.

Compatibility

Tested with OTP releases R16B, 17 and 18.

Installation

Rebar 3

Craterl is built using rebar3, using it in your project is highly recommended.

Here is how you'd add it to your dependencies.

If you want to load a craterl release from hex.pm, configure it like so:

{deps,[
  {craterl,"0.2.3"}
]}.

Or load it directly from github:

{deps,[
    {craterl, {git, "git://github.com/crate/craterl.git", {tag, "0.2.3"}}}
]}.

Rebar 2

You can also use craterl in your project using rebar2, though rebar3 is recommended.

With rebar2 you need to load craterl from github. Use a git tag or a commit ref in order to get reproducable builds:

{deps,[
    {craterl, "0.2.3", {git, "git://github.com/crate.craterl.git", {tag, "0.2.3}}}
]}.

Mix

Craterl releases are hosted on hex.pm.

Declare it as a dependency in your mix.exs file:

def deps do
    [{:craterl, "~> 0.2.3"}]
end

Or load it directly from github:

def deps do
    [{:craterl, git: "git://github.com/crate/craterl.git", tag: "0.2.3"}]
end

Usage

Normally you'd start craterl as part of your application's startup, so the Erlang/OTP application framework will take care of actually starting craterl.

However, if you want to play around with craterl on the erlang or elixir shell you can start it up with the following convenience method:

ok = craterl:start().

Having a running craterl application, the next step is starting a client instance using a variant of the craterl:new() function:

ClientSpec = {local, my_client}.
Servers = [{<<"localhost">>, 4200}, "localhost:4201"].
Options = [{poolsize, 1000}, {timeout, 5000}].
ClientRef = craterl:new(ClientSpec, Servers, Options).

It is possible to create many clients on one erlang node. craterl client instances are created using a client spec which is a tuple you would use when registering a process, like {local, your_name}. The process name, your_name in this example, must be unique on a node.

Options

The following options can be used to change the behaviour of a newly created craterl client:

  • poolname - string() or binary(), the name of the connection pool, handled by hackney (erlang http client)
  • poolsize - integer(), the size of the connection pool, also handled by hackney
  • timeout - integer(), the receive and connect timeout for connections to crate servers, in milliseconds
  • ssl_options - term(), the same options the erlang ssl module accepts as ssloptions()
  • ssl_insecure - boolean(), whether ssl certificates should be validated or not

Example:

Options = [
    {poolname, "my_pool"}, 
    {poolsize, 200}, {timeout, 6000}, 
    {ssl_insecure, false}, 
    {ssl_options, [
        {cipers, [{rsa, aes_256_cbc, sha}]}, 
        {cacerts, MyDerEncodedCaCerts}
    ]}
].
ClientRef = craterl:new({local, craterl}, [{<<"localhost">>, 4200}], Options).

See the documentation of the craterl module for more detailed api documentation.

SQL

Issuing SQL statements using craterl is possible with one of the variants of craterl:sql():

{ok Response} = craterl:sql("select id, name from sys.cluster").
[[<<"89b8f6bf-4082-415b-937e-7de66b67f6fe">>, <<"crate">>]] = craterl_resp:rows(Response).
[<<"id">>,<<"name">>] = craterl_resp:column_names(Response).

Stmt = <<"select * from user where id in (?, ?, ?)">>.
Args = [1, 2, 3].
{ok, Response2} = craterl:sql(ClientRef, Stmt, Args).

For issuing multiple INSERT / DELETE or UPDATE requests with one roundtrip, the craterl:sql_bulk() functions can be used:

Stmt = <<"insert into t (id, name) values (?, ?)">>.
BulkArgs = [[1, <<"Ford">>], [2, <<"Trillian">>], [3, <<"Zaphod">>]].
{ok, BulkResponse} = craterl:sql_bulk(ClientRef, Stmt, BulkArgs).
BulkResults = craterl_resp:bulk_results(BulkResponse).
[1,1,1] = lists:map(fun craterl_resp:row_count/1, BulkResults).

Stmt2 = <<"update t set new_column=? where id=?">>.
BulkArgs2 = [[<<"funky">>, 1], [<<"shizzle">>, 2], [<<"indeed">>, 3]].
{ok, BulkResponse2} = craterl:sql_bulk(Stmt2, BulkArgs2).
BulkResults2 = craterl_resp:bulk_results(BulkResponse2).
[1,1,1] = lists:map(fun craterl_resp:row_count/1, BulkResults2).

{ok, SelectResponse} = craterl:sql("select * from t").
[[1,<<"Ford">>,<<"funky">>],
 [2,<<"Trillian">>,<<"shizzle">>],
 [3,<<"Zaphod">>,<<"indeed">>]] = craterl_resp:rows(SelectResponse).

Every sql api function has a variant that accepts a ClientRef as first argument that is an atom referencing a craterl client that has been started using a variant craterl:new() in order to issue your request against a specific server / cluster.

Blobs

Crate is able to store blobs, files, binary somethings. They can be replicated to make sure you won't lose data.

Craterl fully supports storing, retrieving and manipulating blobs.

At first a blob table is needed to store blobs into:

{ok, SqlResponse} = craterl:sql("create blob table myblobs with (number_of_replicas=1)").

It is possible to upload blobs from stuff you have available in ram or from a file:

Content = <<"awesome!">>.
{ok, {created, HashDigest}} = craterl:blob_put(<<"myblobs">>, Content).
HashDigest = <<"040f06fd774092478d450774f5ba30c5da78acc8">>.

File = <<"/usr/share/dict/words">>
{ok,{created, WordsHash}} = craterl:blob_put_file(ClientRef, <<"myblobs">>, <<"/usr/share/dict/words">>).
WordsHash = <<"a62edf8685920f7d5a95113020631cdebd18a185">>.

You can get blobs to memory or to file. Both methods will make use of chunked HTTP encoding so you will receive the blob piece by piece and won't load big blobs into memory at once.

{ok, GetDataFun} = craterl:blob_get(<<"myblobs">>, WordsHash).
GetDataFun()
{ok,<<"A\na\naa\naal\naalii\naam\nAani\naardvark\naardwolf\nAaron\nAaronic\nAaronical\nAaronite\nAaronitic\nAaru\nAb\naba\nAbabdeh\nA"...>>}
GetDataFun()
{ok,<<"inoposterior\nabdominoscope\nabdominoscopy\nabdominothoracic\nabdominous\nabdominovaginal\nabdominovesical\nabduce\n"...>>}
...
GetDataFun()
{ok, done}

NewWordsFile = <<"/tmp/blobwords">>
{ok, NewWordsFile} = craterl:blob_get_to_file(<<"myblobs">>, WordsHash, NewWordsFile).
file:read_file(NewWordsFile)
{ok,<<"A\na\naa\naal\naalii\naam\nAani\naardvark\naardwolf\nAaron\nAaronic\nAaronical\nAaronite\nAaronitic\nAaru\nAb\naba\nAbabdeh\nA"...>>}

Check for existence of blobs:

ok = craterl:blob_exists(ClientRef, <<"myblobs">>, WordsHash).
{error, 404} = craterl:blob_exists(craterl, <<"myblobs">>, <<"doesnotexist">>).

Delete blobs:

ok = craterl:blob_delete(<<"myblobs">>, WordsHash).
{error, 404} = craterl:blob_delete(<<"myblobs">>, WordsHash).

Every blob api function, like the sql functions, has a variant that accepts a ClientRef as first argument that is an atom referencing a craterl client that has been started using a variant of craterl:new().

Tests

Simply call make test to run the unit and integration tests for craterl.

Contributions

Are very welcome, be it code, constructive feedback, money, or some motivating words!

More Repositories

1

crate

CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.
Java
4,023
star
2

elasticsearch-inout-plugin

An Elasticsearch plugin which provides the ability to export data by query on server side.
Java
112
star
3

crate-python

Python DB API client library for CrateDB, using HTTP.
Python
79
star
4

crate-sample-apps

A JavaScript guestbook app with a number of different backend implementations, each using a different client library to communicate with CrateDB.
Java
66
star
5

cratedb-prometheus-adapter

CrateDB Prometheus Adapter.
Go
60
star
6

crash

Crash is an interactive CrateDB command line interface (CLI) SQL shell with autocompletion.
Python
48
star
7

docker-crate

Source repository for the official CrateDB Docker image
Python
48
star
8

elasticsearch-timefacets-plugin

Elasticsearch Timebased Facets
Java
41
star
9

crate-pdo

CrateDB PHP PDO adapter
PHP
34
star
10

crate_ruby

A Ruby client library for CrateDB.
Ruby
30
star
11

crate-admin

The admin user interface for CrateDB.
JavaScript
26
star
12

activerecord-crate-adapter

Ruby on Rails ActiveRecord adapter for CrateDB
Ruby
26
star
13

crate-mesos-framework

An integration framework that allows you to run and manage CrateDB via Apache Mesos.
Java
23
star
14

crate-jdbc

A JDBC driver for CrateDB.
Java
22
star
15

crate-operator

The CrateDB Kubernetes Operator provides a convenient way to run CrateDB clusters inside Kubernetes.
Python
22
star
16

sql-99

SQL-99 Complete, Really
19
star
17

crate-web

Crate.IO Website
HTML
19
star
18

crate-docs-theme

A Sphinx theme for the CrateDB documentation.
CSS
19
star
19

crate-dbal

Doctrine Database Access Layer for Crate.IO
PHP
16
star
20

crate-npgsql

A plugin that provides extensions to Npgsql which enable usage of Npgsql as a .NET data provider for CrateDB.
C#
11
star
21

crate-howtos

How-to guides for CrateDB.
9
star
22

cratedb-examples

A collection of clear and concise examples how to work with CrateDB.
Jupyter Notebook
9
star
23

crate-utils

Utility scripts for Crate
Python
9
star
24

crate-demo

Crate.IO Demos
CSS
8
star
25

crate-benchmarks

A collection of CrateDB benchmarks.
Python
7
star
26

sphinx_csv_filter

A CSV filter directive for docutils and Sphinx, that extends the "csv-table" reStructuredText directive to add row filtering options.
Python
7
star
27

crate-example-plugin

Crate Example Plugin
Java
6
star
28

docker-oauth2_proxy

Docker Image for oauth2_proxy
Shell
6
star
29

cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.
Python
6
star
30

helm-charts

Helm charts for Kubernetes
Smarty
5
star
31

cratedb-sqlparse

Parsing utilities to validate and split SQL statements for CrateDB.
JavaScript
5
star
32

cratedb-airflow-tutorial

Reference implementations for orchestration project using Astronomer/Airflow
Python
5
star
33

cratedb-terraform

Terraform deployment configurations for CrateDB
HCL
5
star
34

croud

A command line interface for CrateDB Cloud ☁
Python
5
star
35

tsperf

TSPERF Time Series Database Benchmark Suite. Framework for evaluating and comparing the performance of time series databases, in the spirit of TimescaleDB's TSBS.
Python
5
star
36

crate-gc-admin

TypeScript
4
star
37

mongodb-cratedb-migration-tool

Tool to facilitate migrations from MongoDB to CrateDB, now integrated into CrateDB Toolkit's MongoDB I/O subsystem.
Python
4
star
38

cloud-howtos

How-to guides for CrateDB Cloud.
4
star
39

crate-qa

CrateDB Quality Assurance
Python
4
star
40

devrel-shipping-forecast-geo-demo

Spatial data demo application using CrateDB and the Express framework.
JavaScript
4
star
41

webinar.101

CrateDB 101 Webinar series
Python
3
star
42

jmx_exporter

A javaagent to export CrateDB JMX metrics for prometheus
Java
3
star
43

eventhub-consumer-azure-function

Ingesting data into CrateDB with Azure Event Hub and Azure Functions
JavaScript
3
star
44

crate.io.packages

PyPi Package Mirror that can be found under https://crate.io/packages
Python
2
star
45

getting-started

Getting Started With CrateDB
Shell
2
star
46

cloud-tutorials

Tutorials for getting started with CrateDB Cloud.
2
star
47

crate-java-testing

CrateTestServer and CrateTestCluster classes for use as JUnit external resources.
Java
2
star
48

cratedb-flink-jobs

This repository accompanies the article "Build a data ingestion pipeline using Kafka, Flink, and CrateDB" and the "CrateDB Community Day #2".
Java
2
star
49

crate-version-comparison

CrateDB tests for INSERT performance
Shell
2
star
50

crate-commoncrawl

A plugin for importing Common Crawl data into CrateDB.
Java
2
star
51

docker-openresty

Docker Image for Openresty
Shell
2
star
52

crate-docs

Common build system, QA tools, style guides, and other meta miscellanies for doing docs at Crate.io.
Shell
2
star
53

crate-clients-tools

Clients, tools, and integrations for CrateDB.
2
star
54

reveal-crate

Reveal.JS Theme for Crate.IO presentations
JavaScript
1
star
55

crate-pricing-calculator

JavaScript
1
star
56

cratedb-jupyter

Repository containing jupyter notebooks in addition to our machine learning blog post.
Jupyter Notebook
1
star
57

cloud-reference

Reference documentation for CrateDB Cloud.
1
star
58

Stackfile

Crate Stackfile
1
star
59

crate-docs-utils

This project has been superseded by @crate/docs.
Makefile
1
star
60

dockercon-hackathon

https://www.meetup.com/de-DE/preview/Docker-Copenhagen/events/241845278
JavaScript
1
star
61

p-azure

Azure powering Crate.IO
Shell
1
star
62

crate-tutorials

Tutorials for getting started with CrateDB.
1
star
63

.github

Shared community health files
1
star
64

crate-repository-hdfs

A CrateDB plugin that enables Apache Hadoop HDFS support for snapshots and restore.
Java
1
star
65

mesos-deploy

Ansible playbook/roles for deploying an arbitrarily-sized Apache Mesos cluster
Python
1
star
66

academy-fundamentals-course

GitHub repository accompanying the CrateDB Fundamentals Course at the CrateDB Academy.
HTML
1
star
67

mlflow-cratedb

MLflow adapter for CrateDB.
Python
1
star
68

sqlalchemy-cratedb

SQLAlchemy dialect for CrateDB.
Python
1
star