• Stars
    star
    198
  • Rank 196,898 (Top 4 %)
  • Language
    Go
  • License
    BSD 2-Clause "Sim...
  • Created about 9 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Service that exposes Prometheus metrics for a SQL result set.

Prometheus SQL

Docker tag latest is based on the same commit as the latest released version.

GitHub release Github Releases Docker Pulls GoDoc

Service that generates basic metrics for SQL result sets and exposing them as Prometheus metrics.

This service relies on the SQL Agent service to execute and return the SQL result sets.

Changelog

Behavior

  • Static configuration files are used to define the queries to monitor.
  • Each query has a designated worker for execution.
  • An interval is used to define how often to execute the query.
  • Failed queries are automatically retried using a backoff mechanism.
  • Faceted metrics are supported.
  • A single metric's different facets can be filled in from different data sources.

Format

  • Metric names are exposed in the format query_result_<metric name>.
  • With faceted metrics, the name of the data column is determined by the data-field key in config, and all other columns (and column values) are exposed as labels.
  • If the result set consists of a single row and column, the metric value is obvious and data-field is not needed.
  • Label names under the same metric should be consistent.
  • Each different query (query entry in config) for the same metric should lead to different label values.

Usage

Usage of prometheus-sql:
  -config string
        Configuration file to define common data sources etc.
  -host string
        Host of the service. (0.0.0.0)
  -lax
        Tolerate invalid files in queryDir
  -port int
        Port of the service. (default 8080)
  -queries string
        Path to file containing queries. (default "queries.yml")
  -queryDir string
        Path to directory containing queries.
  -service string
        Query of SQL agent service.

Queries file

A queries file is required for the application to know which data source to query and which queries that shall be monitored.

In the repository there is an example file that you can have a look at.

Config file

The config file is optional and can defined some default values for queries and data sources which can be referenced by queries. The benefit of referencing a data source will be reduction of duplication of database connection information. See example config file here and queries file which utilizes the config information.

Run via console

Create a queries.yml file in the current directory and run the following:

prometheus-sql

or for an alternate path, use the -queries or the -queryDir option:

prometheus-sql -queries ${PWD}/queries.yml

Run using Docker

Run the SQL agent service.

docker run -d --name sqlagent dbhi/sql-agent

Run this service. Mount the queries.yml file and link the SQL Agent service.

docker run -d \
  --name prometheus-sql \
  -p 8080:8080 \
  -v ${PWD}/queries.yml:/queries.yml \
  --link sqlagent:sqlagent \
  dbhi/prometheus-sql

If you want to separate database connection information etc you can do that by specifying data sources in separate file which you then can mount:

docker run -d \
  --name prometheus-sql \
  -p 8080:8080 \
  -v ${PWD}/queries.yml:/queries.yml \
  -v ${PWD}/prometheus-sql.yml:/prometheus-sql.yml \
  --link sqlagent:sqlagent \
  dbhi/prometheus-sql \
  -service http://sqlagent:5000 \
  -config prometheus-sql.yml

To view a plain text version of the metrics, open up the browser to the http://localhost:8080/metrics (or http://192.168.59.103:8080/metrics for boot2docker users).

Run using a Docker Compose file

Alternately, use the docker-compose.yml file included in this repository. The volumes section be added for mounting the queries.yml file.

Contributing

Read instructions how to contribute before you start.

FAQ

How do you I provide additional options to the database connection?

Additional options are set in the config.yml file, specifically as additional key-value pairs in the connection map. These are passed to the SQL Agent service which construct a DSN string to establish the connection on the backend (alternately you can set the dsn key as the full string).

As an example, a common gotcha when using Postgres in a development environment is to ignore SSL not being enabled on the server. This can be done by adding the sslmode: disable option in the connection map.

    # ...
    connection:
        host: example.org
        port: 5432
        user: postgres
        password: s3cre7
        database: products
        sslmode: disable

More Repositories

1

sql-agent

HTTP interface for executing ad-hoc SQL queries.
Go
93
star
2

dataexpress

[NOT MAINTAINED] DataExpress is a simple, Scala-based cross database ETL toolkit supporting Postgres, MySql, Oracle, SQLServer, and Sqlite
Scala
71
star
3

dicom-anon

Python DICOM Anonymizer
Python
65
star
4

avocado

Metadata APIs for Django
Python
41
star
5

origins

Origins is an open source bi-temporal database for storing and retrieving facts for slowly-changing data. It support "time travel" queries and has built-in change detection.
Go
39
star
6

django-forkit

**INACTIVE** Adds support for shallow and deep forking (copying) Django model instances.
Python
37
star
7

nats-rpc

RPC layer on top of NATS standardizing on Protocol Buffers.
Go
34
star
8

twitter-adr-blstm

A model for finding mentions of adverse drug reactions in Twitter posts
Python
33
star
9

eda

eda is a library for implementing event-driven architectures.
Go
32
star
10

cilantro

Official client for Harvest (http://harvest.research.chop.edu)
JavaScript
28
star
11

data-models

Collection of various biomedical data models in parseable formats.
28
star
12

harvest

Harvest CLI and starter project template
Python
26
star
13

studycentric

StudyCentric - research DICOM viewer
JavaScript
26
star
14

diff-table

Go
24
star
15

varify

Clinical DNA Sequencing Analysis and Data Warehouse
JavaScript
24
star
16

serrano

Serrano implements a hypermedia API for Avocado. Avocado is a Django app which targets developers who are interested in letting their data do the work for them.
Python
20
star
17

modeltree

Django model relation tree traversal, i.e. automatic joins.
Python
18
star
18

sql-importer

Import CSV files into Postgres with automatic column typing and table creation.
Go
15
star
19

dicom-pipeline

An automated pipeline for reviewing, anonymizing and publishing research DICOM studies
Python
14
star
20

drug_word_embeddings

development and intrinsic evaluation of drug related word embeddings using PubMed abstracts and DrugBank
Python
14
star
21

sepsis_01

Python
12
star
22

data-models-sqlalchemy

SQLAlchemy models and DDL and ERD generation from chop-dbhi/data-models style JSON endpoints.
Python
11
star
23

django-webhooks

Simple webhooks implementation for Django
Python
9
star
24

django-sts

State Transition System for Django
Python
9
star
25

django-dicom-review

JavaScript
8
star
26

harvest-vocab

Abstract models and manager for querying hierarchical data for Harvest
Python
7
star
27

scds

Slowly-Changing Dimension Store (prototype)
Go
7
star
28

django-redcap

Utilities for porting REDCap projects to and from Django models
Python
6
star
29

redcap-repeat

Script for REDCap data dictionaries to support repeating fields and groups of fields.
Python
6
star
30

netezza-docker

Dockerfile for the Netezza ODBC driver.
Shell
5
star
31

omop_harvest

OMOP CDMV4 data model with OSIM2 data in Harvest
JavaScript
5
star
32

vscode-multiline-string-editor

Edit multiline strings that need to be encoded as single-line strings (for example, JSON strings that contain literal `\n` characters) in Visual Studio Code.
TypeScript
5
star
33

arrc

AudgenDB radiology report text classification and REST service
Python
4
star
34

varify-data-warehouse

Data model and loading pipeline for Varify
Python
4
star
35

harvest-openmrs

Harvest demo application for OpenMRS
JavaScript
4
star
36

sql-schema-exporter

Utility to export schema information from a relational database to JSON.
Python
3
star
37

django-concerns

Django app for reporting privacy concerns
Python
3
star
38

schema-registry

General purpose schema registry.
Go
3
star
39

redcap-pdf-plugin

Python
3
star
40

grin_pipeline

The exome pipeline used for the GRIN epilepsy pilot
Python
3
star
41

biorepo-portal

JavaScript
3
star
42

data-models-generator

Script to generate files in the data models format from common data sources.
Python
3
star
43

bitindex

Bitmap index for any, all, not any, and not all set operations.
Go
3
star
44

twitter-adr-convnet

Convolutional NNets for tweet classification
Python
2
star
45

django-objectset

Set-like abstract model class for Django
Python
2
star
46

django-siteauth

Middleware that allows for site-wide URL-based access control
Python
2
star
47

HaaS

Harvest as a Service
Python
2
star
48

sql-extractor

Go
2
star
49

aliases

A service for generating random external-facing aliases to internal identifiers.
Go
2
star
50

htq

Queues HTTP requests and stores the response in Redis for later retrieval
Python
2
star
51

go-unats

A unified Go API for NATS and NATS Streaming.
Go
2
star
52

ehb-datasources

datasource drivers, e.g. REDCap and Nautilus, that can be plugged into the Biorepository Portal
Python
2
star
53

harvest-template

Project template for Harvest applications.
JavaScript
1
star
54

origins-domains

Collection of curated domain models for Origins.
1
star
55

redcap-harvest

Dockerfile and tools for deriving Harvest applications from REDCap projects.
Shell
1
star
56

phenomantics

Multiple projects related to variant prioritization using phenotype semantic similarity
Scala
1
star
57

harvest-site

Website and documentation for The Harvest Stack
HTML
1
star
58

ehb-service

Python
1
star
59

etl-package

An ETL Framework using cbmi/dataExpress
Scala
1
star
60

data-models-django

Django model generator for chop-dbhi/data-models-service style JSON metadata.
Python
1
star
61

coriander

[NO LONGER ACTIVE} SCSS style sheets for django-cilantro
Ruby
1
star
62

typeselect

Wrapper for twitter/typeahead.js to enable multi-select input.
JavaScript
1
star
63

dicom-tools

Utility scripts for pushing and receiving DICOM files
Python
1
star
64

mybic

myBiC bioinformatics portal
JavaScript
1
star
65

updateurls

Utility script to update all urls in a text file to their latest location
Python
1
star
66

masino-lab-obesity-incidence

Obesity incidence research output under PBD project
Rich Text Format
1
star
67

sqltojson

Tool to build/export JSON documents from nested SQL queries.
Go
1
star
68

data-models-service

Service for consuming files in the data models format.
Go
1
star
69

origins-dispatch

Origins Dispatch service.
Go
1
star
70

oss

Guide to CBMi's open sourced software
Python
1
star
71

varify-site

CSS
1
star
72

data-models-validator

Set of tools for validating data that should conform to a data model.
Go
1
star
73

django-pdfappend

Simple django-app to retrieve multiple PDFs and return as a single PDF
Python
1
star
74

twitter-adr-lexicon

Adverse drug reaction Lucene index creator and event detection analysis
Python
1
star