• Stars
    star
    102
  • Rank 323,923 (Top 7 %)
  • Language
    Ruby
  • Created over 11 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

SQL input/output plugin for Fluentd

SQL input plugin for Fluentd event collector

Overview

This SQL plugin has two parts:

  1. SQL input plugin reads records from RDBMSes periodically. An example use case would be getting "diffs" of a table (based on the "updated_at" field).
  2. SQL output plugin that writes records into RDBMSes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.

Requirements

fluent-plugin-sql fluentd ruby
>= 1.0.0 >= v0.14.4 >= 2.1
< 1.0.0 < v0.14.0 >= 1.9

NOTE: fluent-plugin-sql v2's buffer format is different from v1. If you update the plugin to v2, don't reuse v1's buffer.

Installation

$ fluent-gem install fluent-plugin-sql --no-document
$ fluent-gem install pg --no-document # for postgresql

You should install actual RDBMS driver gem together. pg gem for postgresql adapter or mysql2 gem for mysql2 adapter. Other adapters supported by ActiveRecord should work.

We recommend that mysql2 gem is higher than 0.3.12 and pg gem is higher than 0.16.0.

If you use ruby 2.1, use pg gem 0.21.0 (< 1.0.0) because ActiveRecord 5.1.4 or earlier doesn't support Ruby 2.1.

Input: How It Works

This plugin runs following SQL periodically:

SELECT * FROM table WHERE update_column > last_update_column_value ORDER BY update_column ASC LIMIT 500

What you need to configure is update_column. The column should be an incremental column (such as AUTO_ INCREMENT primary key) so that this plugin reads newly INSERTed rows. Alternatively, you can use a column incremented every time when you update the row (such as last_updated_at column) so that this plugin reads the UPDATEd rows as well. If you omit to set update_column parameter, it uses primary key.

It stores last selected rows to a file (named state_file) to not forget the last row when Fluentd restarts.

Input: Configuration

<source>
  @type sql

  host rdb_host
  port rdb_port
  database rdb_database
  adapter mysql2_or_postgresql_or_etc
  username myusername
  password mypassword

  tag_prefix my.rdb  # optional, but recommended

  select_interval 60s  # optional
  select_limit 500     # optional

  state_file /var/run/fluentd/sql_state

  <table>
    table table1
    tag table1  # optional
    update_column update_col1
    time_column time_col2  # optional
  </table>

  <table>
    table table2
    tag table2  # optional
    update_column updated_at
    time_column updated_at  # optional
    time_format %Y-%m-%d %H:%M:%S.%6N # optional
  </table>

  # detects all tables instead of <table> sections
  #all_tables
</source>
  • host RDBMS host
  • port RDBMS port
  • database RDBMS database name
  • adapter RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
  • username RDBMS login user name
  • password RDBMS login password
  • tag_prefix prefix of tags of events. actual tag will be this_tag_prefix.tables_tag (optional)
  • select_interval interval to run SQLs (optional)
  • select_limit LIMIT of number of rows for each SQL (optional)
  • state_file path to a file to store last rows
  • all_tables reads all tables instead of configuring each tables in <table> sections

<table> sections:

  • tag tag name of events (optional; default value is table name)
  • table RDBM table name
  • update_column: see above description
  • time_column (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.
  • primary_key (optional): if you want to get data from the table which doesn't have primary key like PostgreSQL's View, set this parameter.
  • time_format (optional): if you want to specify the format of the date used in the query, useful when using alternative adapters which have restrictions on format

Input: Limitation

You should make sure target tables have index (and/or partitions) on the update_column. Otherwise SELECT causes full table scan and serious performance problem.

You can't replicate DELETEd rows.

Output: How It Works

This plugin takes advantage of ActiveRecord underneath. For host, port, database, adapter, username, password, socket parameters, you can think of ActiveRecord's equivalent parameters.

Output: Configuration

<match my.rdb.*>
  @type sql
  host rdb_host
  port 3306
  database rdb_database
  adapter mysql2_or_postgresql_or_etc
  username myusername
  password mypassword
  socket path_to_socket
  remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql

  <table>
    table table1
    column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'
    # This is the default table because it has no "pattern" argument in <table>
    # The logic is such that if all non-default <table> blocks
    # do not match, the default one is chosen.
    # The default table is required.
  </table>

  <table hello.*> # You can pass the same pattern you use in match statements.
    table table2
    # This is the non-default table. It is chosen if the tag matches the pattern
    # AFTER remove_tag_prefix is applied to the incoming event. For example, if
    # the message comes in with the tag my.rdb.hello.world, "remove_tag_prefix my.rdb"
    # makes it "hello.world", which gets matched here because of "pattern hello.*".
  </table>
  
  <table hello.world>
    table table3
    # This is the second non-default table. You can have as many non-default tables
    # as you wish. One caveat: non-default tables are matched top-to-bottom and
    # the events go into the first table it matches to. Hence, this particular table
    # never gets any data, since the above "hello.*" subsumes "hello.world".
  </table>
</match>
  • host RDBMS host
  • port RDBMS port
  • database RDBMS database name
  • adapter RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)
  • username RDBMS login user name
  • password RDBMS login password
  • socket RDBMS socket path
  • pool A connection pool synchronizes thread access to a limited number of database connections
  • timeout RDBMS connection timeout
  • remove_tag_prefix remove the given prefix from the events. See "tag_prefix" in "Input: Configuration". (optional)

<table> sections:

  • table RDBM table name
  • column_mapping: [Required] Record to table schema mapping. The format is consists of from:to or key values are separated by ,. For example, if set 'item_id:id,item_text:data,updated_at' to column_mapping, item_id field of record is stored into id column and updated_at field of record is stored into updated_at column.
  • <table pattern>: the pattern to which the incoming event's tag (after it goes through remove_tag_prefix, if given). The patterns should follow the same syntax as that of <match>. Exactly one <table> element must NOT have this parameter so that it becomes the default table to store data.

More Repositories

1

fluentd

Fluentd: Unified Logging Layer (project under CNCF)
Ruby
12,329
star
2

fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
C
5,323
star
3

fluentd-kubernetes-daemonset

Fluentd daemonset for Kubernetes and it Docker image
Ruby
1,210
star
4

fluentd-ui

Web UI for Fluentd
Ruby
596
star
5

fluent-operator

Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Go
535
star
6

fluent-bit-kubernetes-logging

Fluent Bit Kubernetes Daemonset
466
star
7

fluentd-docker-image

Docker image for Fluentd
Dockerfile
452
star
8

fluent-logger-python

A structured logger for Fluentd (Python)
Python
424
star
9

fluent-logger-golang

A structured logger for Fluentd (Golang)
Go
380
star
10

helm-charts

Helm Charts for Fluentd and Fluent Bit
Mustache
355
star
11

fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
Ruby
308
star
12

fluent-plugin-kafka

Kafka input and output plugin for Fluentd
Ruby
298
star
13

fluentd-forwarder

Fluentd Forwarder: Lightweight Data Collector in Golang
Go
283
star
14

fluent-logger-node

A structured logger for Fluentd (Node.js)
JavaScript
257
star
15

fluent-plugin-prometheus

A fluent plugin that collects metrics and exposes for Prometheus.
Ruby
253
star
16

fluent-logger-ruby

A structured logger for Fluentd (Ruby)
Ruby
251
star
17

fluent-logger-php

A structured logger for Fluentd (PHP)
PHP
216
star
18

fluent-logger-java

A structured logger for Fluentd (Java)
Java
205
star
19

sigdump

Use signal to show stacktrace of a Ruby process without restarting it
Ruby
183
star
20

fluent-bit-go

Fluent Bit Golang package to build plugins
Go
173
star
21

fluent-plugin-mongo

MongoDB input and output plugin for Fluentd
Ruby
171
star
22

fluent-plugin-rewrite-tag-filter

Fluentd Output filter plugin to rewrite tags that matches specified attribute.
Ruby
168
star
23

fluent-bit-docs

Fluent Bit - Official Documentation
Shell
119
star
24

fluent-plugin-grok-parser

Fluentd's Grok parser
Ruby
103
star
25

nginx-fluentd-module

Nginx module for Fluentd data collector
C
85
star
26

fluent-bit-docker-image

Docker image for Fluent Bit
Shell
67
star
27

fluent-plugin-webhdfs

Hadoop WebHDFS output plugin for Fluentd
Ruby
59
star
28

fluent-plugin-opensearch

OpenSearch Plugin for Fluentd
Ruby
49
star
29

fluentd-docs

This repository is deprecated. Go to fluentd-docs-gitbook repository.
Ruby
49
star
30

fluentd-benchmark

Benchmark collection of fluentd use cases
Shell
47
star
31

fluent-logger-scala

A structured logger implementation in Scala.
Shell
45
star
32

NLog.Targets.Fluentd

C#
44
star
33

fluent-logger-perl

A structured logger for Fluentd (Perl)
Perl
43
star
34

fluent-plugin-multiprocess

Multiprocess agent plugin for Fluentd
Ruby
42
star
35

fluentd-docs-gitbook

Fluentd documentation project in Gitbook format
JavaScript
41
star
36

fluent-plugin-splunk

Fluentd Plugin for Splunk
Ruby
38
star
37

fluent-plugin-parser-cri

CRI log parser for Fluentd
Ruby
32
star
38

fluent-bit-perf

Fluent Bit Performance Tools
C
31
star
39

fluent-plugin-windows-eventlog

Fluentd plugin to collect windows event logs
Ruby
31
star
40

fluent-plugin-flume

Flume input and output plugin for Fluentd
Ruby
23
star
41

kafka-connect-fluentd

Kafka Connect for Fluentd
Java
23
star
42

chunkio

Simple library to manage chunks of data in memory and file system
C
21
star
43

fluent-package-builder

td-agent (Fluentd) Building and Packaging System
Shell
21
star
44

fluent-plugin-scribe

Scribe input/output plugin for Fluentd data collector
Ruby
20
star
45

fluent-plugins

18
star
46

cmetrics

A standalone library to create and manipulate metrics in C
C
15
star
47

website

http://fluentd.org/
CSS
14
star
48

fluent-plugin-sanitizer

Ruby
14
star
49

fluent-bit-plugin

Fluent Bit Dynamic Plugin Development
C
13
star
50

fluent-bit-packaging

Fluent Bit Linux Packaging environment using Docker
Dockerfile
12
star
51

fluent-logger-forward-node

A fluent forward protocol implementation for Node.js
TypeScript
11
star
52

fluentd-website

For fluentd.org
CSS
10
star
53

fluent-logger-erlang

A structured logger for Fluentd (Erlang)
Erlang
10
star
54

fluent-plugin-msgpack-rpc

MessagePack-RPC input plugin for Fluentd data collector
Ruby
8
star
55

fluent-bit-ci

CI/CD for Fluent-bit
Shell
7
star
56

fluent-logger-ocaml

A structured logger for Fluentd (OCaml)
OCaml
7
star
57

fluent-plugin-hoop

Hoop (HDFS over HTTP) Plugin for Fluentd data collector
Ruby
6
star
58

data-collection

Data Collection with Fluentd
6
star
59

fluent-logger-d

A structured logger for Fluentd (D)
JavaScript
6
star
60

diagtool

Bringing productivity of trouble shooting to the next level by automating collection of Fluentd configurations, settings and OS parameters as well as masking sensitive information in logs and configurations.
Ruby
5
star
61

fluent-bit-tutorials

Fluent Bit Tutorials, custom articles to get started
5
star
62

m3-workshop-fluentcon

Shell
4
star
63

fluentbit-website-v3

CSS
4
star
64

fluent.github.com

website
JavaScript
4
star
65

fluentd-aggregator-docker-image

A Fluentd container image to be used for log aggregation and based on the official Fluentd Docker image.
Dockerfile
4
star
66

fluent-bit-observability-demo

JavaScript
3
star
67

fluent-bit-docs-stream-processing

Fluent Bit Stream Processing Guide
3
star
68

onigmo

Onigmo library with security and stable patches on top by Fluent maintainers
C
3
star
69

fluent-bit-website

Fluent Bit Website (work in process)
HTML
3
star
70

fluent-bit-test

Testing infrastructure for Fluent Bit
2
star
71

fluent-bit-labs

Fluent Bit Dev Labs
2
star
72

fluent-bit-website-old

Fluent Bit website
CSS
2
star
73

fluentbit-website-v2

Fluent Bit Website v2
CSS
2
star
74

fluent-plugin-buffer-chunkio

Ruby
2
star
75

fluent-bit-infra

Automation related to fluent-bit infrastructure
HCL
2
star
76

fluent-plugin-sd-dns

DNS based service discovery plugin for Fluentd
Ruby
2
star
77

fluent-plugin-parser-winevt_xml

Fluentd Parser plugin to parse XML rendered windows event log.
Ruby
1
star
78

cfl

Tiny library for data structures management, call it c:\ floppy
C
1
star
79

fluentd-docs-kubernetes

Fluentd DaemonSet Documentation for Kubernetes
1
star
80

fluent-bit-sandbox

A repository to covering the setup and configuration of the Fluent Bit Sandbox.
Shell
1
star
81

fluent-plugin-prometheus_pushgateway

Ruby
1
star
82

fluentd-website-hugo

SCSS
1
star
83

fluent-bit-chatops-demo

Demo of using Fluent Bit for ChatOps - created for Cloud Native Rejekts EU 2024 talk
Java
1
star
84

ctraces

Library to create and manipulate traces in C
C
1
star