• Stars
    star
    688
  • Rank 64,004 (Top 2 %)
  • Language
    Go
  • License
    Other
  • Created over 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Golang framework for streaming ETL, observability data pipeline, and event processing apps

firebolt Code Coverage Badge by Gopherbadger Build Status Go Report Card

firebolt logo

A golang framework for streaming event processing & data pipeline apps

Introduction

Firebolt has a simple model intended to make it easier to write reliable pipeline applications that process a stream of data.

It can be used to build systems such as:

  • logging/observability pipelines
  • streaming ETL
  • event processing pipelines

Every application's pipeline starts with a single source, the component that receives events from some external system. Sources must implement the node.Source interface.

We provide one built-in source:

  • kafkaconsumer - Events come from a Kafka topic, and are passed to the root nodes as []byte

The processing of your application is executed by its nodes which form a processing tree. Data - events - flow down this tree. A parent node passes results down to it's child nodes. Nodes may process events synchronously or asynchronously. A synchronous node type node.FanoutNode returns a slice of results for fanout or 'demultiplexing' use cases. Each node must implement the node.SyncNode, node.FanoutNode, or node.AsyncNode interfaces accordingly.

We provide two built-in node types:

  • kafkaproducer - Events are produced onto a kafka topic by an asynchronous producer.
  • elasticsearch - Events are bulk indexed into Elasticsearch.

Firebolt has both run and compile-time dependencies on librdkafka, see Developing

Example: Logging Pipeline

At DigitalOcean, our first use of Firebolt was in our logging pipeline. This pipeline consumes logs from just about every system we run. The diagram below depicts the source and nodes in this application.

This system uses the built-in kafkaconsumer source (in yellow) and kafkaproducer and elasticsearch nodes (in green). The blue nodes are custom to this application.

Logging Pipeline Node Diagram

What does Firebolt do for me?

Firebolt is intended to address a number of concerns that are common to near-realtime data pipeline applications, making it easy to run a clustered application that scales predictably to handle large data volume.

It is not an analytics tool - it does not provide an easy way to support 'wide operations' like record grouping, windowing, or sorting that require shuffling data within the cluster. Firebolt is for 'straight through' processing pipelines that are not sensitive to the order in which events are processed.

Some of the concerns Firebolt addresses include:

  • kafka sources Minimal configuration and no code required to consume from a Kafka topic, consumer lag metrics included
  • kafka sinks Same for producing to a Kafka topic
  • loose coupling Nodes in the pipeline are loosely coupled, making them easily testable and highly reusable
  • simple stream filtering Filter the stream by returning nil in your nodes
  • convenient error handling Send events that fail processing to a kafka topic for recovery or analysis with a few lines of config
  • outage recovery: offset management Configurable Kafka offset management during recovery lets you determine the maximum "catch up" to attempt after an outage, so you can quickly get back to realtime processing.
  • outage recovery: parallel recovery After an outage, process realtime data and "fill-in" the outage time window in parallel, with a rate limit on the recovery window.
  • monitorability Firebolt exposes Prometheus metrics to track the performance of your Source and all Nodes without writing code. Your nodes can expose their own custom internal metrics as needed.
  • leader election Firebolt uses Zookeeper to conduct leader elections, facilitating any processing that may need to be conducted on one-and-only-one instance.

Documentation

  1. Configuration The configuration file format

  2. Execution How Firebolt processes your data

  3. Registry Adding node types to the registry

  4. Sample Application Code Example code for running the Firebolt executor

  5. Sources Implementing and using sources

  6. Sync Nodes Implementing and using synchronous nodes

  7. Fanout Nodes Implementing and using fanout nodes

  8. Async Nodes Implementing and using asynchronous nodes

  9. Leader Election Starting leader election and accessing election results

  10. Messaging How to send and receive messages between the components of your system

  11. Metrics What metrics are exposed by default, and how to add custom metrics to your nodes

Built-In Types

  1. Kafka Producer Node for producing events onto a Kafka topic

  2. Elasticsearch Node for indexing documents to an Elasticsearch cluster

Developing

Firebolt depends on librdkafka v1.3.0 or later. To get started building a firebolt app (or working on firebolt itself), install it following the instructions here.

An example for debian-based distros:

sudo wget -qO - https://packages.confluent.io/deb/5.4/archive.key | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/5.4 stable main"
sudo apt-get update
sudo apt-get install -y librdkafka1 librdkafka-dev

More Repositories

1

nginxconfig.io

⚙️ NGINX config generator on steroids 💉
JavaScript
27,244
star
2

doctl

The official command line interface for the DigitalOcean API.
Go
3,155
star
3

godo

DigitalOcean Go API client
Go
1,328
star
4

go-libvirt

Package libvirt provides a pure Go interface for interacting with Libvirt. Apache 2.0 Licensed.
Go
815
star
5

do_user_scripts

Shell
804
star
6

Kubernetes-Starter-Kit-Developers

Hands-on tutorial and Automation stack for an operations-ready DigitalOcean Kubernetes (DOKS) cluster.
HCL
705
star
7

go-qemu

Go packages to interact with QEMU using the QEMU Machine Protocol (QMP). Apache 2.0 Licensed.
Go
684
star
8

do-agent

Collects system metrics from DigitalOcean Droplets
Go
586
star
9

csi-digitalocean

A Container Storage Interface (CSI) Driver for DigitalOcean Block Storage
Go
565
star
10

clusterlint

A best practices checker for Kubernetes clusters. 🤠
Go
536
star
11

vulcan

Vulcan extends Prometheus adding horizontal scalability and long-term storage
Go
531
star
12

digitalocean-cloud-controller-manager

Kubernetes cloud-controller-manager for DigitalOcean (beta)
Go
517
star
13

hacktoberfest

Hacktoberfest - App to manage the annual open-source challenge, used for the 2019 & 2020 seasons.
Ruby
510
star
14

droplet_kit

DropletKit is the official DigitalOcean API client for Ruby.
Ruby
507
star
15

terraform-provider-digitalocean

Terraform DigitalOcean provider
Go
487
star
16

action-doctl

GitHub Actions for DigitalOcean - doctl
JavaScript
454
star
17

ceph_exporter

Prometheus exporter that scrapes meta information about a ceph cluster.
Go
396
star
18

engineering-code-of-conduct

Code of Conduct for DigitalOcean's Engineering Team
289
star
19

go-openvswitch

Go packages which enable interacting with Open vSwitch and related tools. Apache 2.0 Licensed.
Go
282
star
20

kubernetes-sample-apps

Example DigitalOcean Kubernetes workload with service exposed through a DO load-balancer.
Python
252
star
21

marketplace-partners

Image validation, automation, and other tools for DigitalOcean Marketplace Vendors and Custom Image users
Shell
190
star
22

gta

gta: do transitive analysis to find packages whose dependencies have changed
Go
182
star
23

heartbot

A shot of love for your favorite chat client.
CoffeeScript
178
star
24

prometheus-client-c

A Prometheus Client in C
C
154
star
25

marketplace-kubernetes

This repository contains the source code and deployment scripts for Kubernetes-based applications listed in the DigitalOcean Marketplace.
Shell
154
star
26

go-smbios

Package smbios provides detection and access to System Management BIOS (SMBIOS) and Desktop Management Interface (DMI) data and structures. Apache 2.0 Licensed.
Go
152
star
27

kartograph

Kartograph makes it easy to generate and convert JSON. It's intention is to be used for API clients.
Ruby
147
star
28

OpenVPN-Pihole

https://marketplace.digitalocean.com/apps/openvpn-pihole
Shell
146
star
29

captainslog

A Syslog Protocol Parser
Go
136
star
30

resource_kit

Resource Kit provides tools to aid in making API Clients. Such as URL resolving, Request / Response layer, and more.
Ruby
134
star
31

go-workers2

better-go-workers
Go
121
star
32

doks-debug

A Docker image with Kubernetes manifests for investigation and troubleshooting.
Dockerfile
109
star
33

droplet-1-clicks

Packer build scripts for DigitalOcean Marketplace 1-clicks.
Shell
105
star
34

supabase-on-do

HCL
98
star
35

openapi

The OpenAPI v3 specification for DigitalOcean's public API.
JavaScript
97
star
36

container-blueprints

DigitalOcean Kubernetes(DOKS) Solution Blueprints
HCL
92
star
37

sample-dockerfile

⛵ App Platform sample Docker application.
Go
90
star
38

DOKS

Managed Kubernetes designed for simple and cost effective container orchestration.
80
star
39

app_action

Deploy to DigitalOcean Container Registry and App Platform
Go
78
star
40

navigators-guide

Book and code examples that help to build infrastructure on DigitalOcean
Shell
76
star
41

do-operator

The Kubernetes Operator for DigitalOcean
Go
76
star
42

pydo

Official DigitalOcean Python Client based on the DO OpenAPIv3 specification
Python
75
star
43

sample-django

Django sample app for DigitalOcean App Platform
Python
74
star
44

logtalez

logtalez is a minimal command line client (and API) for retrieving log streams from the rsyslog logging daemon over zeromq.
Go
73
star
45

do-markdownit

Markdown-It plugin for the DigitalOcean Community.
JavaScript
71
star
46

databases

66
star
47

sample-nodejs

⛵ App Platform sample Node.js application.
JavaScript
60
star
48

debian-sys-maint-roll-passwd

Script to update password for MySQL user "debian-sys-maint"
Shell
58
star
49

sample-nextjs

⛵ App Platform sample Next.js application.
JavaScript
57
star
50

sample-python

⛵ App Platform sample Python application.
Python
52
star
51

vmtop

Real-time monitoring of KVM/Qemu VMs
Python
52
star
52

kubecon-2022-doks-workshop

HCL
48
star
53

sample-flask

Sample Flask Application to be deployed on DigitalOcean's App Platform
HTML
45
star
54

sample-laravel

⛵ App Platform sample Laravel application.
PHP
43
star
55

pgremapper

CLI tool for manipulating Ceph's upmap exception table.
Go
43
star
56

k8s-staticroute-operator

Create static routes for your k8s nodes using CRDs.
Python
42
star
57

sample-functions-nodejs-qrcode

HTML
39
star
58

tos

DigitalOcean's Terms of Service agreement
37
star
59

sample-monorepo

Sample mono repo app (with multiple components) on the DigitalOcean App Platform.
Go
36
star
60

sample-golang

⛵ App Platform sample Golang application.
Go
36
star
61

droplet-agent

Droplet Agent is the daemon that runs on customer droplets to enable some features such as web console access.
Go
36
star
62

openvswitch_exporter

Command openvswitch_exporter implements a Prometheus exporter for Open vSwitch.
Go
32
star
63

sample-php

⛵ App Platform sample PHP application.
PHP
32
star
64

mastodon-on-kubernetes

Setting up Mastodon on DigitalOcean Kubernetes
HCL
30
star
65

sample-html

⛵ App Platform sample HTML application.
HTML
30
star
66

sample-functions-nodejs-helloworld

JavaScript
30
star
67

sample-functions-python-jokes

Python
30
star
68

flipop

Floating IP Controller for Kubernetes
Go
29
star
69

ansible-collection

DigitalOcean Ansible Collection
Python
28
star
70

sample-functions-golang-helloworld

Go
28
star
71

go-metadata

Go client for the metadata API.
Go
27
star
72

sample-react

⛵ App Platform sample React application.
JavaScript
27
star
73

marketplace-pi-hole-vpn

Pi-hole VPN image for Marketplace with Unbound & Wireguard
Shell
26
star
74

omniauth-digitalocean

DigitalOcean OAuth2 Strategy for OmniAuth
Ruby
26
star
75

github-changelog-generator

A tool to generate changelog entries from GitHub repositories.
Go
25
star
76

sample-functions-python-helloworld

Python
22
star
77

terraform-vault-github-oidc

Terraform module to configure Vault for GitHub OIDC authentication from Action runners.
HCL
22
star
78

sample-push-to-deploy-doks

Push-to-deploy example using DOCR and DOKS
Python
21
star
79

netbox-ip-controller

A Kubernetes controller to import the IP addresses and metadata of pods and services into NetBox.
Go
20
star
80

sample-expressjs

⛵ App Platform sample Express.js application.
19
star
81

terraform-provider-sendgrid

Sendgrid Terraform Provider
Go
19
star
82

sample-nuxtjs

⛵ App Platform sample Nuxt.js application.
Vue
19
star
83

sample-vuejs

⛵ App Platform sample Vue.js application.
Vue
17
star
84

production-ready-kubernetes-workshop

The repository for DigitalOcean's Production Ready Kubernetes Workshop
Python
16
star
85

sample-functions-python-twilio-sms

Sending sms via Twilio
Python
16
star
86

sample-rails

⛵ App Platform sample Ruby on Rails application.
Ruby
15
star
87

sample-functions-php-numberstowords

PHP
15
star
88

sample-functions-php-helloworld

A PHP helloworld sample function for Cloud Functions
PHP
14
star
89

sample-hugo

⛵ App Platform sample Hugo application.
14
star
90

github-pr-resource

Github pull request resource for Concourse
Go
13
star
91

sample-functions-python-sendgrid-email

Sending emails via Sendgrid API
Python
13
star
92

icingaweb2-module-netboximport

Icinga2 Director integration for Netbox
PHP
12
star
93

docker-shipit-engine

Docker image for https://github.com/Shopify/shipit-engine
Ruby
11
star
94

sample-functions-golang-presigned-url

Creating a presigned url for DO's Spaces
Go
10
star
95

digitalocean-ceph-lab

Terraform and Ansible automation to provision and configure a Ceph test environment on DigitalOcean.
HCL
10
star
96

k8s-adoption-journey

Hands-on tutorial for going from Day-1 to production on DigitalOcean Kubernetes. Goes with "Kubernetes Adoption Journey" document.
Python
9
star
97

sample-laravel-api

⛵ App Platform sample Laravel API application.
PHP
9
star
98

gnulib

A mirror of the gnulib portability and testing suite for internal builds that use it as a submodule
C
8
star
99

serverless-jamstack

Contains sample code for a serverless Jamstack tutorial published on docs.digitalocean.com
JavaScript
8
star
100

sample-gatsby

⛵ App Platform sample Gatsby application.
CSS
8
star