turbot/flowpipe turbot/flowpipe

  • Stars
    star
    262
  • Rank 156,136 (Top 4 %)
  • Language
    Go
  • License
    GNU Affero Genera...
  • Created over 1 year ago
  • Updated 8 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
turbot/flowpipe
github

turbot

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Flowpipe is a cloud scripting engine. Automation and workflow to connect your clouds to the people, systems and data that matters.

libraries   pipelines   maintained by

Flowpipe is a cloud scripting engine. It provides automation and workflow to connect your clouds to the people, systems and data that matter.

Connect people and tools. Connect your cloud data to people and systems using email, chat & APIs. Workflow steps can even run containers, custom functions, and more.

Orchestrate your cloud. Build simple steps into complex workflows. Run and test locally. Compose solutions across clouds using open source mods.

Respond to events. Run workflows manually or on a schedule. Trigger pipelines from webhooks or changes in data.

Code, not clicks. Build and deploy DevOps workflows like infrastructure. Code in HCL and deploy from version control.

Demo Time!

Watch on YouTube →

Flowpipe Demo

Getting Started

The downloads page shows you how but tl;dr:

Linux or WSL

sudo /bin/sh -c "$(curl -fsSL https://flowpipe.io/install/flowpipe.sh)"

MacOS

brew tap turbot/tap
brew install flowpipe

Now, create and run your first pipeline →

Libraries and samples

Flowpipe library mods are available for services including AWS, Azure, GCP, GitHub, Jira, Okta, PagerDuty, SendGrid, Slack, Teams, Zendesk ... and many more!

Check out Flowpipe samples for ready-to-run samples that use various library mods.

Developing

Developing Flowpipe

Prerequisites:

  • Golang Version 1.21 or higher.

Clone github.com/flowpipe and github.com/turbot/pipe-fittings repositories:

git clone [email protected]:turbot/flowpipe
git clone [email protected]:turbot/pipe-fittings
cd flowpipe

Build will build flowpipe binary in the current directory:

make

Check the version:

./flowpipe --version
Flowpipe v0.0.1-local.1

Flowpipe local version will always be v0.0.1-local.1. The real version is generated during the release process.

Try it!

./flowpipe pipeline list --mod-location ./internal/es/estest/test_suite_mod/
MOD                   NAME                                                                                                        DESCRIPTION
mod.mod_depend_a      mod_depend_a.pipeline.echo_one_depend_a
mod.test_suite_mod    test_suite_mod.pipeline.any_param
mod.test_suite_mod    test_suite_mod.pipeline.bad_email_with_expr
mod.test_suite_mod    test_suite_mod.pipeline.bad_http_ignored                                                                    Ignored bad HTTP step.
mod.test_suite_mod    test_suite_mod.pipeline.bad_http_not_ignored                                                                Pipeline with a HTTP step that will fail. Error is not ignored.
</snip>

Now run a simple pipeline:

./flowpipe pipeline run --mod-location ./internal/es/estest/test_suite_mod/ simple
[flowpipe] Execution ID: exec_clsm62ko47mjp5f74730
[simple] Starting pipeline
[simple.echo] Starting transform
[simple.echo] Output echo_1 = echo 1
[simple.echo] Output echo_2 = echo 2
[simple.echo] Complete 2ms
[simple] Output val = Hello World
[simple] Complete 12ms exec_clsm62ko47mjp5f74730

That's it! You're ready to start developing.

There are other third party tools that are required for the full development suite. These are not required for basic development. To make development easy, we have built a DevContainer that has all the required tools installed. See the Developer Setup section for more details.

Developer Setup

  1. Install Docker

  2. Install VS Code

  3. Pull the Dev Container: docker pull ghcr.io/turbot/flowpipe-devcontainer:latest

  4. In VS Code install devcontainer extension.

  5. Open flowpipe in Dev Containers: Open Folder in Container... option.

  6. Run make to build the Flowpipe binary.

Open Source & Contributing

This repository is published under the AGPL 3.0 license. Please see our code of conduct. Contributors must sign our Contributor License Agreement as part of their first pull request. We look forward to collaborating with you!

Flowpipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #flowpipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

More Repositories

1

steampipe

Use SQL to instantly query your cloud services (AWS, Azure, GCP and more). Open source CLI. No DB required.
Go
4,660
star
2

steampipe-mod-aws-compliance

Run individual controls or full compliance benchmarks for CIS, PCI, NIST, HIPAA and more across all of your AWS accounts using Steampipe.
HCL
278
star
3

steampipe-plugin-aws

Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.
Go
128
star
4

steampipe-mod-aws-perimeter

Is your AWS perimeter secure? Use Powerpipe and Steampipe to check your AWS accounts for public resources, resources shared with untrusted accounts, insecure network configurations and more.
HCL
105
star
5

steampipe-mod-aws-insights

View dashboards and reports across all of your AWS accounts using Steampipe.
HCL
80
star
6

powerpipe

Powerpipe: Dashboards for DevOps. Visualize cloud configurations. Assess security posture against a massive library of benchmarks. Build custom dashboards with code.
TypeScript
69
star
7

steampipe-mod-aws-thrifty

Are you a Thrifty AWS dev? This Steampipe mod checks your AWS accounts for unused and under-utilized resources.
HCL
64
star
8

steampipe-mod-zoom-compliance

Run individual configuration, compliance and security controls or full compliance benchmarks for CIS for Zoom.
HCL
61
star
9

steampipe-plugin-github

Use SQL to instantly query repositories, users, gists and more from GitHub. Open source CLI. No DB required.
Go
43
star
10

steampipe-sqlite

Steampipe SQLite is a zero-ETL engine for SQLite. Virtual tables translate queries into live API calls for cloud services and APIs. Hundreds of plugins with thousands of documented examples.
Go
41
star
11

steampipe-postgres-fdw

Postgres FDW for Steampipe
Go
40
star
12

steampipe-mod-azure-compliance

Run individual configuration, compliance and security controls or full compliance benchmarks for CIS, HIPAA HITRUST, NIST, and more across all of your Azure subscriptions using Steampipe.
HCL
39
star
13

steampipe-samples

Examples, samples, snippets and scripts to use with Steampipe.
HCL
38
star
14

steampipe-mod-github-sherlock

Interrogate your GitHub resources with the help of the world's greatest detectives: Steampipe + Sherlock.
HCL
34
star
15

steampipe-plugin-kubernetes

Use SQL to instantly query Kubernetes API resources. Open source CLI. No DB required.
Go
31
star
16

steampipe-plugin-gcp

Use SQL to instantly query GCP resources across regions, projects and organizations. Open source CLI. No DB required.
Go
28
star
17

steampipe-plugin-azure

Use SQL to instantly query Azure resources across regions and subscriptions. Open source CLI. No DB required.
Go
25
star
18

steampipe-plugin-shodan

Use SQL to instantly query host, DNS and exploit information using Shodan. Open source CLI. No DB required.
Go
24
star
19

steampipe-mod-kubernetes-insights

View dashboards and reports across all of your Kubernetes clusters using Steampipe.
HCL
24
star
20

steampipe-mod-gcp-compliance

Run individual configuration, compliance and security controls or full compliance benchmarks for CIS across all of your GCP projects using Steampipe.
HCL
24
star
21

steampipe-mod-kubernetes-compliance

Run individual controls or full compliance benchmarks for NSA CISA Kubernetes Hardening Guidance across all of your Kubernetes clusters using Steampipe.
HCL
23
star
22

steampipe-plugin-virustotal

Use SQL to instantly query file, domain, URL and IP scanning results from VirusTotal.
Go
21
star
23

guardrails-samples

Provides teams using Turbot Guardrails automation and configuration-as-code examples for effective management of Guardrails for their organization.
Python
21
star
24

steampipe-plugin-terraform

Use SQL to instantly query resources, data sources and more from Terraform code. Open source CLI. No DB required.
Go
20
star
25

steampipe-plugin-sdk

Steampipe Plugin SDK provides a simple abstraction layer for all Steampipe Plugins
Go
19
star
26

steampipe-plugin-oci

Use SQL to instantly query Oracle Cloud resources across regions and accounts. Open source CLI. No DB required.
Go
17
star
27

steampipe-mod-terraform-aws-compliance

Run compliance and security controls to detect Terraform AWS resources deviating from security best practices prior to deployment.
HCL
17
star
28

steampipe-plugin-stripe

Use SQL to instantly query customers, products, invoices and more from Stripe. Open source CLI. No DB required.
Go
16
star
29

steampipe-mod-microsoft365-compliance

Run individual controls or full compliance benchmarks for CIS across all of your Microsoft 365 and Office 365 tenants using Powerpipe and Steampipe.
HCL
16
star
30

steampipe-plugin-reddit

Use SQL to instantly query Reddit posts, comments & more. Open source CLI. No DB required.
Go
15
star
31

steampipe-plugin-jira

Use SQL to instantly query Jira. Open source CLI. No DB required.
Go
14
star
32

steampipe-plugin-prometheus

Use SQL to instantly query Prometheus metrics, alerts, labels and more. Open source CLI. No DB required.
Go
14
star
33

steampipe-plugin-whois

Use SQL to instantly query WHOIS. Open source CLI. No DB required.
Go
14
star
34

steampipe-plugin-net

Use SQL to instantly query DNS records, certificates and other network information. Open source CLI. No DB required.
Go
14
star
35

steampipe-plugin-mastodon

Use SQL to instantly query Mastodon resources. Open source CLI. No DB required.
Go
13
star
36

steampipe-plugin-code

Use SQL to instantly query secrets and more from source code. Open source CLI. No DB required.
Go
13
star
37

flowpipe-samples

Flowpipe sample mods, demonstrating common pipeline patterns and how to use the Flowpipe library mods. Run directly or use as examples for your own pipelines.
HCL
13
star
38

steampipe-plugin-datadog

Use SQL to instantly query Datadog resources across accounts. Open source CLI. No DB required.
Go
12
star
39

steampipe-plugin-linkedin

Use SQL to instantly query LinkedIn for profiles, companies, connections & more. Open source CLI. No DB required.
Go
11
star
40

steampipe-mod-mastodon-insights

View dashboards and reports across your Mastodon resources using Steampipe.
HCL
11
star
41

steampipe-plugin-csv

Use SQL to query data from CSV files. Open source CLI. No DB required.
Go
11
star
42

steampipe-plugin-scaleway

Use SQL to instantly query instances, networks, databases, and more from Scaleway. Open source CLI. No DB required.
Go
11
star
43

steampipe-plugin-hackernews

Use SQL to instantly query stories, users and other items from Hacker News. Open source CLI. No DB required.
Go
10
star
44

steampipe-plugin-slack

Use SQL to instantly query users, channels, emoji and more from your Slack workspace. Open source CLI. No DB required.
Go
10
star
45

steampipe-plugin-googledirectory

Use SQL to instantly query users, groups, domains and more from Google Directory. Open source CLI. No DB required.
Go
9
star
46

steampipe-docs

Steampipe documentation content in markdown format. Automatically published to steampipe.io.
9
star
47

pipe-fittings

Shared components for use across pipe projects.
Go
9
star
48

steampipe-mod-aws-well-architected

Are AWS Well-Architected best practices being followed? Use Powerpipe and Steampipe to check if your AWS accounts are following best practices from each lens and pillar.
HCL
9
star
49

steampipe-mod-azure-thrifty

Are you a Thrifty Azure dev? This Steampipe mod checks your Azure subscription(s) for unused and under-utilized resources.
HCL
9
star
50

steampipe-plugin-alicloud

Use SQL to instantly query Alibaba Cloud resources across regions and accounts. Open source CLI. No DB required.
Go
9
star
51

steampipe-plugin-googleworkspace

Use SQL to instantly query calendar events, drive files, gmail messages, and more from Google Workspace. Open source CLI. No DB required.
Go
9
star
52

steampipe-plugin-ipstack

Use SQL to instantly query IP geolocation and more from ipstack. Open source CLI. No DB required.
Go
9
star
53

steampipe-mod-azure-insights

View dashboards and reports across all of your Azure subscriptions using Steampipe.
HCL
8
star
54

steampipe-plugin-turbot

Use SQL to instantly query the Turbot CMDB. Open source CLI. No DB required.
Go
8
star
55

steampipe-plugin-zendesk

Use SQL to instantly query Zendesk. Open source CLI. No DB required.
Go
8
star
56

steampipe-mod-gcp-insights

View dashboards and reports across all of your GCP projects using Steampipe.
HCL
8
star
57

steampipe-mod-github-compliance

Run individual controls or full compliance benchmarks for across all of your GitHub resources using Powerpipe and Steampipe.
HCL
8
star
58

steampipe-plugin-digitalocean

Use SQL to instantly query droplets, VPCs, users and more from DigitalOcean. Open source CLI. No DB required.
Go
8
star
59

steampipe-plugin-cloudflare

Use SQL to instantly query accounts, zones and more from Cloudflare. Open source CLI. No DB required.
Go
8
star
60

steampipe-mod-aws-tags

Is your AWS tagging strategy following best practice? This Steampipe mod checks if your AWS resource tags are set correctly to help you manage them effectively.
HCL
8
star
61

steampipe-plugin-twitter

Use SQL to instantly query tweets, users and followers from Twitter. Open source CLI. No DB required.
Go
8
star
62

steampipe-plugin-exec

Use SQL to instantly query & run shell commands on local & remote servers. Open source CLI. No DB required.
Go
8
star
63

steampipe-plugin-trivy

Use SQL to instantly query advisories, vulnerabilities, packages, findings and more using Trivy. Open source CLI. No DB required.
Go
8
star
64

steampipe-plugin-tfe

Use SQL to query workspaces, runs and more from Terraform Cloud/Enterprise. Open source CLI. No DB required.
Go
7
star
65

steampipe-plugin-azuread

Use SQL to instantly query groups, service principals, users and more from Azure Active Directory. Open source CLI. No DB required.
Go
7
star
66

steampipe-plugin-databricks

Use SQL to instantly query Databricks resources. Open source CLI. No DB required.
Go
7
star
67

steampipe-plugin-openai

Use SQL to instantly query OpenAI for completions, models & more. Open source CLI. No DB required.
Go
7
star
68

steampipe-plugin-docker

Use SQL to instantly query Dockerfile commands and more from Docker. Open source CLI. No DB required.
Go
7
star
69

terraform-provider-turbot

Terraform Turbot provider
Go
7
star
70

steampipe-plugin-supabase

Use SQL to instantly query Supabase resources. Open source CLI. No DB required.
Go
7
star
71

steampipe-plugin-finance

Use SQL to instantly query financial data including quotes (equities, cryptocurrency, etc) and US public company information. Open source CLI. No DB required.
Go
7
star
72

steampipe-plugin-crowdstrike

Use SQL to instantly query CrowdStrike resources. Open source CLI. No DB required.
Go
7
star
73

steampipe-plugin-googlesheets

Use SQL to query spreadsheets, sheets, and cell data from Google Sheets. Open source CLI. No DB required.
Go
7
star
74

steampipe-mod-gcp-thrifty

Are you a Thrifty GCP dev? This Steampipe mod checks your GCP project(s) for unused and under-utilized resources.
HCL
7
star
75

steampipe-plugin-steampipe

Use SQL to instantly query plugin metadata from the Steampipe Hub. Open source CLI. No DB required.
Go
7
star
76

steampipe-mod-alicloud-compliance

Run individual configuration, compliance and security controls or full compliance benchmarks for CIS across all of your Alibaba Cloud accounts using Steampipe.
HCL
6
star
77

steampipe-plugin-shopify

Use SQL to instantly query Shopify products, orders and more. Open source CLI. No DB required.
Go
6
star
78

homebrew-tap

Turbot's Homebrew Tap for distribution of Steampipe and other tools.
Ruby
6
star
79

steampipe-plugin-ibm

Use SQL to instantly query instances, networks, users and more from IBM Cloud. Open source CLI. No DB required.
Go
6
star
80

steampipe-plugin-chaos

Chaos Plugin for testing Steampipe with the craziest edge cases we can think of. Open source CLI. No DB required.
Go
6
star
81

steampipe-plugin-bitbucket

Use SQL to instantly query Bitbucket. Open source CLI. No DB required.
Go
6
star
82

steampipe-plugin-openapi

Use SQL to instantly query resources from OpenAPI. Open source CLI. No DB required.
Go
6
star
83

steampipe-plugin-rss

Use SQL to instantly query RSS channels and Atom Feeds. Open source CLI. No DB required.
Go
6
star
84

steampipe-plugin-linode

Use SQL to query instances, domains and more from Linode. Open source CLI. No DB required.
Go
6
star
85

steampipe-plugin-crtsh

Use SQL to instantly query crt.sh for certificates, log entries and more. Open source CLI. No DB required.
Go
6
star
86

steampipe-plugin-onepassword

Use SQL to instantly query 1Password vaults, items, files & more. Open source CLI. No DB required.
Go
6
star
87

steampipe-plugin-ldap

Use SQL to instantly query users, groups, OUs and more from LDAP. Open source CLI. No DB required.
Go
6
star
88

steampipe-action-check

Run Steampipe checks against your Infrastructure as Code, deployed infrastructure, and more
JavaScript
6
star
89

steampipe-plugin-okta

Use SQL to instantly query users, groups, applications and more from Okta. Open source CLI. No DB required.
Go
6
star
90

steampipe-plugin-microsoft365

Use SQL to instantly query calendars, contacts, drives, mailboxes and more from Microsoft 365. Open source CLI. No DB required.
Go
6
star
91

flowpipe-mod-aws

AWS pipeline library for the Flowpipe cloud scripting engine. Automation and workflows to connect AWS to the people, systems and data that matters.
HCL
6
star
92

steampipe-plugin-salesforce

Use SQL to instantly query Salesforce resources. Open source CLI. No DB required.
Go
5
star
93

steampipe-mod-oci-compliance

Run individual configuration, compliance and security controls or full compliance benchmarks for CIS across all of your Oracle Cloud Infrastructure accounts using Steampipe.
HCL
5
star
94

steampipe-plugin-grafana

Use SQL to instantly query dashboards, data sources, users and more from Grafana. Open source CLI. No DB required.
Go
5
star
95

guardrails-cli

Turbot Guardrails Command Line Interface (CLI)
Shell
5
star
96

steampipe-plugin-heroku

Use SQL to query apps, dynos and more from Heroku. Open source CLI. No DB required.
Go
5
star
97

steampipe-mod-digitalocean-thrifty

Are you a Thrifty DigitalOcean dev? This Steampipe mod checks your DigitalOcean account(s) for unused and under-utilized resources.
HCL
5
star
98

steampipe-plugin-abuseipdb

Use SQL to instantly query IP abuse scores and more from AbuseIPDB. Open source CLI. No DB required.
Go
5
star
99

steampipe-plugin-pagerduty

Use SQL to instantly query resources from PagerDuty. Open source CLI. No DB required.
Go
5
star
100

steampipe-plugin-jenkins

Use SQL to instantly query Jenkins resources. Open source CLI. No DB required.
Go
5
star