• Stars
    star
    252
  • Rank 161,312 (Top 4 %)
  • Language
    Python
  • License
    Other
  • Created almost 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Helm chart to install Apache Airflow on Kubernetes

Astronomer's Helm Chart for Apache Airflow

This chart will bootstrap an Airflow deployment on a Kubernetes cluster using the Helm package manager.

The version of this chart does not correlate to any other component. If it happens to align with OSS airflow that is just a coincidence. Users should not expect feature parity between OSS airflow chart and the Astronomer airflow-chart for identical version numbers.

Quickstart

To install this helm chart remotely (using helm 3)

kubectl create namespace airflow

helm repo add astronomer https://helm.astronomer.io
helm install airflow --namespace airflow astronomer/airflow

To install this repository from source

kubectl create namespace airflow
helm install --namespace airflow .

Prerequisites

  • Kubernetes 1.12+
  • Helm 3.6+
  • PV provisioner support in the underlying infrastructure

Installing the Chart

To install the chart with the release name my-release:

helm install --name my-release .

The command deploys Airflow on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Upgrading the Chart

First, look at the updating documentation to identify any backwards-incompatible changes.

To upgrade the chart with the release name my-release:

helm upgrade --name my-release .

Uninstalling the Chart

To uninstall/delete the my-release deployment:

helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Updating DAGs

Bake DAGs in Docker image

The recommended way to update your DAGs with this chart is to build a new docker image with the latest code (docker build -t my-company/airflow:8a0da78 .), push it to an accessible registry (docker push my-company/airflow:8a0da78), then update the Airflow pods with that image:

helm upgrade my-release . \
  --set images.airflow.repository=my-company/airflow \
  --set images.airflow.tag=8a0da78

Docker Images

Parameters

The complete list of parameters supported by the community chart can be found on the Parameteres Reference page, and can be set under the airflow key in this chart.

The following tables lists the configurable parameters of the Astronomer chart and their default values.

Parameter Description Default
ingress.enabled Enable Kubernetes Ingress support false
ingress.acme Add acme annotations to Ingress object false
ingress.tlsSecretName Name of secret that contains a TLS secret ~
ingress.webserverAnnotations Annotations added to Webserver Ingress object {}
ingress.flowerAnnotations Annotations added to Flower Ingress object {}
ingress.baseDomain Base domain for VHOSTs ~
ingress.auth.enabled Enable auth with Astronomer Platform true
extraObjects Extra K8s Objects to deploy (these are passed through tpl). More about Extra Objects. []
sccEnabled Enable security context constraints required for OpenShift false
authSidecar.enabled Enable authSidecar false
authSidecar.repository The image for the auth sidecar proxy nginxinc/nginx-unprivileged
authSidecar.tag The image tag for the auth sidecar proxy stable
authSidecar.pullPolicy The K8s pullPolicy for the the auth sidecar proxy image IfNotPresent
authSidecar.port The port the auth sidecar exposes 8084
gitSyncRelay.enabled Enables git sync relay feature. False
gitSyncRelay.repo.url Upstream URL to the git repo to clone. ~
gitSyncRelay.repo.branch Branch of the upstream git repo to checkout. main
gitSyncRelay.repo.depth How many revisions to check out. Leave as default 1 except in dev where history is needed. 1
gitSyncRelay.repo.wait Seconds to wait before pulling from the upstream remote. 60
gitSyncRelay.repo.subPath Path to the dags directory within the git repository. ~

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

helm install --name my-release \
  --set executor=CeleryExecutor \
  --set enablePodLaunching=false .

Walkthrough using kind

Install kind, and create a cluster

We recommend testing with Kubernetes 1.25+, example:

kind create cluster --image kindest/node:v1.25.11

Confirm it's up:

kubectl cluster-info --context kind-kind

Add Astronomer's Helm repo

helm repo add astronomer https://helm.astronomer.io
helm repo update

Create namespace + install the chart

kubectl create namespace airflow
helm install airflow -n airflow astronomer/airflow

It may take a few minutes. Confirm the pods are up:

kubectl get pods --all-namespaces
helm list -n airflow

Run kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow to port-forward the Airflow UI to http://localhost:8080/ to confirm Airflow is working. Login as admin and password admin.

Build a Docker image from your DAGs

  1. Start a project using astro-cli, which will generate a Dockerfile, and load your DAGs in. You can test locally before pushing to kind with astro airflow start.

    mkdir my-airflow-project && cd my-airflow-project
    astro dev init
  2. Then build the image:

    docker build -t my-dags:0.0.1 .
  3. Load the image into kind:

    kind load docker-image my-dags:0.0.1
  4. Upgrade Helm deployment:

    helm upgrade airflow -n airflow \
        --set images.airflow.repository=my-dags \
        --set images.airflow.tag=0.0.1 \
        astronomer/airflow

Extra Objects

This chart can deploy extra Kubernetes objects (assuming the role used by Helm can manage them). For Astronomer Cloud and Enterprise, the role permissions can be found in the Commander role.

extraObjects:
  - apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      name: "{{ .Release.Name }}-somejob"
    spec:
      schedule: "*/10 * * * *"
      concurrencyPolicy: Forbid
      jobTemplate:
        spec:
          template:
            spec:
              containers:
                - name: myjob
                  image: ubuntu
                  command:
                    - echo
                  args:
                    - hello
              restartPolicy: OnFailure

Contributing

Check out our contributing guide!

License

Apache 2.0 with Commons Clause

More Repositories

1

dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Python
1,154
star
2

airflow-guides

Guides and docs to help you get up and running with Apache Airflow.
JavaScript
797
star
3

astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
Python
589
star
4

astronomer

Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes
Python
444
star
5

astro-cli

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer
Go
348
star
6

astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Python
346
star
7

ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
Python
192
star
8

airflow-dbt-demo

A repository of sample code to accompany our blog post on Airflow and dbt.
Python
165
star
9

airflow-provider-great-expectations

Great Expectations Airflow operator
Python
158
star
10

astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
Python
134
star
11

ap-airflow

Astronomer Core Docker Images
Jinja
102
star
12

airflow-data-quality-demo

A repository of sample code to show data quality checking best practices using Airflow.
Python
71
star
13

airflow-provider-sample

A template repo for building and releasing Airflow provider packages.
Python
69
star
14

airflow-example-dags

Sample Airflow DAGs
Python
60
star
15

webinar-dag-writing-best-practices

Python
48
star
16

airflow-quickstart

Get started with Apache Airflow. Check the README for instructions on how to run your first DAGs today. 🚀
Python
46
star
17

airflow-provider-kafka

A provider package for kafka
Python
37
star
18

docs

This repository contains all content and code for Astro and Astronomer Software documentation.
Python
36
star
19

telescope

Python
30
star
20

ray-airflow-demo

Jupyter Notebook
29
star
21

dynamic-dags-tutorial

Python
27
star
22

cosmos-demo

Demo DAGs that show how to run dbt Core in Airflow using Cosmos
Python
25
star
23

airflow-provider-mlflow

An MLflow Provider Package for Apache Airflow
Python
25
star
24

airflow-ui

TypeScript
24
star
25

starship

Python
22
star
26

astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Python
21
star
27

airflow-testing-guide

Python
20
star
28

deploy-action

Custom Github Actions
Python
20
star
29

webinar-demos

20
star
30

airflow-testing-skeleton

A skeleton project for testing Airflow code
Python
18
star
31

airflow-provider-fivetran-async

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
Python
18
star
32

airflow-covid-data

Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.
Python
16
star
33

airflow-provider-duckdb

A provider package for DuckDB
Python
14
star
34

astronomer-fab-securitymanager

Security Manager for the Astronomer Airflow distribution
Python
12
star
35

apache-airflow-providers-transfers

Python
11
star
36

airflow-dbt-elt

This repo contains DAGs demonstrating a variety of ELT patterns using Airflow along with dbt.
Python
11
star
37

ap-vendor

Astronomer Vendor Images
Dockerfile
11
star
38

intro-to-airflow-webinar

Python
10
star
39

airflow-guide-passing-data-between-tasks

Python
10
star
40

terraform-google-astronomer-gcp

Intended for internal use: deploys all infrastructure required for Astronomer to run on GCP
HCL
10
star
41

cs-tutorial-msteams-callbacks

Example DAGs demonstrating how to implement alerting and notifications via Microsoft Teams
Python
9
star
42

astro-provider-venv

Easily create and use Python Virtualenvs in Apache Airflow
Go
9
star
43

astronomer-airflow-scripts

Waits for Apache Airflow database migrations to complete.
Python
9
star
44

terraform-aws-astronomer-aws

Deploys all infrastructure required for Astronomer to run on AWS. For a complete deployment, see https://github.com/astronomer/terraform-aws-astronomer-enterprise
HCL
9
star
45

terraform-provider-astro

Astro Terraform Provider
Go
9
star
46

airflow-scheduling-tutorial

Python
8
star
47

cosmos-example

Python
8
star
48

2-4-example-dags

Python
7
star
49

mlflow-example

Python
7
star
50

2-6-example-dags

Python
7
star
51

airflow-provider-pulumi

Python
6
star
52

academy-genai

Python
6
star
53

kedro-ge-airflow

Python
6
star
54

registry-dag-template

A template repository for contributing DAGs to the Astronomer Registry.
Python
6
star
55

dynamic-task-mapping-tutorial

Python
6
star
56

apache-airflow-providers-alembic

Python
6
star
57

webinar-secrets-management

Python
5
star
58

custom-xcom-backend-tutorial

Jupyter Notebook
5
star
59

airflow-sql-tutorial

Python
5
star
60

terraform-kubernetes-astronomer

Deploy Astronomer on Kubernetes
HCL
5
star
61

cs-tutorial-slack-callbacks

Example DAGs demonstrating how to implement alerting and notifications via Slack
Python
5
star
62

airflow-dags

Example DAGs for Airflow 2.9
Python
5
star
63

airflow-analytics-plugin

Python
5
star
64

airflow-databricks-tutorial

Python
4
star
65

azure-operator-tutorials

Python
4
star
66

education-sandbox

Codespace with Airflow and the Astro CLI
Python
4
star
67

cross-dag-dependencies-tutorial

Python
4
star
68

apache-airflow-providers-isolation

Python
4
star
69

terraform

Getting phased out - please use astronomer/terraform-* modules to track issues
HCL
4
star
70

cdc-cloudsql-airflow-demo

A repository of sample code to accompany our blog post on Change Data Capture and CloudSQL
Python
4
star
71

airflow-ldap-example

Example project for configuring opern source Airflow version with LDAP. Includes prepopulated OpenLDAP server
Python
4
star
72

airflow-wandb-demo

Python
3
star
73

spectra

Reusable UI components for Astronomer projects.
JavaScript
3
star
74

airflow-llm-demo

Python
3
star
75

2-7-example-dags

Python
3
star
76

airflow-snowpark-containers-demo

Python
3
star
77

astro-example-dags

Python
3
star
78

airflow-talend-tutorial

Tutorial for how to use Astronomer+Airflow with Talend. Contains reference DAGs and other supporting materials.
Python
3
star
79

azure_demo

Python
3
star
80

astro-gcp-onboarding

The script needed to set up a customer's Google Cloud Project for an Astro activation
Shell
3
star
81

greenplum-airflow-demo

Python
3
star
82

cs-astro-onboarding

Python
3
star
83

homebrew-tap

Homebrew Formulae to @astronomer binaries, powered by @astronomer
Ruby
3
star
84

debugging-dags-webinar

A repository containing the DAGs shown in the Debugging DAGs webinar on 2023-01-31.
Python
3
star
85

migrate-to-astro

Customer facing utilities to help customers migrating from Software/Nebula to Astro
Python
2
star
86

airflow-connection-docs

Guides and structured metadata about Airflow connections
Python
2
star
87

pass-data-between-tasks-webinar

The repository for example DAGs shown in the 2023-04-11 Astronomer webinar on passing data between tasks.
Python
2
star
88

pyconuk2022

Materials related to the PyCon UK Apache Airflow & Astro SDK workshop
Python
2
star
89

ds-ml-example-dags

Python
2
star
90

react-graphql-code-challenge

Code challenge for Front-End developers applying at Astronomer
JavaScript
2
star
91

airflow-sagemaker-tutorial

Python
2
star
92

databricks-ml-example

Jupyter Notebook
2
star
93

airflow_101_webinar

Repository for the Airflow 101 webinar on June 6th 2023.
Python
2
star
94

pagerduty_airflow_integration_benefits

Repo hosting PagerDuty + Airflow Integration Benefits Doc
2
star
95

sagemaker-batch-inference

Jupyter Notebook
2
star
96

airflow-adf-integration

An example DAG for orchestrating Azure Data Factory pipelines with Apache Airflow.
Python
2
star
97

astro-dbt-provider-tutorial-example

Example code for the dbt core Learn tutorial. The Astro dbt provider, also known as Cosmos, is a tool automatically integrate dbt models into your Airflow DAGs.
Python
2
star
98

cs-tutorial-reporting

How to Load reporting database for Airflow DAGS, DAG Runs, and Task Instances
Python
2
star
99

cosmos-dev

Python
2
star
100

llm-dags-dashboard

Repository for displaying LLM DAG runs status
HTML
2
star