• Stars
    star
    348
  • Rank 121,840 (Top 3 %)
  • Language
    Go
  • License
    Other
  • Created almost 7 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

CLI that makes it easy to create, test and deploy Airflow DAGs to Astronomer

Astro CLI Release GoDoc Go Report Card codecov

The Astro CLI is a command-line interface for data orchestration. It allows you to get started with Apache Airflow quickly and it can be used with all Astronomer products.

Usage

astro [command]

Core commands

  • login Log in to the Astro CLI
  • logout Log out of the Astro CLI
  • dev init Initialize an Astro project in an empty local directory
  • dev start Build your Astro project into a Docker image and spin up a local Docker container for each Airflow component
  • dev stop Pause all Docker containers running your local Airflow environment
  • dev restart Stop your Airflow environment, rebuild your Astro project into a Docker image, and restart your Airflow environment with the new Docker image
  • deploy Deploy code to a Deployment on Astro
  • deployment Manage your Deployments running on Astronomer
  • dev Run your Astro project locally
  • help Help about any Astro CLI command
  • version Show the running version of the Astro CLI
  • workspace Manage Astronomer Workspaces

For a list of available Astro CLI commands, see the Astro CLI command reference.

Install the Astro CLI

Use these instructions to install, upgrade, or uninstall the Astro CLI.

Prerequisites

To install the Astro CLI on Mac, you'll need:

To install the Astro CLI on Windows, you'll need:

To install the Astro CLI on Windows with the Windows Package Manager winget command-line tool, you'll need:

To install the Astro CLI on Linux, you'll need:

Latest version

Mac

brew install astro

Windows

  1. Go to the Releases page of the Astro CLI GitHub repository, scroll to a CLI version, and then download the .exe file that matches the CPU architecture of your machine.

    For example, to install v1.0.0 of the Astro CLI on a Windows machine with an AMD 64 architecture, download astro_1.0.0-converged_windows_amd64.exe.

  2. Rename the file astro.exe.

  3. Add the filepath for the directory containing the new astro.exe as a PATH environment variable. For example, if astro.exe is stored in C:\Users\username\astro.exe, you add C:\Users\username as your PATH environment variable. To learn more about configuring the PATH environment variable, see How do I set or change the PATH system variable?.

  4. Restart your machine.

Windows with winget

Starting with Astro CLI version 1.6, you can use the Windows Package Manager winget command-line tool to install the Astro CLI.

Open Windows PowerShell as an administrator and then run the following command:

winget install -e --id Astronomer.Astro

Linux

curl -sSL install.astronomer.io | sudo bash -s

Specific version

Mac

To install a specific version of the Astro CLI, specify the version you want to install at the end of the command:

brew install astro@<major.minor.patch-version>

Windows

  1. Delete the existing astro.exe file on your machine.

  2. Go to the Releases page of the Astro CLI GitHub repository, scroll to a CLI version, and then download the .exe file that matches the CPU architecture of your machine.

    For example, to install v1.0.0 of the Astro CLI on a Windows machine with an AMD 64 architecture, download astro_1.0.0-converged_windows_amd64.exe.

  3. Rename the file astro.exe.

  4. Add the filepath for the directory containing the new astro.exe as a PATH environment variable. For example, if astro.exe is stored in C:\Users\username\astro.exe, you add C:\Users\username as your PATH environment variable. To learn more about configuring the PATH environment variable, see How do I set or change the PATH system variable?.

  5. Restart your machine.

Windows with winget

Starting with Astro CLI version 1.6, you can use the Windows Package Manager winget command-line tool to install a specific version of the Astro CLI.

To install a specific version of the Astro CLI, specify the version you want to install at the end of the command. For example, running the following command in Windows PowerShell as an administrator installs Astro CLI version 1.6:

winget install -e --id Astronomer.Astro -v 1.6.0

Linux

To install a specific version of the CLI, specify the version number as a flag at the end of the command. For example, to install v1.1.0 of the CLI, you would run:

curl -sSL install.astronomer.io | sudo bash -s -- v1.1.0

If you receive a mkdir error during installation, download and run the godownloader script locally using:

$ cat godownloader.sh | bash -s -- -b /usr/local/bin

Troubleshoot installation issues

If you encounter issues when installing the Astro CLI:

Get started

  1. Create a project

    $ mkdir hello-astro && cd hello-astro
    $ astro dev init
    
  2. Install the binary and Confirm the install worked:

    $ ./astro
    

    This generates a basic project directory:

    .
    β”œβ”€β”€ dags
    β”‚   β”œβ”€β”€ example_dag_advanced.py
    β”‚   β”œβ”€β”€ example_dag_basic.py
    β”œβ”€β”€ tests
    β”‚   β”œβ”€β”€ dags
    β”‚       β”œβ”€β”€ test_dag_integrity.py
    β”œβ”€β”€ Dockerfile
    β”œβ”€β”€ include
    β”œβ”€β”€ packages.txt
    β”œβ”€β”€ plugins
    └── requirements.txt
    

    DAGs can go in the dags folder, custom Airflow plugins in plugins, python packages needed can go in requirements.txt, and OS level packages can go in packages.txt.

  3. Run astro dev start to start a local version of airflow on your machine. This will spin up a few locally running docker containers - one for the airflow scheduler, one for the webserver, and one for postgres. (Run astro dev ps to verify).

Versions

Astro CLI versions are released regularly and use semantic versioning. Backwards compatibility between versions cannot be guaranteed. Compatibility is only guaranteed between matching minor versions of the platform and the Astro CLI. For example, Astro CLI 0.9.0` is guaranteed to be compatible with houston-api v0.9.x, but not houston-api v0.10.x.

Astronomer ships major, minor, and patch releases of the Astro CLI in the following format:

{MAJOR_RELEASE}.{MINOR_RELEASE}.{PATCH_RELEASE}

All Astro CLI releases prior to 1.0.0 are considered beta.

Change Log

Change log between each version can be found on the releases page

Debug

The Astro CLI includes a debug flag that allows you to view queries and internal logs. To enable it, you can pass --verbosity=debug in your commands, or you can add the following entry to your ~/.astro/config.yaml file:

verbosity: debug

Adding this entry to your ~/.astro/config.yaml file turns on debugging for all requests until you change it to info, or you remove it from the file.

Support

To resolve an issue, Astronomer recommends reviewing the Astronomer documentation first.

If you're unable to resolve your issue after reviewing the documentation, you can post a question on the Astronomer web forum or you can contact Astronomer support.

License

Apache 2.0 with Commons Clause

More Repositories

1

dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Python
1,154
star
2

airflow-guides

Guides and docs to help you get up and running with Apache Airflow.
JavaScript
797
star
3

astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
Python
589
star
4

astronomer

Helm Charts for the Astronomer Platform, Apache Airflow as a Service on Kubernetes
Python
444
star
5

astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Python
346
star
6

airflow-chart

A Helm chart to install Apache Airflow on Kubernetes
Python
252
star
7

ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
Python
192
star
8

airflow-dbt-demo

A repository of sample code to accompany our blog post on Airflow and dbt.
Python
165
star
9

airflow-provider-great-expectations

Great Expectations Airflow operator
Python
158
star
10

astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
Python
134
star
11

ap-airflow

Astronomer Core Docker Images
Jinja
102
star
12

airflow-data-quality-demo

A repository of sample code to show data quality checking best practices using Airflow.
Python
71
star
13

airflow-provider-sample

A template repo for building and releasing Airflow provider packages.
Python
69
star
14

airflow-example-dags

Sample Airflow DAGs
Python
60
star
15

webinar-dag-writing-best-practices

Python
48
star
16

airflow-quickstart

Get started with Apache Airflow. Check the README for instructions on how to run your first DAGs today. πŸš€
Python
46
star
17

airflow-provider-kafka

A provider package for kafka
Python
37
star
18

docs

This repository contains all content and code for Astro and Astronomer Software documentation.
Python
36
star
19

telescope

Python
30
star
20

ray-airflow-demo

Jupyter Notebook
29
star
21

dynamic-dags-tutorial

Python
27
star
22

cosmos-demo

Demo DAGs that show how to run dbt Core in Airflow using Cosmos
Python
25
star
23

airflow-provider-mlflow

An MLflow Provider Package for Apache Airflow
Python
25
star
24

airflow-ui

TypeScript
24
star
25

starship

Python
22
star
26

astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Python
21
star
27

airflow-testing-guide

Python
20
star
28

deploy-action

Custom Github Actions
Python
20
star
29

webinar-demos

20
star
30

airflow-testing-skeleton

A skeleton project for testing Airflow code
Python
18
star
31

airflow-provider-fivetran-async

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
Python
18
star
32

airflow-covid-data

Sample Airflow DAGs to load data from the CovidTracking API to Snowflake via an AWS S3 intermediary.
Python
16
star
33

airflow-provider-duckdb

A provider package for DuckDB
Python
14
star
34

astronomer-fab-securitymanager

Security Manager for the Astronomer Airflow distribution
Python
12
star
35

apache-airflow-providers-transfers

Python
11
star
36

airflow-dbt-elt

This repo contains DAGs demonstrating a variety of ELT patterns using Airflow along with dbt.
Python
11
star
37

ap-vendor

Astronomer Vendor Images
Dockerfile
11
star
38

intro-to-airflow-webinar

Python
10
star
39

airflow-guide-passing-data-between-tasks

Python
10
star
40

terraform-google-astronomer-gcp

Intended for internal use: deploys all infrastructure required for Astronomer to run on GCP
HCL
10
star
41

cs-tutorial-msteams-callbacks

Example DAGs demonstrating how to implement alerting and notifications via Microsoft Teams
Python
9
star
42

astro-provider-venv

Easily create and use Python Virtualenvs in Apache Airflow
Go
9
star
43

astronomer-airflow-scripts

Waits for Apache Airflow database migrations to complete.
Python
9
star
44

terraform-aws-astronomer-aws

Deploys all infrastructure required for Astronomer to run on AWS. For a complete deployment, see https://github.com/astronomer/terraform-aws-astronomer-enterprise
HCL
9
star
45

terraform-provider-astro

Astro Terraform Provider
Go
9
star
46

airflow-scheduling-tutorial

Python
8
star
47

cosmos-example

Python
8
star
48

2-4-example-dags

Python
7
star
49

mlflow-example

Python
7
star
50

2-6-example-dags

Python
7
star
51

airflow-provider-pulumi

Python
6
star
52

academy-genai

Python
6
star
53

kedro-ge-airflow

Python
6
star
54

registry-dag-template

A template repository for contributing DAGs to the Astronomer Registry.
Python
6
star
55

dynamic-task-mapping-tutorial

Python
6
star
56

apache-airflow-providers-alembic

Python
6
star
57

webinar-secrets-management

Python
5
star
58

custom-xcom-backend-tutorial

Jupyter Notebook
5
star
59

airflow-sql-tutorial

Python
5
star
60

terraform-kubernetes-astronomer

Deploy Astronomer on Kubernetes
HCL
5
star
61

cs-tutorial-slack-callbacks

Example DAGs demonstrating how to implement alerting and notifications via Slack
Python
5
star
62

airflow-dags

Example DAGs for Airflow 2.9
Python
5
star
63

airflow-analytics-plugin

Python
5
star
64

airflow-databricks-tutorial

Python
4
star
65

azure-operator-tutorials

Python
4
star
66

education-sandbox

Codespace with Airflow and the Astro CLI
Python
4
star
67

cross-dag-dependencies-tutorial

Python
4
star
68

apache-airflow-providers-isolation

Python
4
star
69

terraform

Getting phased out - please use astronomer/terraform-* modules to track issues
HCL
4
star
70

cdc-cloudsql-airflow-demo

A repository of sample code to accompany our blog post on Change Data Capture and CloudSQL
Python
4
star
71

airflow-ldap-example

Example project for configuring opern source Airflow version with LDAP. Includes prepopulated OpenLDAP server
Python
4
star
72

airflow-wandb-demo

Python
3
star
73

spectra

Reusable UI components for Astronomer projects.
JavaScript
3
star
74

airflow-llm-demo

Python
3
star
75

2-7-example-dags

Python
3
star
76

airflow-snowpark-containers-demo

Python
3
star
77

astro-example-dags

Python
3
star
78

airflow-talend-tutorial

Tutorial for how to use Astronomer+Airflow with Talend. Contains reference DAGs and other supporting materials.
Python
3
star
79

azure_demo

Python
3
star
80

astro-gcp-onboarding

The script needed to set up a customer's Google Cloud Project for an Astro activation
Shell
3
star
81

greenplum-airflow-demo

Python
3
star
82

cs-astro-onboarding

Python
3
star
83

homebrew-tap

Homebrew Formulae to @astronomer binaries, powered by @astronomer
Ruby
3
star
84

debugging-dags-webinar

A repository containing the DAGs shown in the Debugging DAGs webinar on 2023-01-31.
Python
3
star
85

migrate-to-astro

Customer facing utilities to help customers migrating from Software/Nebula to Astro
Python
2
star
86

airflow-connection-docs

Guides and structured metadata about Airflow connections
Python
2
star
87

pass-data-between-tasks-webinar

The repository for example DAGs shown in the 2023-04-11 Astronomer webinar on passing data between tasks.
Python
2
star
88

pyconuk2022

Materials related to the PyCon UK Apache Airflow & Astro SDK workshop
Python
2
star
89

ds-ml-example-dags

Python
2
star
90

react-graphql-code-challenge

Code challenge for Front-End developers applying at Astronomer
JavaScript
2
star
91

airflow-sagemaker-tutorial

Python
2
star
92

databricks-ml-example

Jupyter Notebook
2
star
93

airflow_101_webinar

Repository for the Airflow 101 webinar on June 6th 2023.
Python
2
star
94

pagerduty_airflow_integration_benefits

Repo hosting PagerDuty + Airflow Integration Benefits Doc
2
star
95

sagemaker-batch-inference

Jupyter Notebook
2
star
96

airflow-adf-integration

An example DAG for orchestrating Azure Data Factory pipelines with Apache Airflow.
Python
2
star
97

astro-dbt-provider-tutorial-example

Example code for the dbt core Learn tutorial. The Astro dbt provider, also known as Cosmos, is a tool automatically integrate dbt models into your Airflow DAGs.
Python
2
star
98

cs-tutorial-reporting

How to Load reporting database for Airflow DAGS, DAG Runs, and Task Instances
Python
2
star
99

cosmos-dev

Python
2
star
100

llm-dags-dashboard

Repository for displaying LLM DAG runs status
HTML
2
star