🧙 A modern replacement for Airflow.

Documentation   🌪️    Watch 2 min demo   🌊    Play with live tool   🔥    Get instant help

Integrate and synchronize data from 3rd party sources

Build real-time and batch pipelines to transform data using Python, SQL, and R

Run, monitor, and orchestrate thousands of pipelines without losing sleep

1️⃣ 🏗️

Build

Have you met anyone who said they loved developing in Airflow?
That’s why we designed an easy developer experience that you’ll enjoy.

Easy developer experience Start developing locally with a single command or launch a dev environment in your cloud using Terraform. Language of choice Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility. Engineering best practices built-in Each step in your pipeline is a standalone file containing modular code that’s reusable and testable with data validations. No more DAGs with spaghetti code.

↓

2️⃣ 🔮

Preview

Stop wasting time waiting around for your DAGs to finish testing.
Get instant feedback from your code each time you run it.

Interactive code Immediately see results from your code’s output with an interactive notebook UI. Data is a first-class citizen Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use. Collaborate on cloud Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.

↓

3️⃣ 🚀

Launch

Don’t have a large team dedicated to Airflow?
Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.

Fast deploy Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates. Scaling made simple Transform very large datasets directly in your data warehouse or through a native integration with Spark. Observability Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.

🧙 Intro

Mage is an open-source data pipeline tool for transforming and integrating data.

1. Quick start
2. Demo
3. Tutorials
4. Documentation
5. Features
6. Core design principles
7. Core abstractions
8. Contributing

🏃‍♀️ Quick start

You can install and run Mage using Docker (recommended), pip, or conda.

Install using Docker

1. Create a new project and launch tool (change demo_project to any other name if you want):

   docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai \
     /app/run_app.sh mage start demo_project

   • If you want to run Mage locally on a different port, change the first port after -p in the command above. For example, to change the port to 6790, run:

   docker run -it -p 6790:6789 -v $(pwd):/home/src mageai/mageai \
     /app/run_app.sh mage start demo_project

   _{Want to use Spark or other integrations? Read more about integrations.}

2. Open http://localhost:6789 in your browser and build a pipeline.

• If you changed the Docker port for running Mage locally, go to the url http://127.0.0.1:[port] (e.g. http://127.0.0.1:6790) in your browser to view the pipelines dashboard.

Using pip or conda

1. Install Mage

   (a) To the current virtual environment:

   pip install mage-ai

   or

   conda install -c conda-forge mage-ai

   (b) To a new virtual environment (e.g., myenv):

   python3 -m venv myenv
   source myenv/bin/activate
   pip install mage-ai

   or

   conda create -n myenv -c conda-forge mage-ai
   conda activate myenv

   _{For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.}

   _{If you run into errors, please see Install errors.}

2. Create new project and launch tool (change demo_project to any other name if you want):

   mage start demo_project

3. Open http://localhost:6789 in your browser and build a pipeline.

🎮 Demo

Live demo

Build and run a data pipeline with our demo app.

WARNING

The live demo is public to everyone, please don’t save anything sensitive (e.g. passwords, secrets, etc).

Demo video (2 min)

_{Click the image to play video}

👩‍🏫 Tutorials

• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project
• Train model on Titanic dataset
• Set up DBT models and orchestrate DBT runs

🔮 Features

A sample data pipeline defined across 3 files ➝

1. Load data ➝

   @data_loader
   def load_csv_from_file():
       return pd.read_csv('default_repo/titanic.csv')

2. Transform data ➝

   @transformer
   def select_columns_from_df(df, *args):
       return df[['Age', 'Fare', 'Survived']]

3. Export data ➝

   @data_exporter
   def export_titanic_data_to_disk(df) -> None:
       df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI ➝

New? We recommend reading about blocks and learning from a hands-on tutorial.

🏔️ Core design principles

Every user experience and technical design decision adheres to these principles.

🛸 Core abstractions

These are the fundamental concepts that Mage uses to operate.

🙋‍♀️ Contributing and developing

Add features and instantly improve the experience for everyone.

Check out the contributing guide to setup your development environment and start building.

👨‍👩‍👧‍👦 Community

Individually, we’re a mage.

🧙 Mage

Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, we’re Magers!

🧙‍♂️🧙 Magers (/ˈmājər/)

A group of mages who help each other realize their full potential! Let’s hang out and chat together ➝

For real-time news, fun memes, data engineering topics, and more, join us on ➝

	Twitter
	LinkedIn
	GitHub
	Slack

🤔 Frequently Asked Questions (FAQs)

Check out our FAQ page to find answers to some of our most asked questions.

🪪 License

See the LICENSE file for licensing information.