Prefect Recipes 🧑‍🍳 🥐

This repository contains common and extensible Prefect patterns to drive efficient workflows — we like to call these patterns our recipes

Here you'll find starter code and more advanced example use cases.

Contributing = Swag 🧢

We're always looking for new contributions! See our existing Recipe Ideas/Issues for inspiration. Read a detailed guide on how to share your solutions with the Prefect community or run these commands to get started right away.

Issues / Bugs 🐛

To report issues, typos, or link fixes, please open an issue. We appreciate it!

Recipe Ideas 👩‍🍳

What are you interested in seeing examples of? Jot down your big idea here.

Table of Contents 📖

• Getting Started

  • Introductory Videos
  • Subflows
  • Control Flow
  • Optimization
  • Notifications
  • Flow Run Observability
  • Configuration (Using Blocks
  • Parameters
  • Testing
  • Triggering Flow Runs
  • Flow Deployment

• Diving Deeper

  • Deployments & CICD
  • Streaming & Event-Driven Workflows
  • Serverless
  • Data Engineering / DataOps
  • ELT/ETL
  • AWS Infrastructure
  • Azure Infrastructure
  • Helm
  • Kubernetes
  • GitHub Actions
  • Dockerfiles
  • Django
  • Pydantic
  • Hex
  • Product
  • Data Science
  • Modular Data Stack
  • Miscellaneous
  • Legacy (Prefect 1.0)

• Issues & Bugs

• Contributions

• Join the Discussion

• Thanks

Getting Started 🍯

Introductory Videos

• What is Prefect?
• Prefect Demo
• Getting Started with Prefect Cloud

Subflows

• Getting Started With Subflows
• Run Multiple Subflows or Child Flows in Parallel
• Subflow with a Different Task Runner Than Parent Flow
• Create a Subflow and Block Until It's Completed
• Running Subflows On Their Own Infrastructure Using a Separate Deployment

Control Flow

• Conditionally Stop a Task Run
• Ensure Tasks Immediately Fail If Upstream Task Fails
• Define State Dependencies Between Tasks

Optimization

• Cache a Task Result To Prevent Recomputation

Notifications

• Send Notifications with a Slack Webhook
• Sending Slack Notifications in Python with Prefect

Flow Run Observability

• Interact with REST API
• Determine Whether a Flow Run Was Executed Ad Hoc or on a Schedule

Configuration (using Blocks)

• Supercharge your Python Code with Blocks - Blog
• Supercharge your Python Code with Blocks - Video

Parameters

• Use flow parameters

Testing

• Testing Flows, Subflows, and Tasks

Logging

• Explain Your Python Exceptions with OpenAI

Triggering Flow Runs

• Trigger a deployment run when new files land in S3 Event Notifications with EventBridge
• Triggering Flow Runs From a Deployment via API Call or From a Terminal Using Curl
• Event-Driven Data Pipelines with AWS Lambda and GitHub Actions

Flow Deployment

• Deploy Flows to Run as a Local Process, Docker Container or a Kubernetes Job
• Deploy Flows to AWS
• Deploy Flows to GCP
• Deploy Flows to Azure
• Store Flows in GitHub and Execute in a Docker Container
• Python-Based Deployments

Diving Deeper 🍱

Deployments & CICD

• Scheduled Data Pipelines in 5 Minutes with Prefect and GitHub Actions - Blog
  • Scheduling: Get started with Prefect by scheduling your Prefect flows with GitHub Actions - GitHub Repo
• Declarative Dataflow Deployments with Prefect Make CI/CD a Breeze
• Deploy Prefect Pipelines with Python: Perfect!
• Prefect Deployments FAQ (PDF) - DataflowOps for Prefect 2.0
• GCP and Prefect Cloud — from Docker Container to Cloud VM on Google Compute Engine
  • GCP: Repository template to get started with Prefect Cloud & Google Cloud - GitHub Repo
• Deployment patterns & examples: Code examples showing flow deployment to various types of infrastructure - GitHub Repo
• Docker: How to use Prefect Deployments with flow code baked into a Docker image - GitHub Repo
• AWS Fargate: Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate - GitHub Repo
• AWS EKS: Project demonstrating how to automate Prefect 2.0 deployments to AWS EKS - GitHub Repo

Streaming & Event-Driven Workflows

• Event-driven Data Pipelines with AWS Lambda, Prefect and GitHub Actions
  • Streaming: Example project demonstrating deployment patterns for real-time streaming workflows with Prefect 2.0 - GitHub Repo
• Scheduled vs. Event-driven Data Pipelines — Orchestrate Anything with Prefect
• You No Longer Need Two Separate Systems for Batch Processing and Streaming

Serverless

• GCP: Serverless Prefect Flows with Google Cloud Run Jobs
• Azure: Serverless Prefect Flows with Azure Container Instances
• AWS: Prefect + AWS ECS Fargate + GitHub Actions Make Serverless Dataflows As Easy as .py
• AWS Lambda: Deploy a Prefect flow to serverless AWS Lambda function - GitHub Repo

Data Engineering / DataOps

• Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions
• Build a Data Platform with Prefect, dbt, and Snowflake (using blocks)
• Real World Python for Data Engineering - Supercharge Your Data Orchestration with Prefect 2.0
• Create a Maintainable Data Pipeline with Prefect and DVC
• Data engineering & orchestration with Prefect, Docker, Terraform, Google CloudRun, BigQuery and Streamlit

ELT/ETL

• Orchestrating Airbyte with Prefect 2
• Coordinate ELT in 2023 with Airbyte, dbt and Prefect
• Schedule & orchestrate dbt Cloud jobs with Prefect
• Prefect & Fivetran: integrate all the tools & orchestrate them in Python
• Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
• ELT with Snowflake Using Async and Blocks

AWS Infrastructure

• Deploy a Prefect agent to ECS using the AWS CLI
• Deploy a Prefect agent to ECS with Terraform
• Deploy Flows Using ECSTask Infrastructure Blocks
• Deploy a Prefect agent to ECS Fargate using CloudFormation and GitHub Actions

Azure Infrastructure

• Setup Azure with Prefect
• Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
• Setup an Azure VM and Run the Prefect Agent
• Deploy Flows Using Azure Container Instances Infrastructure Blocks

GCP Infrastructure

• Setup GCP Managed Instance Group with Prefect

Helm

• Deploy Prefect Agent using Helm and Terraform

Kubernetes

• How to use Kubernetes with Prefect
• How to use Kubernetes with Prefect: Part 2
• How to use Kubernetes with Prefect: Part 3

Github Actions

• Build flow image and Prefect deployment with storage and infra Blocks on push to branch
• Build and Push flow docker image to Google Artifact Registry
• Build / Apply prefect deployment with blocks on change to python files
• Automate Prefect Deployments to AWS ECS Fargate Using GitHub Actions
• Automate Python-Based Deployments with GitHub Actions

Django

• Using Django with Prefect 2

Pydantic

• Using Pydantic BaseModel with Prefect 2
• Build a Full-Stack ML Application With Pydantic And Prefect

Hex

• Create Observable and Reproducible Notebooks with Hex - Article
• Create Observable and Reproducible Notebooks with Hex - Video

Product

• (Re)Introducing Prefect: The Global Coordination Plane
• The Dataflow Coordination Spectrum
• Why Prefect?
• Workflow Orchestration vs. Data Orchestration — Are Those Different?
• Dataflow Design Patterns

Data Science

• How to Structure an ML Project for Reproducibility and Maintainability
• Orchestrate Your Data Science Project with Prefect 2.0
• Build a Full-Stack ML Application With Pydantic And Prefect

Modular Data Stack

• How to Build a Modular Data Stack — Data Platform with Prefect, dbt and Snowflake
• How to Build Modular Dataflows with Tasks, Flows and Subflows in Prefect

Miscellaneous

• Merge Dependabot Pull Requests with Prefect 2 & a GitHubCredentials block

Discourse

Prefect Agents and Work-Queues FAQ

Prefect 1.0 Legacy

• Register a Prefect 1 Flow
• Run GraphQL Queries
• Airbyte Orchestration
• ETL with AWS S3 and Snowflake
• Use AWS Lambda for Event-Driven Flows
• Read Secrets into Prefect 1 Cloud tenant
• Handle DBT Model Failures
• S3 Prefect 1 Flow Storage on EKS
• Use LocalExecutor to run Dask computations on a Coiled cluster
• Use DaskExecutor to run Prefect 1 tasks in parallel on a Coiled cluster

Contributions

We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:

1. Clone the prefect-recipes repo:

git clone [email protected]:PrefectHQ/prefect-recipes.git

2. Create and checkout a new branch:

git checkout -b feat/new-recipe-name

3. Add your code under the appropriate category. Unsure? Add it under flows-advanced/.
4. Add your recipe to this README.
5. Commit and push the code to your remote branch.
6. Create a PR 🤌

Join our discussions 🗣️

We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation 😄

Thanks 💙

Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!