terraform-databricks-examples
This repository contains the following:
-
Examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or Github Actions.
-
Multiple examples of Databricks workspace and resources deployment on Azure, AWS and GCP using Databricks Terraform provider.
There are two ways to use this repository:
- Use examples as a reference for your own Terraform code: Please refer to
examplesfolder for individual examples. - Reuse modules from this repository: Please refer to
modulesfolder.
Repository structure
Code in the repository is organized into following folders:
modules- implementation of specific Terraform modules:examples- specific instances that use Terraform modules.cicd-pipelines- Detailed examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or Github Actions.
Repository content
Note
For detailed information about the examples, modules or CICD pipelines, refer to README.md file inside corresponding folder for a detailed guide on how to setup the CICD pipeline.
Examples
The folder examples contains the following Terraform implementation examples :
| Cloud | Example | Description |
|---|---|---|
| Azure | adb-lakehouse | Lakehouse terraform blueprints |
| Azure | adb-with-private-link-standard | Provisioning Databricks on Azure with Private Link - Standard deployment |
| Azure | adb-vnet-injection | A basic example of VNet injected Azure Databricks workspace |
| Azure | adb-exfiltration-protection | A sample implementation of Data Exfiltration Protection |
| Azure | adb-external-hive-metastore | Example template to implement external hive metastore |
| Azure | adb-kafka | ADB - single node kafka template |
| Azure | adb-private-links | Azure Databricks Private Links |
| Azure | adb-splunk | ADB workspace with single VM splunk integration |
| Azure | adb-squid-proxy | ADB clusters with HTTP proxy |
| Azure | adb-teradata | ADB with single VM Teradata integration |
| Azure | adb-uc | ADB Unity Catalog Process |
| Azure | adb-unity-catalog-basic-demo | ADB Unity Catalog end to end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants |
| AWS | aws-workspace-basic | Provisioning AWS Databricks E2 |
| AWS | aws-workspace-with-firewall | Provisioning AWS Databricks E2 with an AWS Firewall |
| AWS | aws-exfiltration-protection | An implementation of Data Exfiltration Protection on AWS |
| AWS | aws-workspace-with-private-link | Coming soon |
| AWS | aws-databricks-flat | AWS Databricks simple example |
| AWS | aws-databricks-modular-privatelink | Deploy multiple AWS Databricks workspaces |
| AWS | aws-workspace-uc-simple | Provisioning AWS Databricks E2 with Unity Catalog in a single apply |
| AWS | aws-databricks-uc | AWS UC |
| AWS | aws-databricks-uc-bootstrap | AWS UC |
| AWS | aws-remote-backend-infra | Simple example on remote backend |
| AWS | aws-workspace-config | Configure workspace objects |
| GCP | gcp-sa-provisionning | Provisionning of the identity with the permissions required to deploy on GCP. |
| GCP | gcp-basic | Workspace Deployment with managed vpc |
| GCP | gcp-byovpc | Workspace Deployment with customer-managed vpc |
Modules
The folder modules contains the following Terraform modules :
| Cloud | Module | Description |
|---|---|---|
| All | databricks-department-clusters | Terraform module that creates Databricks resources for a team |
| Azure | adb-lakehouse | Lakehouse terraform blueprints |
| Azure | adb-lakehouse-uc | Provisioning Unity Catalog resources and accounts principals |
| Azure | adb-with-private-link-standard | Provisioning Databricks on Azure with Private Link - Standard deployment |
| Azure | adb-exfiltration-protection | A sample implementation of Data Exfiltration Protection |
| Azure | adb-with-private-links-exfiltration-protection | Provisioning Databricks on Azure with Private Link and Data Exfiltration Protection |
| AWS | aws-workspace-basic | Provisioning AWS Databricks E2 |
| AWS | aws-databricks-base-infra | Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace |
| AWS | aws-databricks-unity-catalog | Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog |
| AWS | aws-databricks-workspace | Provisioning AWS Databricks E2 Workspace using pre-created AWS Infra |
| AWS | aws-workspace-with-firewall | Provisioning AWS Databricks E2 with an AWS Firewall |
| AWS | aws-exfiltration-protection | An implementation of Data Exfiltration Protection on AWS |
| AWS | aws-workspace-with-private-link | Coming soon |
| GCP | gcp-sa-provisionning | Provisions the identity (SA) with the correct permissions |
| GCP | gcp-workspace-basic | Provisions a workspace with managed VPC |
| GCP | gcp-workspace-byovpc | Workspace with customer-managed VPC. |
CICD pipelines
The folder cicd-pipelines contains the following implementation examples of pipeline:
| Tool | CICD Pipeline |
|---|---|
| Github Actions | manual-approve-with-github-actions |
| Azure DevOps | manual-approve-with-azure-devops |