This repository contains notebooks & instructions for setting up the demo of development workflow & CI/CD (on Azure DevOps) using the Databricks notebooks and Repos feature. Testing of notebooks is done using the Nutter library developed by Microsoft.

Two approaches are demonstrated:

Using notebooks & including the code using %run (doc) - the "main" code is in the notebooks Code1.py and Code2.py, and the testing code is in the unit-tests/test_with_percent_run.py.
Using notebook for test itself, but including main code as Python packages using arbitrary files in Repos functionality (DBR 9.1+). Main code is in the my_package/code1.py and my_package/code2.py files, and test is in unit-tests/test_with_arbitrary_files.py.

This demo shows how you can use Repos to work on your own copy of notebooks, test them after commit in the "staging" environment, and promote to "production" on successful testing of releases branch.

The workflow
Setup on Databricks side
Setup Azure DevOps pipelines * Create variables group to keep common configuration * Create a build pipeline * Create a release pipeline
FAQ & Troubleshooting

There is a possibility of automated setup of this demo using the Terraform. Look into terraform folder for existing implementations.

The workflow

The development workflow is organized as on following image:

Developer works on the code in the separate environment (personal space on Databricks, etc.). When code changes are done, they are committed into some branch
CI/CD implementation (Azure DevOps here) picks up the changes, and tests them in a staging environment (executes the "build pipeline"). This consists of several steps (see azure-pipelines.yml for technical details):
- Update repository checkout in the "Staging" folder
- Execute tests with updated code
- Publish tests results
In current setup, there are different jobs for the "normal" branches, and for "release" branch (releases in this setup), this would allow to run different sets of tests when we're preparing the release
If commit is done to the "release branch, and there are no test failures, then the "release pipeline" is triggered, and it updates the production environment by updating the repository checkout in the "Production" folder.

Setup on Databricks side

Your Databricks workspace needs to have Repos functionality enabled. If it's enabled, you should see the "Repos" icon in the navigation panel:

Fork repository into your environment - Github, or Azure DevOps (follow Databricks documentation on using it)
In the Repos, click "Create Repo" and link it to the Git repository that you've forked - this will be your personal copy of the code that will be used for work:

Create the staging & production checkouts
- In the Repos, in the top-level part, click on the "ᐯ" near the "Repos" header, select "Create" and select "Folder" (see image). Give it the name "Staging":

Click on the "ᐯ" near the "Staging" folder, and click "Create" and select "Repo":

Link it to the Git repository, similarly how you did it for your personal checkout
Create the "Production" folder with repository inside, repeating two previous steps
Create a new cluster that will be used for execution of the tests, you will need to pass the cluster ID to the Nutter to execute the tests

Setup Azure DevOps pipelines

The Azure DevOps setup consists of the several steps, described in the next sections. It's assumed that project in Azure DevOps already exists.

We need to create a personal access token (PAT) that will be used for execution of the tests & updating the repository. This token will be used to authenticate to Databricks workspace, and then it will fetch configured token to authenticate to Git provider. We also need to connect Databricks workspace to the Git provider - usually it's done by using the provider-specific access tokens - see documentation on details of setting the integration with specific Git provider (note, that when repository is on Azure DevOps, you still need to generate Azure DevOps token to make API working!, and also provide the user name in the Git settings).

⚠️ the previous instructions on using Repos + Azure DevOps with service principals weren't correct, so were removed!

Create variables group to keep common configuration

Because we have several pipelines, the it's makes sense to define variable group to store the data that are necessary for execution of tests & deployment of the code. We need following configuration properties for execution of our pipelines:

databricks_host - the URL of your workspace where tests will be executed (host name with https://, without ?o=, and without trailing slash character. For example: https://adb-1568830229861029.9.azuredatabricks.net).
databricks_token - personal access token for executing commands against the workspace. Mark this variable as private!
cluster_id - the ID of the cluster where tests will be executed. DBR 9.1+ should be used to support arbitrary files.
staging_directory - the directory for staging checkout that we created above. For example, /Repos/Staging/databricks-nutter-repos-demo.

The name of the variable group is used in the azure-pipelines.yml. By default its name is "Nutter Testing". Change the azure-pipelines.yml if you use another name for variable group.

Create a build pipeline

Azure DevOps can work with GitHub repositories as well - see documentation for more details on how to link DevOps with GitHub.

In the Azure DevOps, in the Pipelines section, select Pipelines, and click "New pipeline"
Select GitHub and repository
In the "Configure" step select the "Existing Azure Pipelines YAML file" and specify the name of the existing file: azure-pipelines.yml
Save pipeline

Create a release pipeline

In the Azure DevOps, in the Pipelines section, select Releases, and click "New release pipeline"
Select "Empty Job" in the dialog
In the Stage dialog enter some meaningful name for it
In the "Variables" tab, link the variable group that was created previously
Configure job & task:
- Configure agent - in the "Agent Specification" select "ubuntu-18.04"
- Click on "+" and find the "Command line" task
- Enter following code that will connect to the production environment & update the checkout of the repository (via Repos REST API):

python -m pip install --upgrade databricks-cli
databricks repos update --path /Repos/Production/databricks-nutter-repos-demo --branch releases

Below the code, add environment variable DATABRICKS_TOKEN with value $(DATABRICKS_TOKEN) - this will pull it from the variable group into the script's execution context
Save task & job
We also need to configure an artifact:
- Click on "Add artifact", select project, and source (the name of the build pipeline). Also, for "Default version" select "Latest from specific branch with tags" and select the "releases" branch. Click on "Add" to add artifact into pipeline
- Click on the "⚡" icon to configure the continuous deployment (by default, release is triggered manually). Add branch filter and also select the releases branch
Your release pipeline should look as following:

Save the pipeline

After all of this done, the release pipeline will be automatically executed on every successful build in the releases branch.

Github Actions Workflow:

We need to create a personal access token (PAT) that will be used for execution of the tests & updating the repository. This token will be used to authenticate to Databricks workspace, and then it will fetch configured token to authenticate to Git provider. We also need to connect Databricks workspace to the Git provider - usually it's done by using the provider-specific access tokens - see documentation on details of setting the integration with specific Git provider
Create dev, stage and prod Environments in github settings. With environments it is easy to use the same variables names and secret names accross different environments

Create the following properties within each environment:

databricks_host - the URL of your workspace where tests will be executed (host name with https://, without ?o=, and without trailing slash character. For example: https://adb-1568830229861029.9.azuredatabricks.net).
databricks_token - personal access token for executing commands against the workspace. Create this as a secrete variable
cluster_id - the ID of the cluster where tests will be executed. DBR 9.1+ should be used to support arbitrary files.
repo_directory - the directory for checkout for specific environment. For example, /Repos/Staging/databricks-nutter-repos-demo.

The workflow is the same as above and the pipeline looks as following:

FAQ & Troubleshooting

I'm getting "Can’t find repo ID for /Repos/..." when trying to update a repo

This often happens when you're trying to use databricks repos update for workspace that have IP Access Lists enabled. The error message is a misleading, and will be fixed by this pull request.

I'm getting "Error fetching repo ID for ... Unauthorized access to Org..."

This usually happens when you're trying to run CI/CD pipeline against a Databricks workspace with IP Access Lists enabled, and CI/CD server not in the allow list.

How can I perform Repos operations using the service principal?

To perform operations on Repos (update, etc.) we need to associate a Git token with an identity that performs that operation. Please see the following documentation:

alexott/databricks-nutter-repos-demo

alexott

Reviews

Repository Details