Orchestration-based Saga on Serverless

Contoso Bank is building a new payment platform leveraging the development of microservices to rapidly offer new features in the market, where legacy and new applications coexist. Operations are now distributed across applications and databases, and Contoso needs a new architecture and implementation design to ensure data consistency on financial transactions.

The traditional ACID approach is not suited anymore for Contoso Bank as the data of operations are now spanned into isolated databases. Instead of ACID transactions, a Saga addresses the challenge by coordinating a workflow through a message-driven sequence of local transactions to ensure data consistency.

About the solution

The solution simulates a money transfer scenario, where an amount is transferred between bank accounts through credit/debit operations and an operation receipt is generated for the requester. It is a Saga pattern implementation reference through an orchestration approach in a serverless architecture on Azure. The solution leverages Azure Functions for the implementation of Saga participants, Azure Durable Functions for the implementation of the Saga orchestrator, Azure Event Hubs as the data streaming platform and Azure Cosmos DB as the database service.

The implementation reference addresses the following challenges and concerns from Contoso Bank:

• Developer experience: A solution that allows developers focus only on the business logic of the Saga participants and simplify the implementation of stateful workflows on the Saga orchestrator. The proposed solution leverages the Azure Functions programming model, reducing the overhead on state management, checkpointing (mechanism that updates the offset of a messaging partition when a consumer service processes a message) and restarts in case of failures.

• Resiliency: A solution capable of handling a set of potential transient failures (e.g. operation retries on databases and message streaming platforms, timeout handling). The proposed solution applies a set of design patterns (e.g. Retry and Circuit Breaker) on operations with Event Hubs and Cosmos DB, as well as timeout handling on the production of commands and events.

• Idempotency: A solution where each Saga participant can execute multiple times and provide the same result to reduce side effects, as well as to ensure data consistency. The proposed solution relies on validations on Cosmos DB for idempotency, making sure there is no duplication on the transaction state and no duplication on the creation of events.

• Observability: A solution that is capable of monitoring and tracking the Saga workflow states per transaction. The proposed solution leverages Cosmos DB collections that allows the track of the workflow by applying a single query.

Potential use cases

These other uses cases have similar design patterns:

• Settlement transactions
• Order services (e.g. e-commerce, food delivery, flight/hotel/taxi booking)

Architecture

Check the following sections about the core components of the solution, workflows and design decisions:

• Architecture: Core Components
• Architecture: Workflows
• Architecture: Additional patterns applied
• Architecture: Alternatives & Considerations

Maturity Level

These are the top skills required for Solution Architects and Developer Leads that will reuse the proposed solution or parts of the implementation as reference:

• Architectures: Microservices, Event-driven and Serverless
• Design Patterns: Saga, Database-per-microservice, Async HTTP API, Retry, Circuit Breaker
• Azure services: Azure Functions (including Event Hubs + Cosmos DB bindings and Durable Functions extension), Event Hubs, Cosmos DB, Azure Storage
• Programming Language: C#
• Additional concepts: Infrastructure as Code, RESTful APIs, idempotency

Repository Structure

Outline the file contents of the repository. It helps users navigate the codebase, build configuration and any related assets.

The src folder contains a set of .NET Core solutions:

Prerequisites

• An Azure account with an active subscription. Create an account for free.
• Azure CLI
• Terraform 0.2.6 or later
• Azure Functions Core Tools v3.0 or later
• .NET Core 3.1 or later

If Visual Studio Code is your platform preference for development:

• C# extension
• Azure Functions extension

If Visual Studio is your platform preference for development:

• Visual Studio 2019 with Azure development workload

Getting Started

1. Create the infrastructure

Follow the instructions on Create Infrastructure as Code to create a new GitHub Actions pipeline that creates the required infrastructure on Azure through Terraform.

2. Creating the application settings

The solution requires a set of environment variables to be configured on Saga.Functions solution for running the project locally. Follow the Configuring the Azure Functions Application Settings instructions to create a local.settings.json. Make sure the settings are configured before proceeding to the next step.

4. Running unit tests

Run the following command on test projects:

dotnet test

5. Running the solution

For running the solution through command line, make sure you're on the src/Saga.Functions directory then run the following command:

func host start

The solution provides a Swagger UI on http://localhost:<port>/api/swagger/ui.

You can start a new HTTP request on /api/saga/start route by providing the following json as part of the body content:

{
    "accountFromId": "<random ID>",
    "accountToId": "<random ID>",
    "amount": 100.00
}

After executing the request, it's expected to receive the transaction ID as part of the HTTP response.

You can check the documents created on Cosmos DB. It is expected to have the following document structure on the orchestrator collection:

{
  "accountFromId": "<random bank account ID>",
  "accountToId": "<random bank account ID>",
  "amount": 100,
  "id": "<random transaction ID>",
  "type": "Default",
  "state": "Success",
  ...
}

For observability, check the saga collection that contains all events processed by the orchestrator. It is expected to have the following structure:

{
  "transactionId": "<random transaction ID>",
  "source": "<Validator, Transfer or Receipt>",
  "messageType": "<event name>",
  "creationDate": "<event creation date>",
  ...
}

Next Steps

• Understanding the Saga Orchestrator
• Adding new Saga participants in a workflow
• Commands and Events Contracts

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.