serverless.tf - Doing serverless with Terraform
This repository / serverless.tf framework aims to provide information, solutions, and tools for building, deploying, and managing serverless applications and infrastructure using Terraform.
serverless.tf has started as an organic response to the accidental complexity of many existing tools used by serverless developers.
serverless.tf is not an official AWS or HashiCorp product, and not to be confused with the Serverless Framework.
Terms of Content
Current Status
This project is in the beta. Going through the development process with Betajob's products, and with external customers, serverless.tf's concepts will be verifying and updating.
Check out open-source projects terraform-aws-notify-slack and serverless.tf playground for examples of using serverless.tf approach.
We focus on AWS specific serverless features and services, but most of the information described here can also be applied to Google Cloud Functions, Azure Functions, and any other provider with decent support for the resources via Terraform provider.
Getting Started with "why?"
Most likely, the first question you are wondering - Why do you do this? Or, if you know me and have been following my projects for some time, you may think: Yes, we can do a lot with Terraform, but what is wrong with existing solutions available already?
Before answering what is wrong, let's set the stage by highlighting the good parts of existing toolset available for serverless developers.
There are plenty of tools and frameworks with overlapping functionality, which is excellent - developers now have a choice, if they want.
Assuming that working with serverless would automatically bring developers everything better is one of the misconceptions many developers have after starting playing with it. There is Law of conservation of complexity that can be applied to serverless architectures, too. In simple words, it means that the complexity is not changing, but the complexity can be allocated differently between application developers, tools developers, and cloud providers.
As serverless application developers, we shouldn't be exposed to accidental complexity enforced by tools we have to use. Complexity should be simplified as much as possible (always).
There is flexibility at the cost of accidental complexity developers have to deal with.
Existing tools for serverless developers
When creating a serverless application, at the minimum least, developers have to deal with:
- Serverless application frameworks (Serverless Framework, AWS Chalice, Zappa) to develop, test, and deploy code and serverless infrastructure resources.
- Infrastructure management (Terraform, AWS CloudFormation) manages traditional resources which are not related to serverless, and sometimes to create resources for serverless applications, too.
- Application deployment (Shell scripts, Makefile, AWS CodeDeploy, AWS CLI) to orchestrate non-trivial builds of dependencies and do deployments, also.
As you can see, there is an overlap in the functionality of these tools.
What does a serverless application consist of?
Let's list some parts a typical serverless application has. An unordered list of items include:
- Code of your serverless function
- Code dependencies
- Shell scripts, Makefile, or similar, what installs dependencies and create a build package
- Serverless infrastructure resources (AWS Lambda Function, Lambda Layers, related IAM roles, and policies)
- Infrastructure resources which can be used by serverless functions (e.g., S3 buckets, SQS queues, SNS notifications, etc.)
- Traditional infrastructure resources typically used and managed without related serverless applications (VPC, ALB, Security Groups, etc.)
- Event mapping for your serverless resources
- Monitoring metrics and logs (e.g., CloudWatch logs)
- Deployment, rollback, scaling policies
All of these items are mostly nodes with dependencies developers should describe declaratively (see Infrastructure as Code: Imperative vs. Declarative). We can treat all of these entities using a somewhat similar approach as we do with the Infrastructure as code and DevOps automation at large.
Serverless.tf's approach is to use Terraform, one of the most popular and powerful infrastructures as a code management tool.
Additionally, developers can use Terragrunt as an orchestration tool for Terraform. It is not required but highly recommended since it reduces the complexity of working with Terraform configurations and keeping them DRY as the number of resources increases.
Lack of high-quality reusable components
Some of the existing solutions support plugins that extend the functionality of existing frameworks and simplify infrastructure services.
In reality, often, developers still have to dive into those to learn internals and archive what they need. Adding functionality to those plugins usually requires writing Javascript code.
Using serverless.tf approach developers rely on open-source Terraform AWS modules, which have been developing by Betajob and huge community during several years, you get to build your serverless project on top of the verified, reusable components.
Using existing modules allows serverless developers to focus on their primary tasks instead of learning the internals of Terraform or googling the right piece of AWS CloudFormation snippet. Developers can see and execute working examples in each of the modules, integrate modules into the project, and get to know the modules' source code when necessary.
See the whole list of Supported AWS Serverless Platform Services on serverless.tf. More non-serverless modules listed on the Terraform Registry.
Development Workflows
serverless.tf does not restrict any workflow, but shows how to build, package, test, deploy, monitor, and some other steps can be implemented using Terraform.
serverless.tf's approach advises management of all infrastructure resources equally, independently of their nature or provider - build or deploy of the code of the serverless function, manage VPC resources, manage GitHub users - all of this should be managed using the same commands - terraform apply.
Build
Lambda functions usually have dependencies (libraries and binaries) built locally, in Docker, or by using external tools or services (e.g., AWS CodeBuild).
Terraform AWS Lambda module supports all of these ways of building dependencies (see examples/build-package there).
Building Lambda layer and Lambda functions is an identical process, and it is already supported by the module.
Using commands like sam build provided by AWS SAM can be a feasible option if you are using AWS SAM already and want to perform gradual migration towards serverless.tf's approach and start using Terraform AWS Lambda module where extra features like exclude files by masks, configurable storage, and conditional creation already supported.
Package
Creation of Lambda deployment package (for a function or a layer) supported by Terraform AWS Lambda module and can be customized, or completely disabled in favor of using external tool or script to do that (see examples there).
Test
Running any tests required for serverless application with Terraform efficiently is rather tricky simply because Terraform was not designed to run scripts and get outputs. There are several options developers can use:
- null_resource to run shell scripts without worrying about output.
- Shell provider to manage Shell scripts as fully-fledged resources and have full control of a resource lifecycle handled by Terraform.
- External provider to run any command or script.
By using Terragrunt or other tools which allow developers to orchestrate Terraform code, developers can run extra commands (e.g., shell scripts) that are not part of the infrastructure code itself (see Before and After Hooks there) to perform tests without putting irrelevant code into main infrastructure repository, for example.
Deploy
There are two ways how Lambda function can be updated: by publishing new version (simple deployments) and controlled deployments.
Terraform AWS Lambda module supports both of these deployments of Lambda functions. See examples/deploy for complete end-to-end process (build/update/deploy) of Lambda function using AWS CodeDeploy.
Let's look into each in details.
Simple deployments
Typically, Lambda function updates when source code changes. A new Lambda Function version will also be created, when it is being published.
Published Lambda Function can be invoked using either version number or using $LATEST (unqualified alias). This type of updates is the simplest way of deployment.
Controlled deployments (rolling, canary, rollbacks)
In order to do controlled deployments (rolling, canary, rollbacks) of Lambda Functions we need to use Lambda Function aliases.
In simple terms, Lambda alias is like a pointer to either one version of Lambda Function (when deployment complete), or to two weighted versions of Lambda Function (during rolling or canary deployment).
One Lambda Function can be used in multiple aliases. Using aliases gives large control of which version deployed when having multiple environments.
There is alias module, which simplifies working with alias (create, manage configurations, updates, etc). See examples/alias for various use-cases how aliases can be configured and used.
There is deploy module, which creates required resources to do deployments using AWS CodeDeploy. It also creates the deployment, and wait for completion.
AWS CodeDeploy supports a variety of deployment configuration types and can do rolling, canary, and all-in-one deployments of Lambda function. It is also possible to specify rollback settings and hooks to run before and after the deployment. All of these options already supported by the module mentioned above.
serverless.tf-compatible Terraform modules
All Terraform modules listed on serverless.tf plus all modules in Terraform AWS Modules GitHub Organization available in the Terraform Registry were also designed and implemented with serverless.tf-compatibility in mind (e.g., naming, features, dependencies, quality, etc.).
What is next?
Open-ended feedback is welcome through Github issues. You can also reach out to me via email - [email protected] . I am particularly interested in hearing about these topics:
- What is your biggest challenge working with serverless?
- Why do you use or don't Terraform with serverless?
- What kind of content is missing or wrong in serverless.tf? Remember,
serverless.tfis currently determining and revising its concepts and approaches.
Follow AWS Serverless Heroes to learn about serverless from the experts.
Authors
serverless.tf project managed by Anton Babenko, and is not affiliated with AWS or HashiCorp.
Like this? Please follow me and share it with your network!
Consider support my work on GitHub Sponsors, Buy me a coffee, or PayPal.
License
This code is released under the Apache 2.0 License. Please see LICENSE for more details.
Copyright © 2020 Betajob AS