• This repository has been archived on 10/Sep/2022
  • Stars
    star
    4,316
  • Rank 9,981 (Top 0.2 %)
  • Language
    Ruby
  • License
    MIT License
  • Created over 9 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Export existing AWS resources to Terraform style (tf, tfstate) / No longer actively maintained

Terraforming

Project Status (2021-12-11): No longer actively maintained

Terraforming is no longer actively maintained.

If you want to generate Terraform configurations from existing cloud resources, consider using other tools, such as

  • Terraformer which supports many cloud providers not only AWS but also GCP, Azure, GitHub, Kubernetes, etc., and is able to generate configurations based on the latest provider's resource schema.
  • Terracognita

Thank you for your contributions and supports in the past 6 years.


Build Status Code Climate Coverage Status Gem Version MIT License Docker Repository on Quay.io

Export existing AWS resources to Terraform style (tf, tfstate)

Supported version

  • Ruby 2.3 or higher is required
  • Terraform v0.9.3 or higher is recommended
    • Some resources (e.g. iam_instance_profile) uses newer resource specification

Installation

Add this line to your application's Gemfile:

gem 'terraforming'

And then execute:

$ bundle

Or install it yourself as:

$ gem install terraforming

Prerequisites

You need to set AWS credentials.

export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export AWS_REGION=xx-yyyy-0

You can also specify credential profile in ~/.aws/credentials by --profile option.

$ cat ~/.aws/credentials
[hoge]
aws_access_key_id = Hoge
aws_secret_access_key = FugaFuga

# Pass profile name by --profile option
$ terraforming s3 --profile hoge

You can assume a role by using the --assume option.

$ terraforming s3 --assume arn:aws:iam::123456789123:role/test-role

You can force the AWS SDK to utilize the CA certificate that is bundled with the SDK for systems where the default OpenSSL certificate is not installed (e.g. Windows) by utilizing the --use-bundled-cert option.

PS C:\> terraforming ec2 --use-bundled-cert

Usage

$ terraforming
Commands:
  terraforming alb             # ALB
  terraforming asg             # AutoScaling Group
  terraforming cwa             # CloudWatch Alarm
  terraforming dbpg            # Database Parameter Group
  terraforming dbsg            # Database Security Group
  terraforming dbsn            # Database Subnet Group
  terraforming ddb             # DynamoDB
  terraforming ec2             # EC2
  terraforming ecc             # ElastiCache Cluster
  terraforming ecsn            # ElastiCache Subnet Group
  terraforming efs             # EFS File System
  terraforming eip             # EIP
  terraforming elb             # ELB
  terraforming help [COMMAND]  # Describe available commands or one specific command
  terraforming iamg            # IAM Group
  terraforming iamgm           # IAM Group Membership
  terraforming iamgp           # IAM Group Policy
  terraforming iamip           # IAM Instance Profile
  terraforming iamp            # IAM Policy
  terraforming iampa           # IAM Policy Attachment
  terraforming iamr            # IAM Role
  terraforming iamrp           # IAM Role Policy
  terraforming iamu            # IAM User
  terraforming iamup           # IAM User Policy
  terraforming igw             # Internet Gateway
  terraforming kmsa            # KMS Key Alias
  terraforming kmsk            # KMS Key
  terraforming lc              # Launch Configuration
  terraforming nacl            # Network ACL
  terraforming nat             # NAT Gateway
  terraforming nif             # Network Interface
  terraforming r53r            # Route53 Record
  terraforming r53z            # Route53 Hosted Zone
  terraforming rds             # RDS
  terraforming rs              # Redshift
  terraforming rt              # Route Table
  terraforming rta             # Route Table Association
  terraforming s3              # S3
  terraforming sg              # Security Group
  terraforming sn              # Subnet
  terraforming snst            # SNS Topic
  terraforming snss            # SNS Subscription
  terraforming sqs             # SQS
  terraforming vgw             # VPN Gateway
  terraforming vpc             # VPC

Options:
  [--merge=MERGE]                                # tfstate file to merge
  [--overwrite], [--no-overwrite]                # Overwrite existng tfstate
  [--tfstate], [--no-tfstate]                    # Generate tfstate
  [--profile=PROFILE]                            # AWS credentials profile
  [--region=REGION]                              # AWS region
  [--use-bundled-cert], [--no-use-bundled-cert]  # Use the bundled CA certificate from AWS SDK

Export tf

$ terraforming <resource> [--profile PROFILE]

(e.g. S3 buckets):

$ terraforming s3
resource "aws_s3_bucket" "hoge" {
    bucket = "hoge"
    acl    = "private"
}

resource "aws_s3_bucket" "fuga" {
    bucket = "fuga"
    acl    = "private"
}

Export tfstate

$ terraforming <resource> --tfstate [--merge TFSTATE_PATH] [--overwrite] [--profile PROFILE]

(e.g. S3 buckets):

$ terraforming s3 --tfstate
{
  "version": 1,
  "serial": 1,
  "modules": {
    "path": [
      "root"
    ],
    "outputs": {
    },
    "resources": {
      "aws_s3_bucket.hoge": {
        "type": "aws_s3_bucket",
        "primary": {
          "id": "hoge",
          "attributes": {
            "acl": "private",
            "bucket": "hoge",
            "id": "hoge"
          }
        }
      },
      "aws_s3_bucket.fuga": {
        "type": "aws_s3_bucket",
        "primary": {
          "id": "fuga",
          "attributes": {
            "acl": "private",
            "bucket": "fuga",
            "id": "fuga"
          }
        }
      }
    }
  }
}

If you want to merge exported tfstate to existing terraform.tfstate, specify --tfstate --merge=/path/to/terraform.tfstate option. You can overwrite existing terraform.tfstate by specifying --overwrite option together.

Existing terraform.tfstate:

# /path/to/terraform.tfstate

{
  "version": 1,
  "serial": 88,
  "remote": {
    "type": "s3",
    "config": {
      "bucket": "terraforming-tfstate",
      "key": "tf"
    }
  },
  "modules": {
    "path": [
      "root"
    ],
    "outputs": {
    },
    "resources": {
      "aws_elb.hogehoge": {
        "type": "aws_elb",
        "primary": {
          "id": "hogehoge",
          "attributes": {
            "availability_zones.#": "2",
            "connection_draining": "true",
            "connection_draining_timeout": "300",
            "cross_zone_load_balancing": "true",
            "dns_name": "hoge-12345678.ap-northeast-1.elb.amazonaws.com",
            "health_check.#": "1",
            "id": "hogehoge",
            "idle_timeout": "60",
            "instances.#": "1",
            "listener.#": "1",
            "name": "hoge",
            "security_groups.#": "2",
            "source_security_group": "default",
            "subnets.#": "2"
          }
        }
      }
    }
  }
}

To generate merged tfstate:

$ terraforming s3 --tfstate --merge=/path/to/tfstate
{
  "version": 1,
  "serial": 89,
  "remote": {
    "type": "s3",
    "config": {
      "bucket": "terraforming-tfstate",
      "key": "tf"
    }
  },
  "modules": {
    "path": [
      "root"
    ],
    "outputs": {
    },
    "resources": {
      "aws_elb.hogehoge": {
        "type": "aws_elb",
        "primary": {
          "id": "hogehoge",
          "attributes": {
            "availability_zones.#": "2",
            "connection_draining": "true",
            "connection_draining_timeout": "300",
            "cross_zone_load_balancing": "true",
            "dns_name": "hoge-12345678.ap-northeast-1.elb.amazonaws.com",
            "health_check.#": "1",
            "id": "hogehoge",
            "idle_timeout": "60",
            "instances.#": "1",
            "listener.#": "1",
            "name": "hoge",
            "security_groups.#": "2",
            "source_security_group": "default",
            "subnets.#": "2"
          }
        }
      },
      "aws_s3_bucket.hoge": {
        "type": "aws_s3_bucket",
        "primary": {
          "id": "hoge",
          "attributes": {
            "acl": "private",
            "bucket": "hoge",
            "id": "hoge"
          }
        }
      },
      "aws_s3_bucket.fuga": {
        "type": "aws_s3_bucket",
        "primary": {
          "id": "fuga",
          "attributes": {
            "acl": "private",
            "bucket": "fuga",
            "id": "fuga"
          }
        }
      }
    }
  }
}

After writing exported tf and tfstate to files, execute terraform plan and check the result. There should be no diff.

$ terraform plan
No changes. Infrastructure is up-to-date. This means that Terraform
could not detect any differences between your configuration and
the real physical resources that exist. As a result, Terraform
doesn't need to do anything.

Example: Export all

Example assuming you want to export everything from us-west-2 and you are using ~/.aws/credentials with a default profile

export AWS_REGION=us-west-2
terraforming help | grep terraforming | grep -v help | awk '{print "terraforming", $2, "--profile", "default", ">", $2".tf";}' | bash
# find files that only have 1 empty line (likely nothing in AWS)
find . -type f -name '*.tf' | xargs wc -l | grep ' 1 .'

Caveats

  • terraforming kmsk does not export EXTERNAL origin key, bacause Terraform does not support it.

Run as Docker container Docker Repository on Quay.io

Terraforming Docker Image is available at quay.io/dtan4/terraforming and developed at dtan4/dockerfile-terraforming.

Pull the Docker image:

$ docker pull quay.io/dtan4/terraforming:latest

And then run Terraforming as a Docker container:

$ docker run \
    --rm \
    --name terraforming \
    -e AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX \
    -e AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
    -e AWS_REGION=xx-yyyy-0 \
    quay.io/dtan4/terraforming:latest \
    terraforming s3

Development

After checking out the repo, run script/setup to install dependencies. Then, run script/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release to create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Please read Contribution Guide at first.

  1. Fork it ( https://github.com/dtan4/terraforming/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Similar projects

There are some similar tools to import your existing infrastructure to Terraform configuration.

License

MIT License

More Repositories

1

k8sec

CLI tool to manage Kubernetes Secrets easily
Go
258
star
2

k8stail

tail -f experience for Kubernetes Pods
Go
138
star
3

nginx-basic-auth-proxy

Docker image of Nginx Proxy with Basic Auth
Shell
83
star
4

paus

Docker Compose PaaS
HTML
75
star
5

s3url

Generate S3 object pre-signed URL in one command
Go
39
star
6

k8s-pod-notifier

Notify Pod status to Slack
Go
34
star
7

k8s-job-cleaner

Clean up completed Kubernetes Jobs
Go
33
star
8

dockerfile-terraforming

Docker Image for Terraforming
Ruby
27
star
9

terraforming-gke

Generate Terraform HCL files from existng GKE resources
Ruby
26
star
10

ct2stimer

Convert crontab to systemd timer
Go
22
star
11

sendgrid2datadog

Send SendGrid metrics to DataDog
Go
18
star
12

k8s-unused-secret-detector

Detect unused Kubernetes Secrets
Go
18
star
13

valec

Handle application secrets securely
Go
16
star
14

ec2-event-slack-notifier

AWS Lambda function / Docker image to notify EC2 Scheduled Events to Slack
JavaScript
10
star
15

ghrls

List & Describe GitHub Releases
Go
10
star
16

esnctl

Elasticsearch Node Controller with AWS Auto Scaling Group
Go
10
star
17

aws-sign-proxy

HTTP proxy that signs requests for AWS service endpoints
Go
10
star
18

k8ship

Deploy image to Kubernetes
Go
7
star
19

systemd-timers

Better systemctl list-timers
Go
7
star
20

terraforming-dnsimple

Terraforming extension for DNSimple
Ruby
6
star
21

yaml2dirs

Create directory tree from YAML
Go
5
star
22

md2pukiwiki

Convert Markdown to Pukiwiki-notation
Ruby
5
star
23

ecr-gc

AWS Lambda function to clean up untagged images stored in ECR
JavaScript
5
star
24

terraforming-datadog

Terraforming extension for Datadog Monitors
Ruby
4
star
25

ec2c

Simple AWS EC2 CLI
Go
4
star
26

rdm2bq

Send Amazon RDS Enhanced Monitoring process metrics to Google BigQuery
Ruby
4
star
27

paus-gitreceive

Git server of Paus
Go
4
star
28

lambda-go-ping-function

Go
4
star
29

aws4_signer.cr

Crystal
4
star
30

apig-sample

Go
4
star
31

spotrack

Make spot instance requests and watch them whether accepted or rejected
Ruby
3
star
32

xronor

Timezone-aware Job Scheduler DSL and Converter
Ruby
3
star
33

skytraq-datalogger

C
3
star
34

docker-service-rails

Ruby
3
star
35

lambda-smart-autoscaling

JavaScript
3
star
36

vuejs-sample

Ruby
3
star
37

whosekey

Find the IAM user who has the specified AWS_ACCESS_KEY_ID
Go
2
star
38

xlapse

Serverless periodical image collector
Starlark
2
star
39

nowtv

What TV programs are broadcasted now?
Ruby
2
star
40

picam2s3

Send mjpeg-streamer snapshot to S3
Go
2
star
41

aperdeen

Amazon API Gateway client and local proxy
Go
2
star
42

grpc-pg-todo

Ruby
2
star
43

dockerfile-vault

Docker Image for HashiCorp Vault
Ruby
2
star
44

mado

Realtime Github Flavored Markdown Preview with WebSocket
Ruby
2
star
45

valecjr

Go
1
star
46

photomosaic

Photomosaic Generator
Ruby
1
star
47

paus-frontend

Web frontend of Paus
Go
1
star
48

homebrew-tools

Homebrew recipes of @dtan4 tools
Ruby
1
star
49

grpc-private-bff-example

Go
1
star
50

dockerfile-texlive

1
star
51

dot.zsh

.zsh
Shell
1
star
52

gamesh

Go implementation of ramesh
Go
1
star
53

design-pattern-ruby

GoF design patterns implemented in Ruby
Ruby
1
star
54

k8s-ls-pods

List Kubernetes Pods!
Go
1
star
55

try-prometheus

1
star
56

teppan

CLI tool to generate text from template with .env
Go
1
star
57

ramesh-dashboard

Dashboard for ramesh
Ruby
1
star
58

xygon

Simple Password Manager
Ruby
1
star
59

sintaku

Check EC2 Spot Instance Termination Notice from outside
Go
1
star
60

multihost-coreos-es-cluster

Build Elasticsearch cluster between multiple CoreOS instances
HTML
1
star
61

memot

Synchronize Evernote and Markdown in Dropbox
Ruby
1
star
62

coreos-vulcand-sample

Zero-downtime Blue-Green deployment on CoreOS + Vulcand
Ruby
1
star
63

drone-on-docker

[WIP] Run Drone CI as Docker container
Ruby
1
star
64

aprice

AWS Price List CLI
Go
1
star
65

ramesh

Command Line Tool for ๆฑไบฌใ‚ขใƒกใƒƒใ‚ทใƒฅ (Tokyo-Amesh)
Ruby
1
star
66

2son

API Server to Convert 2ch to JSON
Ruby
1
star
67

dockerfile-emacs

Shell
1
star
68

imageup

Pull Docker images via HTTP webhook
Go
1
star