Frugal Mechanic SBT S3 Resolver

This SBT plugin adds support for using Amazon S3 for resolving and publishing using s3:// urls.

Table of Contents generated with DocToc

• SBT 1.X Support
• SBT 1.3 Support
• SBT 1.1 Support
• SBT 1.0 Support
• Examples
  • Resolving Dependencies via S3
  • Publishing to S3
  • Valid s3:// URL Formats
• Usage
  • Add this to your project/plugins.sbt file:
  • S3 Credentials
    • Bucket Specific Environment Variables
    • Bucket Specific Java System Properties
    • Bucket Specific Property Files
    • Environment Variables
    • Java System Properties
    • Property File
  • Custom S3 Credentials
• IAM Policy Examples
  • Read/Write Policy (for publishing)
  • Read-Only Policy
  • Releases Read-Only, Snapshots Read/Write
• IAM Role Policy Examples
• S3 Server-Side Encryption
• Maintainer
• Copyright
• License

SBT 1.X Support

Recent versions of this plugin should work fine on all SBT 1.X versions. If you run into problems please open up an Issue.

SBT 1.3 Support

SBT 1.3 support is available using version >= 0.19.0:

addSbtPlugin("com.frugalmechanic" % "fm-sbt-s3-resolver" % "0.20.0")

SBT 1.1 Support

SBT 1.1 support is available using version >= 0.14.0:

SBT 1.0 Support

Note: You need to use at least SBT 1.0.4 for this plugin to work with SBT 1.0 due to sbt/librarymanagement#175 which was fixed in this pull request: sbt/librarymanagement#183

Examples

Resolving Dependencies via S3

Maven Style:

resolvers += "FrugalMechanic Snapshots" at "s3://fm-sbt-s3-resolver-example-bucket/snapshots"

Ivy Style:

resolvers += Resolver.url("FrugalMechanic Snapshots", url("s3://fm-sbt-s3-resolver-example-bucket/snapshots"))(Resolver.ivyStylePatterns)

Publishing to S3

Maven Style:

publishMavenStyle := true
publishTo := Some("FrugalMechanic Snapshots" at "s3://fm-sbt-s3-resolver-example-bucket/snapshots")

Ivy Style:

publishMavenStyle := false
publishTo := Some(Resolver.url("FrugalMechanic Snapshots", url("s3://fm-sbt-s3-resolver-example-bucket/snapshots"))(Resolver.ivyStylePatterns))

Valid s3:// URL Formats

The examples above are using the Static Website Using a Custom Domain functionality of S3.

These would also be equivalent (for the fm-sbt-s3-resolver-example-bucket bucket):

s3://s3-us-west-2.amazonaws.com/fm-sbt-s3-resolver-example-bucket/snapshots
s3://fm-sbt-s3-resolver-example-bucket.s3-us-west-2.amazonaws.com/snapshots
s3://fm-sbt-s3-resolver-example-bucket.s3.amazonaws.com/snapshots
s3://s3.amazonaws.com/fm-sbt-s3-resolver-example-bucket/snapshots

All of these forms should work:

s3://[BUCKET]/[OPTIONAL_PATH]
s3://s3.amazonaws.com/[BUCKET]/[OPTIONAL_PATH]
s3://[BUCKET].s3.amazonaws.com/[OPTIONAL_PATH]
s3://s3-[REGION].amazonaws.com/[BUCKET]/[OPTIONAL_PATH]
s3://[BUCKET].s3-[REGION].amazonaws.com/[OPTIONAL_PATH]

Usage

Add this to your project/plugins.sbt file:

addSbtPlugin("com.frugalmechanic" % "fm-sbt-s3-resolver" % "0.20.0")

S3 Credentials

S3 Credentials are checked in the following places and order (e.g. bucket specific settings (~/.sbt/.<bucket_name>_s3credentials) get resolved before global settings (~/.sbt/.s3credentials)):

Note: I think this logic has changed a little bit. See the S3URLHandler.defaultCredentialsProviderChain for the current implementation: https://github.com/frugalmechanic/fm-sbt-s3-resolver/blob/master/src/main/scala/fm/sbt/S3URLHandler.scala#L166

Bucket Specific Environment Variables

AWS_ACCESS_KEY_ID_<BUCKET_NAME> -or- <BUCKET_NAME>_AWS_ACCESS_KEY_ID
AWS_SECRET_KEY_<BUCKET_NAME> -or- <BUCKET_NAME>_AWS_SECRET_KEY

NOTE - The following transforms are applied to the bucket name before looking up the environment variable:

1. The name is upper-cased
2. Dots (.) and dashes (-) are replaced with an underscore (_)
3. Everything other than A-Z, 0-9, and underscores are removed.

Example:

The bucket name "fm-sbt-s3-resolver-example-bucket" becomes "MAVEN_FRUGALMECHANIC_COM":

AWS_ACCESS_KEY_ID_MAVEN_FRUGALMECHANIC_COM="XXXXXX" AWS_SECRET_KEY_MAVEN_FRUGALMECHANIC_COM="XXXXXX" sbt

Bucket Specific Java System Properties

-Daws.accessKeyId.<bucket_name>=XXXXXX -Daws.secretKey.<bucket_name>=XXXXXX
-D<bucket_name>.aws.accessKeyId=XXXXXX -D<bucket_name>.aws.secretKey=XXXXXX

Example:

SBT_OPTS="-Daws.accessKeyId.fm-sbt-s3-resolver-example-bucket=XXXXXX -Daws.secretKey.fm-sbt-s3-resolver-example-bucket=XXXXXX" sbt

Bucket Specific Property Files

~/.sbt/.<bucket_name>_s3credentials
~/.sbt/.s3credentials_<bucket_name>

Environment Variables

AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY)
AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)
AWS_ROLE_ARN

Example:

// Basic Credentials
AWS_ACCESS_KEY_ID="XXXXXX" AWS_SECRET_KEY="XXXXXX" sbt

// IAM Role Credentials
AWS_ACCESS_KEY_ID="XXXXXX" AWS_SECRET_KEY="XXXXXX" AWS_ROLE_ARN="arn:aws:iam::123456789012:role/RoleName" sbt

Java System Properties

// Basic Credentials
-Daws.accessKeyId=XXXXXX -Daws.secretKey=XXXXXX 

// IAM Role
-Daws.accessKeyId=XXXXXX -Daws.secretKey=XXXXXX -Daws.arnRole=arn:aws:iam::123456789012:role/RoleName

Example:

// Basic Credentials
SBT_OPTS="-Daws.accessKeyId=XXXXXX -Daws.secretKey=XXXXXX" sbt

// IAM Role Credentials
SBT_OPTS="-Daws.accessKeyId=XXXXXX -Daws.secretKey=XXXXXX -Daws.arnRole=arn:aws:iam::123456789012:role/RoleName" sbt

Property File

~/.sbt/.s3credentials

The property files should have the following format:

accessKey = XXXXXXXXXX
secretKey = XXXXXXXXXX
// Optional IAM Role
roleArn = arn:aws:iam::123456789012:role/RoleName

Custom S3 Credentials

If the default credential providers do not work for you then you can specify your own AWSCredentialsProvider using the s3CredentialsProvider SettingKey in your build.sbt file:

import com.amazonaws.auth.{AWSCredentialsProviderChain, DefaultAWSCredentialsProviderChain}
import com.amazonaws.auth.profile.ProfileCredentialsProvider

s3CredentialsProvider := { (bucket: String) =>
  new AWSCredentialsProviderChain(
    new ProfileCredentialsProvider("my_profile"),
    DefaultAWSCredentialsProviderChain.getInstance()
  )
}

If you are really lazy and want to provide static credentials using this in your build.sbt file will work:

import com.amazonaws.auth.{AWSStaticCredentialsProvider, BasicAWSCredentials}

s3CredentialsProvider := { (bucket: String) =>
  new AWSStaticCredentialsProvider(new BasicAWSCredentials("your_accessKey", "your_secretKey"))
}

IAM Policy Examples

I recommend that you create IAM Credentials for reading/writing your Maven S3 Bucket. Here are some examples for our fm-sbt-s3-resolver-example-bucket bucket:

Read/Write Policy (for publishing)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:DeleteObject","s3:GetObject","s3:PutObject"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket/*"]
    }
  ]
}

Read-Only Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket/*"]
    }
  ]
}

Releases Read-Only, Snapshots Read/Write

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket/releases/*"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:DeleteObject","s3:GetObject","s3:PutObject"],
      "Resource": ["arn:aws:s3:::fm-sbt-s3-resolver-example-bucket/snapshots/*"]
    }
  ]
}

IAM Role Policy Examples

This is a simple example where a Host AWS Account, can create a Role with permissions for a Client AWS Account to access the Host maven bucket.

1. Host AWS Account, creates an IAM Role named "ClientAccessRole" with policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::[Client AWS Account Id]:user/[Client User Name]"
        },
        "Action": "sts:AssumeRole"
    }
  ]
}

2. Associate the proper IAM Policy Examples to the Host Role
3. Client AWS Account needs to create an AWS IAM User [Client User Name] and associated a policy to gives it permissions to AssumeRole from the Host AWS Account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::[Host AWS Account Id]:role/ClientAccessRole"
    }
  ]
}

S3 Server-Side Encryption

S3 supports server side encryption. The plugin will automatically detect if it needs to ask S3 to use SSE, based on the policies you have on your bucket. If your bucket denies PutObject requests that aren't using SSE, the plugin will include the SSE header in future requests.

To make use of SSE, configure your bucket to enforce the SSE header for PutObject requests.

Example:

{
  "Version": "2012-10-17",
  "Id": "PutObjPolicy",
  "Statement": [
    {
      "Sid": "DenyIncorrectEncryptionHeader",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET_HERE/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    },
    {
      "Sid": "DenyUnEncryptedObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET_HERE/*",
      "Condition": {
        "Null": {
          "s3:x-amz-server-side-encryption": "true"
        }
      }
    }
  ]
}

Maintainer

Tim Underwood (GitHub, LinkedIn, Twitter)

Copyright

Copyright Frugal Mechanic

License

Apache License, Version 2.0