• This repository has been archived on 03/Feb/2022
  • Stars
    star
    202
  • Rank 193,691 (Top 4 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated about 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A format-preserving encryption implementation in Go

Due to changes in the priorities, this project is currently not being supported. The project is archived as of 11/5/21 and will be available in a read-only state. Please note, since archival, the project is not maintained or reviewed.

Godoc Build Status Go Report Card Sourcegraph License

fpe - Format Preserving Encryption Implementation in Go

An implementation of the NIST approved Format Preserving Encryption (FPE) FF1 and FF3 algorithms in Go.

NIST Recommendation SP 800-38G

This follows the FF1 and FF3 schemes for Format Preserving Encryption outlined in the NIST Recommendation, released in March 2016. For FF1, it builds on and formalizes (differing from but remaining mathematically equivalent to) the FFX-A10 scheme by Bellare, Rogaway and Spies as defined here and here. For FF3, it formalizes the BPS scheme.

A note about FF3: There was some recent cryptanalysis about the FF3 algorithm that is important to review. NIST has concluded that FF3 is no longer suitable as a general-purpose FPE method.

A note about FF2: FF2 was originally NOT recommended by NIST, but it is under review again as DFF. You can read about it here.

Testing

There are some official test vectors for both FF1 and FF3 provided by NIST, which are used for testing in this package.

To run unit tests on this implementation with all test vectors from the NIST link above, run the built-in tests:

  1. go test -v github.com/capitalone/fpe/ff1
  2. go test -v github.com/capitalone/fpe/ff3

To run only benchmarks:

  1. go test -v -bench=. -run=NONE github.com/capitalone/fpe/ff1
  2. go test -v -bench=. -run=NONE github.com/capitalone/fpe/ff3

Example Usage

The example code below can help you get started. Copy it into a file called main.go, and run it with go run main.go.

package main

import (
	"encoding/hex"
	"fmt"
	"github.com/capitalone/fpe/ff1"
)

// panic(err) is just used for example purposes.
func main() {
	// Key and tweak should be byte arrays. Put your key and tweak here.
	// To make it easier for demo purposes, decode from a hex string here.
	key, err := hex.DecodeString("EF4359D8D580AA4F7F036D6F04FC6A94")
	if err != nil {
		panic(err)
	}
	tweak, err := hex.DecodeString("D8E7920AFA330A73")
	if err != nil {
		panic(err)
	}

	// Create a new FF1 cipher "object"
	// 10 is the radix/base, and 8 is the tweak length.
	FF1, err := ff1.NewCipher(10, 8, key, tweak)
	if err != nil {
		panic(err)
	}

	original := "123456789"

	// Call the encryption function on an example SSN
	ciphertext, err := FF1.Encrypt(original)
	if err != nil {
		panic(err)
	}

	plaintext, err := FF1.Decrypt(ciphertext)
	if err != nil {
		panic(err)
	}

	fmt.Println("Original:", original)
	fmt.Println("Ciphertext:", ciphertext)
	fmt.Println("Plaintext:", plaintext)
}

Usage notes

There is a FIPS Document that containsRequirements for Vendor Affirmation of SP 800-38G on page 155.

There are some patent related details for FF1 and FF3 as Voltage Security (which was acquired by what is now HP Enterprise) originally developed FFX, which became FF1. They provided NIST with a Letter of Assurance on the matter.

It can be used as part of sensitive data tokenization, especially in regards to PCI and cryptographically reversible tokens. This implementation does not provide any gaurantees regarding PCI DSS or other validation.

It's important to note that, as with any cryptographic package, managing and protecting the key appropriately to your situation is crucial. This package does not provide any guarantees regarding the key in memory.

Implementation Notes

Overall, this was originally written with the following goals:

  • Be as idiomatic as possible from a Go language, package, and interface perspective
  • Follow the algorithm as outlined in the NIST recommendation as closely as possible
  • Attempt to be a reference implementation since one does not exist yet

As such, it was not necessarily written from a performance perspective. While some performance optimizations have been added in v1.1, the nature of format preserving encryption is to operate on strings, which are inherently slow as compared to traditional encryption algorithms which operate on bytes.

Further, while the test vectors all pass, the lack of a reference implementation makes it difficult to test ALL input combinations for correctness.

As of Go 1.9, the standard library's math/big package did not support radices/bases higher than 36. As such, the initial release only supports base 36 strings, which can contain numeric digits 0-9 or lowercase alphabetic characters a-z.

Base 62 support should be available come Go 1.10. See this commit and this tracking issue.

The only cryptographic primitive used for FF1 and FF3 is AES. This package uses Go's standard library's crypto/aes package for this. Note that while it technically uses AES-CBC mode, in practice it almost always is meant to act on a single-block with an IV of 0, which is effectively ECB mode. AES is also the only block cipher function that works at the moment, and the only allowed block cipher to be used for FF1/FF3, as per the spec.

In the spec, it says that the radix and minimum length (minLen) of the message should be such that radix^minLen >= 100. In Appendix A, it mentions this is to prevent a generic MITM against the Feistel structure, but for better security, radix^minLen >= 1,000,000. In ff1.go and ff3.go there is a const called FEISTEL_MIN that can be changed to a sufficient value (like 1,000,000), but by default, it only follows the bare spec.

Regarding how the "tweak" is used as input: I interpreted the spec as setting the tweak in the initial NewCipher call, instead of in each Encrypt and Decrypt call. In one sense, it is similar to passing an IV or nonce once when creating an encryptor object. It's likely that this can be modified to do it in each Encrypt/Decrypt call, if that is more applicable to what you are building.

Existing Implementations

Based on searching GitHub and the Internet, there are no known reference implementations for either algorithm.

An existing Go implementation based on the earlier FFX spec was already created, but the implementation differs slightly from the final NIST recommendation. Further, that implementation doesn't work for higher radices or long strings as it doesn't use math/big (big integers).

There is a Java implementation that was also used for testing and comparison while developing this, as it was the only other fully working implementation available.

Contributors

We welcome your interest in Capital One’s Open Source Projects (the “Project”). Any Contributor to the project must accept and sign a CLA indicating agreement to the license terms. Except for the license granted in this CLA to Capital One and to recipients of software distributed by Capital One, you reserve all right, title, and interest in and to your contributions; this CLA does not impact your rights to use your own contributions for any other purpose.

Link to Individual CLA

Link to Corporate CLA

This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.

More Repositories

1

DataProfiler

What's in your data? Extract schema, statistics and entities from datasets
Python
1,426
star
2

react-native-pathjs-charts

Android and iOS charts based on react-native-svg and paths-js
JavaScript
878
star
3

datacompy

Pandas, Polars, and Spark DataFrame comparison for humans and more!
Python
480
star
4

cqrs-manager-for-distributed-reactive-services

Experimental CQRS and Event Sourcing service
Java
304
star
5

SWHttpTrafficRecorder

A simple library empowering you to record/capture HTTP(s) traffic of an iOS app for mocking/stubbing later.
Objective-C
205
star
6

rubicon-ml

Capture all information throughout your model's development in a reproducible way and tie results directly to the model code!
Jupyter Notebook
127
star
7

giraffez

User-friendly Teradata client for Python
Python
108
star
8

locopy

locopy: Loading/Unloading to Redshift and Snowflake using Python.
Python
104
star
9

checks-out

Checks-Out pull request approval system
Go
76
star
10

dataCompareR

dataCompareR is an R package that allows users to compare two datasets and view a report on the similarities and differences.
R
75
star
11

stack-deployment-tool

Go
66
star
12

bash_shell_mock

A shell script mocking utility/framework for the BASH shell
Shell
66
star
13

architecture-viewer

Visualize your PlantUML sequence diagrams as interactive architecture diagrams!
JavaScript
60
star
14

go-future-context

A simple Future (Promise) library for Go.
Go
54
star
15

AI_Dictionary_English_Spanish

TeX
49
star
16

acronym-decoder

Acronym Decoder
TypeScript
43
star
17

synthetic-data

Generating complex, nonlinear datasets appropriate for use with deep learning/black box models which 'need' nonlinearity

Python
43
star
18

Particle-Cloud-Framework

Python
36
star
19

slackbot-destroyer

📣 ❌ Slack integration that can destroy all incoming messages from Slackbot.
Python
34
star
20

global-attribution-mapping

GAM (Global Attribution Mapping) explains the landscape of neural network predictions across subpopulations
Python
33
star
21

federated-model-aggregation

The Federated Model Aggregation (FMA) Service is a collection of installable python components that make up the generic workflow/infrastructure needed for federated learning.
Python
30
star
22

oas-nodegen

A library for generating completely customizable code from the Open API Specification (FKA Swagger) RESTful API documentation using the scripting power of Node.js.
JavaScript
28
star
23

easy-screenshots

Android Instrumentation Test Screenshots made Easy.
Java
21
star
24

edgetest

edgetest is a tox-inspired python library that will loop through your project's dependencies, and check if your project is compatible with the latest version of each dependency
Python
19
star
25

ablation

Evaluating XAI methods through ablation studies.
Python
15
star
26

serverless-shell

⚡️🐚 Serverless Shell with environment variables plugin
JavaScript
14
star
27

OAuthClient

Awesome OAuth Client for Java.
Java
13
star
28

otvPlots

ovtPlots: An R Package for Variable Level Monitoring
R
13
star
29

json-syntax

Generates functions to convert Python classes to and from JSON friendly objects.
Python
12
star
30

screen-object

screen-object (ruby gem for mobile app automation)
Ruby
12
star
31

jwt-security

JavaScript
11
star
32

BankAccountStarter-API-reference-app

CSS
10
star
33

CreditOffers-API-reference-app

JavaScript
10
star
34

Rewards-API-reference-app

JavaScript
10
star
35

local-crontab

🗺️⏰ Convert local crontabs to UTC crontabs
JavaScript
8
star
36

modtracker

JSON unmarshaling in Go that includes detection of modified fields
Go
7
star
37

grpc-cucumber-js

JavaScript
7
star
38

edgetest-hub

hub plugin for edgetest
Python
2
star
39

oas-nodegen-example

Example project that shows how to customize generated code to fit a specific design pattern using oas-nodegen
Java
2
star
40

edgetest-conda

Conda plugin for edgetest
Python
1
star
41

edgetest-pip-tools

pip-tools plugin for edgetest
Python
1
star