• Stars
    star
    958
  • Rank 47,718 (Top 1.0 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Accelerate SHA256 computations in pure Go using AVX512, SHA Extensions for x86 and ARM64 for ARM. On AVX512 it provides an up to 8x improvement (over 3 GB/s per core). SHA Extensions give a performance boost of close to 4x over native.

sha256-simd

Accelerate SHA256 computations in pure Go using AVX512, SHA Extensions for x86 and ARM64 for ARM. On AVX512 it provides an up to 8x improvement (over 3 GB/s per core). SHA Extensions give a performance boost of close to 4x over native.

Introduction

This package is designed as a replacement for crypto/sha256. For ARM CPUs with the Cryptography Extensions, advantage is taken of the SHA2 instructions resulting in a massive performance improvement.

This package uses Golang assembly. The AVX512 version is based on the Intel's "multi-buffer crypto library for IPSec" whereas the other Intel implementations are described in "Fast SHA-256 Implementations on Intel Architecture Processors" by J. Guilford et al.

Support for Intel SHA Extensions

Support for the Intel SHA Extensions has been added by Kristofer Peterson (@svenski123), originally developed for spacemeshos here. On CPUs that support it (known thus far Intel Celeron J3455 and AMD Ryzen) it gives a significant boost in performance (with thanks to @AudriusButkevicius for reporting the results; full results here).

$ benchcmp avx2.txt sha-ext.txt
benchmark           AVX2 MB/s    SHA Ext MB/s  speedup
BenchmarkHash5M     514.40       1975.17       3.84x

Thanks to Kristofer Peterson, we also added additional performance changes such as optimized padding, endian conversions which sped up all implementations i.e. Intel SHA alone while doubled performance for small sizes, the other changes increased everything roughly 50%.

Support for AVX512

We have added support for AVX512 which results in an up to 8x performance improvement over AVX2 (3.0 GHz Xeon Platinum 8124M CPU):

$ benchcmp avx2.txt avx512.txt
benchmark           AVX2 MB/s    AVX512 MB/s  speedup
BenchmarkHash5M     448.62       3498.20      7.80x

The original code was developed by Intel as part of the multi-buffer crypto library for IPSec or more specifically this AVX512 implementation. The key idea behind it is to process a total of 16 checksums in parallel by “transposing” 16 (independent) messages of 64 bytes between a total of 16 ZMM registers (each 64 bytes wide).

Transposing the input messages means that in order to take full advantage of the speedup you need to have a (server) workload where multiple threads are doing SHA256 calculations in parallel. Unfortunately for this algorithm it is not possible for two message blocks processed in parallel to be dependent on one another — because then the (interim) result of the first part of the message has to be an input into the processing of the second part of the message.

Whereas the original Intel C implementation requires some sort of explicit scheduling of messages to be processed in parallel, for Golang it makes sense to take advantage of channels in order to group messages together and use channels as well for sending back the results (thereby effectively decoupling the calculations). We have implemented a fairly simple scheduling mechanism that seems to work well in practice.

Due to this different way of scheduling, we decided to use an explicit method to instantiate the AVX512 version. Essentially one or more AVX512 processing servers (Avx512Server) have to be created whereby each server can hash over 3 GB/s on a single core. An hash.Hash object (Avx512Digest) is then instantiated using one of these servers and used in the regular fashion:

import "github.com/minio/sha256-simd"

func main() {
	server := sha256.NewAvx512Server()
	h512 := sha256.NewAvx512(server)
	h512.Write(fileBlock)
	digest := h512.Sum([]byte{})
}

Note that, because of the scheduling overhead, for small messages (< 1 MB) you will be better off using the regular SHA256 hashing (but those are typically not performance critical anyway). Some other tips to get the best performance:

  • Have many go routines doing SHA256 calculations in parallel.
  • Try to Write() messages in multiples of 64 bytes.
  • Try to keep the overall length of messages to a roughly similar size ie. 5 MB (this way all 16 ‘lanes’ in the AVX512 computations are contributing as much as possible).

More detailed information can be found in this blog post including scaling across cores.

Drop-In Replacement

The following code snippet shows how you can use github.com/minio/sha256-simd. This will automatically select the fastest method for the architecture on which it will be executed.

import "github.com/minio/sha256-simd"

func main() {
        ...
	shaWriter := sha256.New()
	io.Copy(shaWriter, file)
        ...
}

Performance

Below is the speed in MB/s for a single core (ranked fast to slow) for blocks larger than 1 MB.

Processor SIMD Speed (MB/s)
3.0 GHz Intel Xeon Platinum 8124M AVX512 3498
3.7 GHz AMD Ryzen 7 2700X SHA Ext 1979
1.2 GHz ARM Cortex-A53 ARM64 638

asm2plan9s

In order to be able to work more easily with AVX512/AVX2 instructions, a separate tool was developed to convert SIMD instructions into the corresponding BYTE sequence as accepted by Go assembly. See asm2plan9s for more information.

Why and benefits

One of the most performance sensitive parts of the Minio object storage server is related to SHA256 hash sums calculations. For instance during multi part uploads each part that is uploaded needs to be verified for data integrity by the server.

Other applications that can benefit from enhanced SHA256 performance are deduplication in storage systems, intrusion detection, version control systems, integrity checking, etc.

ARM SHA Extensions

The 64-bit ARMv8 core has introduced new instructions for SHA1 and SHA2 acceleration as part of the Cryptography Extensions. Below you can see a small excerpt highlighting one of the rounds as is done for the SHA256 calculation process (for full code see sha256block_arm64.s).

sha256h    q2, q3, v9.4s
sha256h2   q3, q4, v9.4s
sha256su0  v5.4s, v6.4s
rev32      v8.16b, v8.16b
add        v9.4s, v7.4s, v18.4s
mov        v4.16b, v2.16b
sha256h    q2, q3, v10.4s
sha256h2   q3, q4, v10.4s
sha256su0  v6.4s, v7.4s
sha256su1  v5.4s, v7.4s, v8.4s

Detailed benchmarks

Benchmarks generated on a 1.2 Ghz Quad-Core ARM Cortex A53 equipped Pine64.

minio@minio-arm:$ benchcmp golang.txt arm64.txt
benchmark                 golang         arm64        speedup
BenchmarkHash8Bytes-4     0.68 MB/s      5.70 MB/s      8.38x
BenchmarkHash1K-4         5.65 MB/s    326.30 MB/s     57.75x
BenchmarkHash8K-4         6.00 MB/s    570.63 MB/s     95.11x
BenchmarkHash1M-4         6.05 MB/s    638.23 MB/s    105.49x

License

Released under the Apache License v2.0. You can find the complete text in the file LICENSE.

Contributing

Contributions are welcome, please send PRs for any enhancements.

More Repositories

1

minio

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.
Go
47,069
star
2

mc

Simple | Fast tool to manage MinIO clusters ☁️
Go
2,803
star
3

minio-go

MinIO Go client SDK for S3 compatible object storage
Go
2,406
star
4

simdjson-go

Golang port of simdjson: parsing gigabytes of JSON per second
Go
1,789
star
5

c2goasm

C to Go Assembly
Go
1,307
star
6

operator

Simple Kubernetes Operator for MinIO clusters 💻
Go
1,209
star
7

minio-java

MinIO Client SDK for Java
Java
1,074
star
8

minio-js

MinIO Client SDK for Javascript
JavaScript
933
star
9

highwayhash

Native Go version of HighwayHash with optimized assembly implementations on Intel and ARM. Able to process over 10 GB/sec on a single core on Intel CPUs - https://en.wikipedia.org/wiki/HighwayHash
Go
870
star
10

console

Simple UI for MinIO Object Storage 🧮
JavaScript
832
star
11

minio-py

MinIO Client SDK for Python
Python
811
star
12

awesome-minio

A curated list of Awesome MinIO community projects.
665
star
13

selfupdate

Build self-updating Go programs
Go
663
star
14

minio-dotnet

MinIO Client SDK for .NET
C#
556
star
15

directpv

Kubernetes CSI driver for Direct Attached Storage 💽
Go
547
star
16

docs

MinIO Object Storage Documentation
SCSS
547
star
17

warp

S3 benchmarking tool
Go
539
star
18

sidekick

High Performance HTTP Sidecar Load Balancer
Go
538
star
19

kes

Key Managament Server for Object Storage and more
Go
455
star
20

minfs

A network filesystem client to connect to MinIO and Amazon S3 compatible cloud storage servers
Go
455
star
21

dsync

A distributed sync package.
Go
400
star
22

doctor

Doctor is a documentation server for your docs in github
Ruby
389
star
23

minio-service

Collection of MinIO server scripts for upstart, systemd, sysvinit, launchd.
Shell
366
star
24

minsql

High-performance log search engine.
Rust
358
star
25

sio

Go implementation of the Data At Rest Encryption (DARE) format.
Go
354
star
26

blake2b-simd

Fast hashing using pure Go implementation of BLAKE2b with SIMD instructions
Go
251
star
27

minio-rs

MinIO Rust SDK for Amazon S3 Compatible Cloud Storage
Rust
208
star
28

concert

Concert is a console based certificate generation tool for https://letsencrypt.org.
Go
194
star
29

md5-simd

Accelerate aggregated MD5 hashing performance up to 8x for AVX512 and 4x for AVX2. Useful for server applications that need to compute many MD5 sums in parallel.
Go
172
star
30

asm2plan9s

Tool to generate BYTE sequences for Go assembly as generated by YASM
Go
169
star
31

certgen

A dead simple tool to generate self signed certificates for MinIO TLS deployments
Go
140
star
32

minio-cpp

MinIO C++ Client SDK for Amazon S3 Compatible Cloud Storage
C++
119
star
33

thumbnailer

A thumbnail generator example using Minio's listenBucketNotification API
JavaScript
104
star
34

charts

MinIO Helm Charts
Mustache
98
star
35

spark-select

A library for Spark DataFrame using MinIO Select API
Scala
96
star
36

mint

Collection of tests to detect overall correctness of MinIO server.
Go
81
star
37

madmin-go

The MinIO Admin Go Client SDK provides APIs to manage MinIO services
Go
81
star
38

openlake

Build Data Lake using Open Source tools
Jupyter Notebook
70
star
39

minio-java-rest-example

REST example using minio-java library.
Java
65
star
40

minio-go-media-player

A HTML5 media player using minio-go library.
HTML
57
star
41

dperf

Drive performance measurement tool
Go
53
star
42

minio-js-store-app

Store Application using minio-js library to manage product assets
HTML
50
star
43

minio-hs

MinIO Client SDK for Haskell
Haskell
47
star
44

hperf

Distributed HTTP Speed Test.
Go
45
star
45

msf

MFS (Minio Federation Service) is a namespace, identity and access management server for Minio Servers
Go
43
star
46

zipindex

Package for indexing zip files and storing a compressed index
Go
43
star
47

nifi-minio

A custom ContentRepository implementation for NiFi to persist data to MinIO Object Storage
Java
34
star
48

simdcsv

Go
33
star
49

benchmarks

Collection of benchmarks captured for MinIO server.
30
star
50

lxmin

Backup and Restore LXC instances from MinIO
Go
28
star
51

m3

MinIO Kubernetes Cloud
Go
27
star
52

android-photo-app

Android Photo App example using minio-java library.
Java
26
star
53

minio-ruby

MinIO Client SDK for Ruby
Ruby
26
star
54

radio

Redundant Array of Distributed Independent Objectstores in short RADIO performs synchronous mirroring, erasure coding across multiple object stores
Go
24
star
55

pkg

Repository to hold all the common packages imported by MinIO projects
Go
24
star
56

blog-assets

Collection of assets used for various articles at https://blogs.min.io
Jupyter Notebook
24
star
57

parquet-go

Go library to work with Parquet Files
Go
23
star
58

presto-minio

How to use Presto (with Hive metastore) and MinIO?
23
star
59

bottlenet

Find bottlenecks in distributed network
Go
21
star
60

lsync

Local syncing package with support for timeouts. This package offers both a sync.Mutex and sync.RWMutex compatible interface.
Go
17
star
61

simple-ci

Stateless. Infinite scalability. Easy Setup. Microservice. Minimalist CI
JavaScript
17
star
62

ming

Object Storage Gateway for Hybrid Cloud
Go
17
star
63

gluegun

Glues Github markdown docs to present a beautiful documentation site.
CSS
16
star
64

swift-photo-app

Swift photo app
Swift
15
star
65

homebrew-stable

Homebrew tap for MinIO
Ruby
15
star
66

mnm

Minimal Minio API aggregates many minio instances to look like one
Go
14
star
67

rsync-go

This is a pure go implementation of the rsync algorithm with highwayhash signature
Go
13
star
68

minio-iam-testing

Shell
13
star
69

mds

MinIO Design System is a common library of all the UI design elements.
TypeScript
12
star
70

perftest

Collection of scripts used in Minio performance testing.
Go
12
star
71

ror-resumeuploader-app

Ruby on rails app using aws-sdk-ruby
JavaScript
11
star
72

select-simd

Go
8
star
73

spark-streaming-checkpoint

Spark Streaming Checkpoint File Manager for MinIO
Scala
8
star
74

kms-go

MinIO Key Managment SDK
Go
8
star
75

chaos

A framework for testing Minio's fault tolerance capability.
Go
8
star
76

hdfs-to-minio

A simple containerized hadoop CLI to migrate content between various HCFS implementations
Dockerfile
7
star
77

simdjson-fuzz

Fuzzers and corpus for https://github.com/minio/simdjson-go
Go
7
star
78

minio-lambda-notification-example

Example App that uses MinIO Lambda Notification with Postgres
JavaScript
7
star
79

buzz

A prototype for github issue workflow management
Less
7
star
80

dmt

Direct MinIO Tunnel
Go
6
star
81

go-cv

Golang wrapper for https://github.com/ermig1979/Simd
Go
6
star
82

spark-data-generator

Generates dummy parquet, csv, json files for testing and validating MinIO compatibility
Scala
6
star
83

xxml

Package xml implements a simple XML 1.0 parser that understands XML name spaces, extended support for control characters.
Go
5
star
84

minio-jenkins

This is a simple Jenkins plugin that lets you upload Jenkins artifacts to a Minio Server
Java
5
star
85

disco

Disco discovery service for MinIO.
Go
5
star
86

docs-k8s

MinIO Docs for Kubernetes
Python
4
star
87

attic

Collection of deprecated packages 😟
C++
4
star
88

pkger

Debian, RPMs and APKs for MinIO
Go
4
star
89

confess

Object store consistency checker
Go
4
star
90

kitchensink

Go
3
star
91

colorjson

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions
Go
3
star
92

webhook

HTTP events to file logger
Go
3
star
93

marketplace

Makefile
3
star
94

minio-pcf-adapter

MinIO Service Adapter for Pivotal
Go
2
star
95

training

Materials for supporting MinIO-led training and curriculum.
Python
2
star
96

xfile

Determines information about the object.
Go
2
star
97

wiki

MinIO's Wiki
2
star
98

docs-vsphere

MinIO Docs for VMware Cloud Foundation
Python
2
star
99

hcp-to-minio

About A simple CLI to migrate content from HCP to MinIO
Go
2
star
100

csvparser

Package csv reads and writes comma-separated values (CSV) files.
Go
2
star