• Stars
    star
    732
  • Rank 61,915 (Top 2 %)
  • Language
    C
  • License
    Other
  • Created over 8 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Zstd wrapper for Go

Zstd Go Wrapper

CircleCI GoDoc

C Zstd Homepage

The current headers and C files are from v1.5.0 (Commit 10f0e699).

Usage

There are two main APIs:

  • simple Compress/Decompress
  • streaming API (io.Reader/io.Writer)

The compress/decompress APIs mirror that of lz4, while the streaming API was designed to be a drop-in replacement for zlib.

Building against an external libzstd

By default, zstd source code is vendored in this repository and the binding will be built with the vendored source code bundled.

If you want to build this binding against an external static or shared libzstd library, you can use the external_libzstd build tag. This will look for the libzstd pkg-config file and extract build and linking parameters from that pkg-config file.

Note that it requires at least libzstd 1.4.0.

go build -tags external_libzstd

Simple Compress/Decompress

// Compress compresses the byte array given in src and writes it to dst.
// If you already have a buffer allocated, you can pass it to prevent allocation
// If not, you can pass nil as dst.
// If the buffer is too small, it will be reallocated, resized, and returned bu the function
// If dst is nil, this will allocate the worst case size (CompressBound(src))
Compress(dst, src []byte) ([]byte, error)
// CompressLevel is the same as Compress but you can pass another compression level
CompressLevel(dst, src []byte, level int) ([]byte, error)
// Decompress will decompress your payload into dst.
// If you already have a buffer allocated, you can pass it to prevent allocation
// If not, you can pass nil as dst (allocates a 4*src size as default).
// If the buffer is too small, it will retry 3 times by doubling the dst size
// After max retries, it will switch to the slower stream API to be sure to be able
// to decompress. Currently switches if compression ratio > 4*2**3=32.
Decompress(dst, src []byte) ([]byte, error)

Stream API

// NewWriter creates a new object that can optionally be initialized with
// a precomputed dictionary. If dict is nil, compress without a dictionary.
// The dictionary array should not be changed during the use of this object.
// You MUST CALL Close() to write the last bytes of a zstd stream and free C objects.
NewWriter(w io.Writer) *Writer
NewWriterLevel(w io.Writer, level int) *Writer
NewWriterLevelDict(w io.Writer, level int, dict []byte) *Writer

// Write compresses the input data and write it to the underlying writer
(w *Writer) Write(p []byte) (int, error)

// Flush writes any unwritten data to the underlying writer
(w *Writer) Flush() error

// Close flushes the buffer and frees C zstd objects
(w *Writer) Close() error
// NewReader returns a new io.ReadCloser that will decompress data from the
// underlying reader.  If a dictionary is provided to NewReaderDict, it must
// not be modified until Close is called.  It is the caller's responsibility
// to call Close, which frees up C objects.
NewReader(r io.Reader) io.ReadCloser
NewReaderDict(r io.Reader, dict []byte) io.ReadCloser

Benchmarks (benchmarked with v0.5.0)

The author of Zstd also wrote lz4. Zstd is intended to occupy a speed/ratio level similar to what zlib currently provides. In our tests, the can always be made to be better than zlib by chosing an appropriate level while still keeping compression and decompression time faster than zlib.

You can run the benchmarks against your own payloads by using the Go benchmarks tool. Just export your payload filepath as the PAYLOAD environment variable and run the benchmarks:

go test -bench .

Compression of a 7Mb pdf zstd (this wrapper) vs czlib:

BenchmarkCompression               5     221056624 ns/op      67.34 MB/s
BenchmarkDecompression           100      18370416 ns/op     810.32 MB/s

BenchmarkFzlibCompress             2     610156603 ns/op      24.40 MB/s
BenchmarkFzlibDecompress          20      81195246 ns/op     183.33 MB/s

Ratio is also better by a margin of ~20%. Compression speed is always better than zlib on all the payloads we tested; However, czlib has optimisations that make it faster at decompressiong small payloads:

Testing with size: 11... czlib: 8.97 MB/s, zstd: 3.26 MB/s
Testing with size: 27... czlib: 23.3 MB/s, zstd: 8.22 MB/s
Testing with size: 62... czlib: 31.6 MB/s, zstd: 19.49 MB/s
Testing with size: 141... czlib: 74.54 MB/s, zstd: 42.55 MB/s
Testing with size: 323... czlib: 155.14 MB/s, zstd: 99.39 MB/s
Testing with size: 739... czlib: 235.9 MB/s, zstd: 216.45 MB/s
Testing with size: 1689... czlib: 116.45 MB/s, zstd: 345.64 MB/s
Testing with size: 3858... czlib: 176.39 MB/s, zstd: 617.56 MB/s
Testing with size: 8811... czlib: 254.11 MB/s, zstd: 824.34 MB/s
Testing with size: 20121... czlib: 197.43 MB/s, zstd: 1339.11 MB/s
Testing with size: 45951... czlib: 201.62 MB/s, zstd: 1951.57 MB/s

zstd starts to shine with payloads > 1KB

Stability - Current state: STABLE

The C library seems to be pretty stable and according to the author has been tested and fuzzed.

For the Go wrapper, the test cover most usual cases and we have succesfully tested it on all staging and prod data.

More Repositories

1

go-profiler-notes

felixge's notes on the various go profiling methods that are available.
Jupyter Notebook
3,255
star
2

glommio

Glommio is a thread-per-core crate that makes writing highly parallel asynchronous applications in a thread-per-core architecture easier for rustaceans.
Rust
3,076
star
3

datadog-agent

Main repository for Datadog Agent
Go
2,859
star
4

stratus-red-team

☁️ ⚡ Granular, Actionable Adversary Emulation for the Cloud
Go
1,752
star
5

dd-agent

Datadog Agent Version 5
Python
1,291
star
6

integrations-core

Core integrations of the Datadog Agent
Python
917
star
7

KubeHound

Tool for building Kubernetes attack paths
Go
737
star
8

dd-trace-go

Datadog Go Library including APM tracing, profiling, and security monitoring.
Go
659
star
9

dd-trace-js

JavaScript APM Tracer
JavaScript
636
star
10

the-monitor

Markdown files for Datadog's longform blog posts: https://www.datadoghq.com/blog/
Python
613
star
11

guarddog

🐍 🔍 GuardDog is a CLI tool to Identify malicious PyPI and npm packages
Python
588
star
12

datadogpy

The Datadog Python library
Python
575
star
13

dd-trace-py

Datadog Python APM Client
Python
519
star
14

dd-trace-java

Datadog APM client for Java
Java
500
star
15

yubikey

YubiKey at Datadog
Shell
494
star
16

dd-trace-php

Datadog PHP Clients
PHP
491
star
17

kafka-kit

Kafka storage rebalancing, automated replication throttle, cluster API and more
Go
485
star
18

documentation

The source for Datadog's documentation site.
HTML
471
star
19

dd-trace-dotnet

.NET Client Library for Datadog APM
C#
443
star
20

security-labs-pocs

Proof of concept code for Datadog Security Labs referenced exploits.
Shell
355
star
21

go-python3

Go bindings to the CPython-3 API
Go
344
star
22

helm-charts

Helm charts for Datadog products
Go
344
star
23

datadog-go

go dogstatsd client library for datadog
Go
332
star
24

terraform-provider-datadog

Terraform Datadog provider
Go
329
star
25

datadog-serverless-functions

Repo of AWS Lambda and Azure Functions functions that process streams and send data to Datadog
Python
326
star
26

docker-dd-agent

Datadog Agent Dockerfile for Trusted Builds.
Roff
302
star
27

browser-sdk

Datadog Browser SDK
TypeScript
302
star
28

datadog-operator

Kubernetes Operator for Datadog Resources
Go
301
star
29

ansible-datadog

Ansible role for Datadog Agent
Jinja
299
star
30

dd-trace-rb

Datadog Tracing Ruby Client
Ruby
261
star
31

threatest

Threatest is a CLI and Go framework for end-to-end testing threat detection rules.
Go
260
star
32

integrations-extras

Community developed integrations and plugins for the Datadog Agent.
Python
252
star
33

watermarkpodautoscaler

Custom controller that extends the Horizontal Pod Autoscaler
Go
210
star
34

pupernetes

Spin up a full fledged Kubernetes environment designed for local development & CI
Go
201
star
35

Miscellany

Miscellaneous scripts and tools
Python
197
star
36

dd-sdk-ios

Datadog SDK for iOS - Swift and Objective-C.
Swift
195
star
37

php-datadogstatsd

A PHP client for DogStatsd
PHP
185
star
38

java-dogstatsd-client

Java statsd client library
Java
175
star
39

sketches-go

Go implementations of the distributed quantile sketch algorithm DDSketch
Go
167
star
40

dogstatsd-ruby

A Ruby client for DogStatsd
Ruby
166
star
41

dd-sdk-android

Datadog SDK for Android (Compatible with Kotlin and Java)
Kotlin
150
star
42

chaos-controller

🐒 🔥 Datadog Failure Injection System for Kubernetes
C
142
star
43

kvexpress

Go program to move data in and out of Consul's KV store.
Go
128
star
44

HASH

HASH (HTTP Agnostic Software Honeypot)
JavaScript
128
star
45

malicious-software-packages-dataset

An open-source dataset of malicious software packages found in the wild, 100% vetted by humans.
Python
125
star
46

ebpf-manager

This manager helps handle the life cycle of your eBPF programs
Go
118
star
47

docker-compose-example

A working example of using Docker Compose with Datadog
Python
116
star
48

orchestrion

Automatic compile-time instrumentation of Go code
Go
115
star
49

dd-sdk-reactnative

Datadog SDK for ReactNative
TypeScript
115
star
50

trace-examples

trace sample apps
Python
113
star
51

sketches-java

DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees.
Java
113
star
52

datadog-lambda-js

The Datadog AWS Lambda Library for Node
TypeScript
105
star
53

gohai

System information collector
Go
102
star
54

build-plugins

A set of universal bundler plugins to interact with Datadog directly from your builds.
TypeScript
101
star
55

jmxfetch

Export JMX metrics
Java
98
star
56

chef-datadog

Chef cookbook for Datadog Agent & Integrations
Ruby
97
star
57

piecewise

Functions for piecewise regression on time series data
Python
96
star
58

extendeddaemonset

Kubernetes Extended Daemonset controller
Go
96
star
59

datadog-static-analyzer

Datadog Static Analyzer
Rust
96
star
60

datadog-api-client-go

Golang client for the Datadog API
Go
95
star
61

ansible-datadog-callback

Ansible callback to get stats & events directly into Datadog http://datadoghq.com
Python
94
star
62

dogstatsd-csharp-client

A DogStatsD client for C#/.NET
C#
94
star
63

gostackparse

Package gostackparse parses goroutines stack traces as produced by panic() or debug.Stack() at ~300 MiB/s.
Go
94
star
64

dogapi-rb

Ruby client for Datadog's API
Ruby
92
star
65

redux-doghouse

Scoping helpers for building reusable components with Redux
JavaScript
90
star
66

ecommerce-workshop

Example eCommerce App for workshops and observability
Ruby
89
star
67

serverless-plugin-datadog

Serverless plugin to automagically instrument your Lambda functions with Datadog
TypeScript
87
star
68

ebpfbench

## Auto-archived due to inactivity. ## profile eBPF programs from Go
Go
87
star
69

datadog-ci

Use Datadog from your CI.
TypeScript
85
star
70

sketches-py

Python implementations of the distributed quantile sketch algorithm DDSketch
Python
83
star
71

datadog-lambda-python

The Datadog AWS Lambda Layer for Python
Python
83
star
72

ddqa

Datadog's QA manager for releases of GitHub repositories
Python
82
star
73

datadog-api-client-typescript

Typescript client for the Datadog API
TypeScript
77
star
74

dirtypipe-container-breakout-poc

Container Excape PoC for CVE-2022-0847 "DirtyPipe"
77
star
75

datadog-trace-agent

Datadog Trace Agent archive (pre-6.10.0)
70
star
76

heroku-buildpack-datadog

Heroku Buildpack to run the Datadog Agent in a Dyno
Shell
69
star
77

datadog-lambda-extension

Rust
68
star
78

datadog-api-client-python

Python client for the Datadog API
Python
68
star
79

datadog-cdk-constructs

CDK construct library to automagically instrument your Lambda functions with Datadog
TypeScript
65
star
80

lading

A suite of data generation and load testing tools
Rust
64
star
81

jsonapi

A marshaler/unmarshaler for JSON:API.
Go
62
star
82

datadog-api-client-java

Java client for the Datadog API
Java
61
star
83

managed-kubernetes-auditing-toolkit

All-in-one auditing toolkit for identifying common security issues in managed Kubernetes environments. Currently supports AWS EKS.
Go
60
star
84

datadog-lambda-go

The Datadog AWS Lambda package for Go
Go
59
star
85

serilog-sinks-datadog-logs

Serilog Sink that sends log events to Datadog https://www.datadoghq.com/
C#
53
star
86

puppet-datadog-agent

Puppet module to install the Datadog agent
Ruby
50
star
87

effective-dashboards

A curated list of useful Datadog dashboards and Dashboard design best practices
47
star
88

opencensus-go-exporter-datadog

Datadog exporter for OpenCensus metrics
Go
47
star
89

gello

## Auto-archived due to inactivity. ## :octocat: A self-hosted server for managing Trello cards based on GitHub webhook events
Python
45
star
90

datadog-cloudformation-resources

Python
44
star
91

ebpf-training

Go
44
star
92

dd-sdk-flutter

Flutter bindings and tools for utilizing Datadog Mobile SDKs
Dart
43
star
93

jenkins-datadog-plugin

ARCHIVED: Current repository is now located https://github.com/jenkinsci/datadog-plugin
Java
42
star
94

synthetics-ci-github-action

Use Browser and API tests in your CI/CD with Datadog Continuous Testing
TypeScript
41
star
95

dd-opentracing-cpp

Datadog Opentracing C++ Client
C++
40
star
96

rum-react-integration-examples

rum-react-integration
TypeScript
39
star
97

fluent-plugin-datadog

Fluentd output plugin for Datadog: https://www.datadog.com
Ruby
38
star
98

datadog-sync-cli

Datadog cli tool to sync resources across organizations.
Python
38
star
99

ddprof

The Datadog Native Profiler for Linux
C++
36
star
100

cloudformation-template

Easily set up the Datadog AWS integration using CloudFormation
Shell
34
star