• Stars
    star
    724
  • Rank 61,160 (Top 2 %)
  • Language
    C
  • License
    Other
  • Created about 8 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Zstd wrapper for Go

Zstd Go Wrapper

CircleCI GoDoc

C Zstd Homepage

The current headers and C files are from v1.5.0 (Commit 10f0e699).

Usage

There are two main APIs:

  • simple Compress/Decompress
  • streaming API (io.Reader/io.Writer)

The compress/decompress APIs mirror that of lz4, while the streaming API was designed to be a drop-in replacement for zlib.

Building against an external libzstd

By default, zstd source code is vendored in this repository and the binding will be built with the vendored source code bundled.

If you want to build this binding against an external static or shared libzstd library, you can use the external_libzstd build tag. This will look for the libzstd pkg-config file and extract build and linking parameters from that pkg-config file.

Note that it requires at least libzstd 1.4.0.

go build -tags external_libzstd

Simple Compress/Decompress

// Compress compresses the byte array given in src and writes it to dst.
// If you already have a buffer allocated, you can pass it to prevent allocation
// If not, you can pass nil as dst.
// If the buffer is too small, it will be reallocated, resized, and returned bu the function
// If dst is nil, this will allocate the worst case size (CompressBound(src))
Compress(dst, src []byte) ([]byte, error)
// CompressLevel is the same as Compress but you can pass another compression level
CompressLevel(dst, src []byte, level int) ([]byte, error)
// Decompress will decompress your payload into dst.
// If you already have a buffer allocated, you can pass it to prevent allocation
// If not, you can pass nil as dst (allocates a 4*src size as default).
// If the buffer is too small, it will retry 3 times by doubling the dst size
// After max retries, it will switch to the slower stream API to be sure to be able
// to decompress. Currently switches if compression ratio > 4*2**3=32.
Decompress(dst, src []byte) ([]byte, error)

Stream API

// NewWriter creates a new object that can optionally be initialized with
// a precomputed dictionary. If dict is nil, compress without a dictionary.
// The dictionary array should not be changed during the use of this object.
// You MUST CALL Close() to write the last bytes of a zstd stream and free C objects.
NewWriter(w io.Writer) *Writer
NewWriterLevel(w io.Writer, level int) *Writer
NewWriterLevelDict(w io.Writer, level int, dict []byte) *Writer

// Write compresses the input data and write it to the underlying writer
(w *Writer) Write(p []byte) (int, error)

// Flush writes any unwritten data to the underlying writer
(w *Writer) Flush() error

// Close flushes the buffer and frees C zstd objects
(w *Writer) Close() error
// NewReader returns a new io.ReadCloser that will decompress data from the
// underlying reader.  If a dictionary is provided to NewReaderDict, it must
// not be modified until Close is called.  It is the caller's responsibility
// to call Close, which frees up C objects.
NewReader(r io.Reader) io.ReadCloser
NewReaderDict(r io.Reader, dict []byte) io.ReadCloser

Benchmarks (benchmarked with v0.5.0)

The author of Zstd also wrote lz4. Zstd is intended to occupy a speed/ratio level similar to what zlib currently provides. In our tests, the can always be made to be better than zlib by chosing an appropriate level while still keeping compression and decompression time faster than zlib.

You can run the benchmarks against your own payloads by using the Go benchmarks tool. Just export your payload filepath as the PAYLOAD environment variable and run the benchmarks:

go test -bench .

Compression of a 7Mb pdf zstd (this wrapper) vs czlib:

BenchmarkCompression               5     221056624 ns/op      67.34 MB/s
BenchmarkDecompression           100      18370416 ns/op     810.32 MB/s

BenchmarkFzlibCompress             2     610156603 ns/op      24.40 MB/s
BenchmarkFzlibDecompress          20      81195246 ns/op     183.33 MB/s

Ratio is also better by a margin of ~20%. Compression speed is always better than zlib on all the payloads we tested; However, czlib has optimisations that make it faster at decompressiong small payloads:

Testing with size: 11... czlib: 8.97 MB/s, zstd: 3.26 MB/s
Testing with size: 27... czlib: 23.3 MB/s, zstd: 8.22 MB/s
Testing with size: 62... czlib: 31.6 MB/s, zstd: 19.49 MB/s
Testing with size: 141... czlib: 74.54 MB/s, zstd: 42.55 MB/s
Testing with size: 323... czlib: 155.14 MB/s, zstd: 99.39 MB/s
Testing with size: 739... czlib: 235.9 MB/s, zstd: 216.45 MB/s
Testing with size: 1689... czlib: 116.45 MB/s, zstd: 345.64 MB/s
Testing with size: 3858... czlib: 176.39 MB/s, zstd: 617.56 MB/s
Testing with size: 8811... czlib: 254.11 MB/s, zstd: 824.34 MB/s
Testing with size: 20121... czlib: 197.43 MB/s, zstd: 1339.11 MB/s
Testing with size: 45951... czlib: 201.62 MB/s, zstd: 1951.57 MB/s

zstd starts to shine with payloads > 1KB

Stability - Current state: STABLE

The C library seems to be pretty stable and according to the author has been tested and fuzzed.

For the Go wrapper, the test cover most usual cases and we have succesfully tested it on all staging and prod data.

More Repositories

1

go-profiler-notes

felixge's notes on the various go profiling methods that are available.
Jupyter Notebook
3,255
star
2

glommio

Glommio is a thread-per-core crate that makes writing highly parallel asynchronous applications in a thread-per-core architecture easier for rustaceans.
Rust
2,907
star
3

datadog-agent

Main repository for Datadog Agent
Go
2,716
star
4

stratus-red-team

โ˜๏ธ โšก Granular, Actionable Adversary Emulation for the Cloud
Go
1,664
star
5

dd-agent

Datadog Agent Version 5
Python
1,291
star
6

integrations-core

Core integrations of the Datadog Agent
Python
878
star
7

the-monitor

Markdown files for Datadog's longform blog posts: https://www.datadoghq.com/blog/
Python
613
star
8

dd-trace-js

JavaScript APM Tracer
JavaScript
605
star
9

datadogpy

The Datadog Python library
Python
575
star
10

dd-trace-go

Datadog Go Library including APM tracing, profiling, and security monitoring.
Go
545
star
11

guarddog

๐Ÿ ๐Ÿ” GuardDog is a CLI tool to Identify malicious PyPI and npm packages
Python
530
star
12

dd-trace-py

Datadog Python APM Client
Python
502
star
13

dd-trace-java

Datadog APM client for Java
Java
500
star
14

yubikey

YubiKey at Datadog
Shell
493
star
15

kafka-kit

Kafka storage rebalancing, automated replication throttle, cluster API and more
Go
480
star
16

dd-trace-php

Datadog PHP Clients
PHP
473
star
17

documentation

The source for Datadog's documentation site.
JavaScript
418
star
18

dd-trace-dotnet

.NET Client Library for Datadog APM
C#
412
star
19

security-labs-pocs

Proof of concept code for Datadog Security Labs referenced exploits.
Shell
355
star
20

go-python3

Go bindings to the CPython-3 API
Go
344
star
21

datadog-go

go dogstatsd client library for datadog
Go
332
star
22

terraform-provider-datadog

Terraform Datadog provider
Go
329
star
23

datadog-serverless-functions

Repo of AWS Lambda and Azure Functions functions that process streams and send data to Datadog
Python
326
star
24

helm-charts

Helm charts for Datadog products
Go
322
star
25

docker-dd-agent

Datadog Agent Dockerfile for Trusted Builds.
Roff
302
star
26

ansible-datadog

Ansible role for Datadog Agent
Jinja
294
star
27

datadog-operator

Datadog Agent Kubernetes Operator
Go
285
star
28

browser-sdk

Datadog Browser SDK
TypeScript
279
star
29

dd-trace-rb

Datadog Tracing Ruby Client
Ruby
261
star
30

threatest

Threatest is a CLI and Go framework for end-to-end testing threat detection rules.
Go
260
star
31

integrations-extras

Community developed integrations and plugins for the Datadog Agent.
Python
243
star
32

watermarkpodautoscaler

Custom controller that extends the Horizontal Pod Autoscaler
Go
207
star
33

pupernetes

Spin up a full fledged Kubernetes environment designed for local development & CI
Go
200
star
34

Miscellany

Miscellaneous scripts and tools
Python
197
star
35

php-datadogstatsd

A PHP client for DogStatsd
PHP
185
star
36

dd-sdk-ios

Datadog SDK for iOS - Swift and Objective-C.
Swift
183
star
37

java-dogstatsd-client

Java statsd client library
Java
177
star
38

dogstatsd-ruby

A Ruby client for DogStatsd
Ruby
166
star
39

sketches-go

Go implementations of the distributed quantile sketch algorithm DDSketch
Go
142
star
40

chaos-controller

๐Ÿ’ ๐Ÿ”ฅ Datadog Failure Injection System for Kubernetes
C
142
star
41

dd-sdk-android

Datadog SDK for Android (Compatible with Kotlin and Java)
Kotlin
140
star
42

kvexpress

Go program to move data in and out of Consul's KV store.
Go
128
star
43

HASH

HASH (HTTP Agnostic Software Honeypot)
JavaScript
119
star
44

docker-compose-example

A working example of using Docker Compose with Datadog
Python
116
star
45

malicious-software-packages-dataset

An open-source dataset of malicious software packages found in the wild, 100% vetted by humans.
Python
116
star
46

ebpf-manager

This manager helps handle the life cycle of your eBPF programs
Go
114
star
47

trace-examples

trace sample apps
Python
113
star
48

sketches-java

DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees.
Java
108
star
49

dd-sdk-reactnative

Datadog SDK for ReactNative
TypeScript
105
star
50

gohai

System information collector
Go
102
star
51

datadog-lambda-js

The Datadog AWS Lambda Library for Node
TypeScript
101
star
52

chef-datadog

Chef cookbook for Datadog Agent & Integrations
Ruby
97
star
53

piecewise

Functions for piecewise regression on time series data
Python
96
star
54

orchestrion

A tool for adding instrumentation to Go code
Go
96
star
55

jmxfetch

Export JMX metrics
Java
96
star
56

extendeddaemonset

Kubernetes Extended Daemonset controller
Go
95
star
57

datadog-api-client-go

Golang client for the Datadog API
Go
95
star
58

dogstatsd-csharp-client

A DogStatsD client for C#/.NET
C#
94
star
59

gostackparse

Package gostackparse parses goroutines stack traces as produced by panic() or debug.Stack() at ~300 MiB/s.
Go
94
star
60

ansible-datadog-callback

Ansible callback to get stats & events directly into Datadog http://datadoghq.com
Python
93
star
61

dogapi-rb

Ruby client for Datadog's API
Ruby
92
star
62

redux-doghouse

Scoping helpers for building reusable components with Redux
JavaScript
90
star
63

build-plugin

Track your build performances like never before.
TypeScript
89
star
64

serverless-plugin-datadog

Serverless plugin to automagically instrument your Lambda functions with Datadog
TypeScript
87
star
65

ecommerce-workshop

Example eCommerce App for workshops and observability
Ruby
86
star
66

datadog-ci

Use Datadog from your CI.
TypeScript
85
star
67

ebpfbench

profile eBPF programs from Go
Go
83
star
68

datadog-lambda-python

The Datadog AWS Lambda Layer for Python
Python
80
star
69

sketches-py

Python implementations of the distributed quantile sketch algorithm DDSketch
Python
77
star
70

dirtypipe-container-breakout-poc

Container Excape PoC for CVE-2022-0847 "DirtyPipe"
77
star
71

datadog-api-client-typescript

Typescript client for the Datadog API
TypeScript
74
star
72

ddqa

Datadog's QA manager for releases of GitHub repositories
Python
73
star
73

datadog-trace-agent

Datadog Trace Agent archive (pre-6.10.0)
70
star
74

heroku-buildpack-datadog

Heroku Buildpack to run the Datadog Agent in a Dyno
Shell
69
star
75

datadog-api-client-python

Python client for the Datadog API
Python
68
star
76

datadog-static-analyzer

Datadog Static Analyzer
Rust
64
star
77

managed-kubernetes-auditing-toolkit

All-in-one auditing toolkit for identifying common security issues in managed Kubernetes environments. Currently supports AWS EKS.
Go
60
star
78

lading

A suite of data generation and load testing tools
Rust
60
star
79

datadog-lambda-extension

Rust
60
star
80

jsonapi

A marshaler/unmarshaler for JSON:API.
Go
59
star
81

datadog-cdk-constructs

CDK construct library to automagically instrument your Lambda functions with Datadog
TypeScript
58
star
82

datadog-lambda-go

The Datadog AWS Lambda package for Go
Go
57
star
83

datadog-api-client-java

Java client for the Datadog API
Java
54
star
84

serilog-sinks-datadog-logs

Serilog Sink that sends log events to Datadog https://www.datadoghq.com/
C#
53
star
85

puppet-datadog-agent

Puppet module to install the Datadog agent
Ruby
50
star
86

opencensus-go-exporter-datadog

Datadog exporter for OpenCensus metrics
Go
47
star
87

gello

:octocat: A self-hosted server for managing Trello cards based on GitHub webhook events
Python
45
star
88

datadog-cloudformation-resources

Python
44
star
89

effective-dashboards

A curated list of useful Datadog dashboards and Dashboard design best practices
44
star
90

ebpf-training

Go
44
star
91

jenkins-datadog-plugin

ARCHIVED: Current repository is now located https://github.com/jenkinsci/datadog-plugin
Java
42
star
92

dd-sdk-flutter

Flutter bindings and tools for utilizing Datadog Mobile SDKs
Dart
40
star
93

dd-opentracing-cpp

Datadog Opentracing C++ Client
C++
40
star
94

synthetics-ci-github-action

Use Browser and API tests in your CI/CD with Datadog Continuous Testing
TypeScript
40
star
95

rum-react-integration-examples

rum-react-integration
TypeScript
39
star
96

fluent-plugin-datadog

Fluentd output plugin for Datadog: https://www.datadog.com
Ruby
38
star
97

import-in-the-middle

Like `require-in-the-middle`, but for ESM import
JavaScript
38
star
98

ddprof

The Datadog Native Profiler for Linux
C++
35
star
99

datadog-sync-cli

Datadog cli tool to sync resources across organizations.
Python
33
star
100

apigentools

Generate API clients with ease
Python
32
star