• Stars
    star
    128
  • Rank 281,044 (Top 6 %)
  • Language
    C++
  • License
    BSD 3-Clause "New...
  • Created about 11 years ago
  • Updated about 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Shorthair : Generational Block Streaming Erasure Codes

Shorthair has been deprecated by CauchyCaterpillar

Please check out the newer codec here: https://github.com/catid/CauchyCaterpillar

Shorthair Low-Latency Networking

Shorthair is a portable C++ library uses the Longhair library to provide low-latency loss-recovery, which uses extra bandwidth to make almost any unreliable packet channel suitable for real-time communication.

Erasure codes in software are fast with low-overhead.

Advantages of using erasure codes in online games:

  • 50% faster delivery over UDP than over TCP

  • 50% less bandwidth used than naive redundancy

  • 50% better recovery rate than parity redundancy

Improves the experience for multiplayer mobile apps.

Please take a look at my presentation on Erasure codes for more motivation: Erasure Codes in Software

Features

It is designed for applications that send no more than 2000 packets per second (about 2 MB/s). Beyond that, it will hit asserts and not be able to protect the traffic.

  • Full-duplex communication over a lossy channel.
  • Automatically calculates the amount of redundancy to add based on channel conditions.
  • Uses a fast Reed-Solomon block codec: Longhair
  • It supports variable-length data packets.
  • Overhead is 5 bytes/packet.

Usage

Step 1: Add the source code to your project. #include "Shorthair.hpp"

Step 2: Your C++ code should have a class instance that derives from IShorthair and implements its interface:

class MyClass : public IShorthair
{
	void OnPacket(u8 *packet, int bytes) override
	{
		// Called with the latest data packet from remote host
	}

	void OnOOB(u8 *packet, int bytes) override
	{
		// Called with the latest OOB packet from remote host
	}

	void SendData(u8 *buffer, int bytes) override
	{
		// Send raw data to remote host over UDP socket
	}
...

Step 3: Create a Shorthair object on the server and client side. Initialize it with some settings for your application:

    cat::shorthair::ShorthairCodec CodecObject;

    Settings settings;
    settings.min_fec_overhead = 0.1f; // 10%+
    settings.max_delay = 100; // 100 milliseconds
    settings.max_data_size = 1400; // Normal UDP packet
    settings.interface_ptr = &myClass; // Pointer to IShorthair object

    CodecObject.Initialize(settings);

Step 4: When sending a FEC-protected packet, call the ShorthairCodec::Send method. When sending an unprotected packet, call the ShorthairCodec::SendOOB() method:

    char message[3] = {};
    CodecObject.Send(message, sizeof(message));

Packets sent with Send can still be duplicated by the network, but they will be protected by FEC and be much less likely to be lost.

    char message[3] = {};
    CodecObject.SendOOB(message, sizeof(message));

Packets sent with SendOOB will not be protected by FEC. It is appropriate to send statistics and data that changes every packet with SendOOB.

It's safe to call the Send functions from any thread.

Step 5: Periodically call ShorthairCodec::Tick(). Around a 10-20 ms interval is best.

Step 6: When a packet is received, call ShorthairCodec::Recv() with the raw UDP packet data.

The ShorthairCodec object is not thread-safe, so be sure to hold a lock if multiple threads are using it.

Credits

Software by Christopher A. Taylor [email protected]

Please reach out if you need support or would like to collaborate on a project.

More Repositories

1

supercharger

Supercharge Open-Source AI Models
Python
348
star
2

dora

Implementation of DoRA
Python
276
star
3

Zpng

Better lossless compression than PNG with a simpler algorithm
C
267
star
4

wirehair

Wirehair : O(N) Fountain Code for Large Data
C++
267
star
5

self-discover

Implementation of Google's SELF-DISCOVER
Python
263
star
6

WLANOptimizer

Single-header C library that fixes WiFi performance issues for online gaming and other low-latency real-time network traffic.
C++
223
star
7

longhair

Longhair : O(N^2) Cauchy Reed-Solomon Block Erasure Code for Small Data
C++
156
star
8

leopard

Leopard-RS : O(N Log N) MDS Reed-Solomon Block Erasure Code for Large Data
C++
135
star
9

TimeSync

TimeSync: Time Synchronization Library in Portable C++
C++
122
star
10

cm256

Fast GF(256) Cauchy MDS Block Erasure Codec in C
C++
107
star
11

tonk

Tonk : Reliable UDP (rUDP) Network Library and Infinite Window Erasure Code
C++
101
star
12

Zdepth

Zdepth :: Streaming Depth Compressor in C++ for Azure Kinect DK
C++
97
star
13

xrcap

Azure Kinect multi-camera secure network capture/record/replay
C++
76
star
14

snowshoe

Snowshoe - Portable, Secure, Fast Elliptic Curve Math Library in C
C++
62
star
15

XRmonitors

XRmonitors : User-Friendly Virtual Multi-Monitors for the Workplace
C++
54
star
16

tabby

Tabby - Strong, Fast, and Portable Cryptographic Signatures, Handshakes, and Password Authentication
C++
50
star
17

gf256

GF256 - Fast 8-bit Galois Field Math in C
C++
50
star
18

kvm

Low-Bandwidth IP KVM using Raspberry Pi 4
C++
48
star
19

siamese

Siamese : Infinite-Window Streaming Erasure Code (HARQ)
C++
47
star
20

bitnet_cpu

Experiments with BitNet inference on CPU
C++
46
star
21

ZdepthLossy

Lossy version of Zdepth using video encoders
C++
41
star
22

fecal

FEC-AL : O(N^2) Fountain Code for Small Data
C++
36
star
23

CauchyCaterpillar

Cauchy Caterpillar : O(N^2) Short-Window Streaming Erasure Code
C++
35
star
24

aiwebcam2

Second attempt at AI webcam, this time with OpenAI API
Python
32
star
25

upsampling

Image Upsampling with PyTorch
Python
22
star
26

calico

Calico - Strong, Fast, and Portable Authenticated Encryption
C++
21
star
27

cymric

Cymric - Portable secure random number generator
C++
20
star
28

loraftp

File transfer between two Raspberry Pis using the LoRa Pi HAT from Waveshare
C
19
star
29

mau

Network simulator for reliable UDP testing in C++
C++
16
star
30

oaillama3

Simple setup to self-host LLaMA3-70B model with an OpenAI API
16
star
31

lllm

Latent Large Language Models
Python
16
star
32

PacketAllocator

C++ Memory allocator for packet queues that free() in roughly the same order that they alloc().
C++
15
star
33

spectral_ssm

Implementation of Spectral State Space Models
Python
15
star
34

libcat

Common code library
C++
14
star
35

AutoAudiobook

Automatically create an audiobook using OpenAI
Python
14
star
36

minigpt4

MiniGPT-4 :: Updated to Torch 2.0, simple setup, easier API, cut out training code
Python
13
star
37

sdxl

SDXL GPU cluster scripts
Python
13
star
38

dataloader

High-performance tokenized language data-loader for Python C++ extension
C++
12
star
39

fp61

Experiment: Fast finite field Fp=2^61-1 in C++
C++
11
star
40

cuda_float_compress

Python package for compressing floating-point PyTorch tensors
Cuda
10
star
41

phind

Locally hosted: 60% HumanEval
Python
8
star
42

rtmp_receiver

Simple unidirectional RTMP video stream receiver
C++
7
star
43

counter

C++ wrapper for counters that can roll-over (e.g. timestamps/ack-ids)
C++
6
star
44

z16

16-bit monochrome image compressor based on Zstd
C
5
star
45

AQLM

Fixes for AQLM
Python
5
star
46

llamanal.cpp

Static code analysis for C++ projects using llama.cpp and the best LLM you can run offline without an expensive GPU.
C
5
star
47

unfiltered-diffusers

Simple fork that disables NSFW filter
Python
5
star
48

hloc

Python
4
star
49

boss-balloon

BossBalloon.io
TypeScript
4
star
50

libcatid

Automatically exported from code.google.com/p/libcatid
C++
3
star
51

voron

Voron 3D Printer files
3
star
52

halide-test

Test v14/v15 performance regression
CMake
3
star
53

logger

Feature-rich portable C++ logging subsystem in 650 lines of code
C++
3
star
54

cifar10deepspeed

Using DeepSpeed and Nvidia DALI to train various models to solve CIFAR-10
Python
3
star
55

chainlit-anthropic

Chainlit AI UI with Anthropic Backend
Python
3
star
56

fastest_gf_matrix_mult

A fairly hacked together piece of code that can quickly search for the optimal GF polynomials and Cauchy matrices for XOR-based GF matrix multiplication for erasure code encoders. May be useful if the matrices are of fixed size.
C++
3
star
57

recfilter-2020-fail

This is a failed attempt to port Recfilter to latest Halide from 2020. Maybe someone else can figure this out?
C++
2
star
58

bentpipe

Simple UDP rebroadcaster
C++
2
star
59

whisper3

Testing out Whisper 3
Python
2
star
60

sphynx

Sphynx - High Performance Network Transport Layer
C++
2
star
61

textworld_llm_benchmark

TextWorld LLM Benchmark
Python
2
star
62

pixel-perfect-sfm

pixel-perfect-sfm with some minor fixes
C++
2
star
63

train_ticket

Gated PRNGs are all you need? Spoilers: No.
Python
2
star
64

CatsChoice

PRNG Parameter Generation
C++
1
star
65

CRC16Recovery

Optimized CRC16 with error recovery in C
C++
1
star
66

Splane

Archived old code - /ban_ids/ is kind of interesting
C
1
star
67

rust_webgl_demo

Hello World : Rust Web Assembly
Rust
1
star
68

blog2022

HTML
1
star
69

swe_agent_playground

swe_agent_playground
1
star
70

Exapunks_Solutions

Exapunks Walkthrough Solutions
1
star
71

audio_prediction

Simple audio prediction example with RNN
Python
1
star
72

never_forget

Implementation of Overcoming Catastrophic Forgetting
Python
1
star
73

DependencyInjected

DependencyInjected : Light-weight and powerful Dependency Injection pattern for C++
C++
1
star
74

quicsend

quicsend :: Super-fast Internet-ready file transfer right from Python
C++
1
star