• Stars
    star
    303
  • Rank 137,655 (Top 3 %)
  • Language
    C
  • License
    Other
  • Created about 10 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Radically unbloated DEFLATE/zlib/gzip compression/decompression library. Can decompress any gzip/zlib data, and offers simplified compressor which produces gzip-compatible output, while requiring much less resources (and providing less compression ratio of course).

uzlib - Deflate/Zlib-compatible LZ77 compression/decompression library

uzlib is a library which can decompress any valid Deflate, Zlib, and Gzip (further called just "Deflate") bitstream, and compress data to Deflate- compatible bitstream, albeit with lower compression ratio than Zlib Deflate algorithm (very basic LZ77 compression algorithm is used instead, static Deflate Huffman tree encoding is used for bitstream).

uzlib aims for minimal code size and runtime memory requirements, and thus suitable for (deeply) embedded systems.

uzlib is based on:

  • tinf library by Joergen Ibsen (Deflate decompression)
  • Deflate Static Huffman tree routines by Simon Tatham
  • LZ77 compressor by Paul Sokolovsky

Library integrated and maintained by Paul Sokolovsky.

(c) 2014-2020 Paul Sokolovsky

uzlib library is licensed under Zlib license.

Decompressor features

Handling of input (compressed) stream:

  • Can reside (fully) in memory.
  • Can be received, byte by byte, from an application-defined callback function (which e.g. can read it from file or another I/O device).
  • Combination of the above: a chunk of input is buffered in memory, when buffer is exhausted, the application callback is called to refill it.

Handling of output (decompressed) stream:

  • In-memory decompression, where output stream fully resides in memory.
  • Streaming decompression, which allows to process arbitrary-sized streams (longer than available memory), but requires in-memory buffer for Deflate dictionary window.
  • Application specifies number of output bytes it wants to decompress, which can be as high as UINT_MAX to decompress everything into memory at once, or as low as 1 to decompress byte by byte, or any other value to decompress a chunk of that size.

Note that in regard to input stream handling, uzlib employs callback-based, "pull-style" design. The control flow looks as follows:

  1. Application requests uzlib to decompress given number of bytes.
  2. uzlib performs decompression.
  3. If more input is needed to decompress given number of bytes, uzlib calls back into application to provide more input bytes. (An implication of this is that uzlib will always return given number of output bytes, unless end of stream (or error) happens).

The original Zlib library instead features "push-style" design:

  1. An application prepares arbitrary number of input bytes in a buffer, and free space in output buffer, and calls Zlib with these buffers.
  2. Zlib tries to decode as much as possible input and produce as much as possible output. It returns back to the application if input buffer is exhausted, or output buffer is full, whatever happens first.

Currently, uzlib doesn't support push-style operation a-la Zlib.

Compressor features

Compressor uses very basic implementation of LZ77 algorithm using hash table to find repeating substrings. The size of the hash table (on which compression efficiency depends), pointer to the hashtable memory, and the size of LZ77 dictionary should be configured in struct uzlib_comp.

Currently, compressor doesn't support streaming operation, both input and output must reside in memory. Neither it supports incremental operation, entire input buffer is compressed at once with a single call to uzlib.

API and configuration

The API is defined in the file uzlib.h and should be largely self-describing. There are also examples implementing gzip-compatible compression and decompression applications in examples/ for further reference. (You may also refer to the original tinf README below for additional information, but it's mostly provided for historical reference, and uzlib largely evolved beyond it).

There are some compile-time options for the library, defined in the file uzlib_conf.h. They can be altered directly in the file, or passed as the compiler options (-DXXX=YYY) when building library.

Binary sizes

To give an impression of code/data sizes of uzlib, the following figures are provided. Numbers for *.o files are code sizes of individual components (tinflate.o is decompressor, genlz77.o and defl_static.o - compressor), and TINF_DATA is the size of the corresponding data structure. These numbers are provided for different architectures, with default uzlib configuration, and with compilers/their options as specified.

gcc -m32 -Os
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
2891 src/tinflate.o
381 src/genlz77.o
1685 src/defl_static.o
1284 TINF_DATA

arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -Os
arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]
1620 src/tinflate.o
180 src/genlz77.o
1131 src/defl_static.o
1284 TINF_DATA

Original tinf library README

For historical reference and to provide proper credit, the original tinf library README follows. NOTE: Many parts no longer apply to uzlib. In particular, API is different, features supported are largely extended, and during decompression, there are checks to avoid erroneous/undefined behavior on incorrect Deflate bitstreams.

tinf - tiny inflate library

Version 1.00

Copyright (c) 2003 Joergen Ibsen

http://www.ibsensoftware.com/

About

tinf is a small library implementing the decompression algorithm for the deflate compressed data format (called 'inflate'). Deflate compression is used in e.g. zlib, gzip, zip and png.

I wrote it because I needed a small in-memory zlib decompressor for a self- extracting archive, and the zlib library added 15k to my program. The tinf code added only 2k.

Naturally the size difference is insignificant in most cases. Also, the zlib library has many more features, is more secure, and mostly faster. But if you have a project that calls for a small and simple deflate decompressor, give it a try :-)

While the implementation should be fairly compliant, it does assume it is given valid compressed data, and that there is sufficient space for the decompressed data.

Simple wrappers for decompressing zlib streams and gzip'ed data in memory are supplied.

tgunzip, an example command-line gzip decompressor in C, is included.

The inflate algorithm and data format are from 'DEFLATE Compressed Data Format Specification version 1.3' (RFC 1951).

The zlib data format is from 'ZLIB Compressed Data Format Specification version 3.3' (RFC 1950).

The gzip data format is from 'GZIP file format specification version 4.3' (RFC 1952).

Ideas for future versions:

  • the fixed Huffman trees could be built by tinf_decode_trees() using a small table
  • memory for the TINF_DATA object should be passed, to avoid using more than 1k of stack space
  • wrappers for unpacking zip archives and png images
  • implement more in x86 assembler
  • more sanity checks
  • in tinf_uncompress, the (entry value of) destLen and sourceLen are not used
  • blocking of some sort, so everything does not have to be in memory
  • optional table-based huffman decoder

Functionality

void tinf_init();

Initialise the global uninitialised data used by the decompression code. This function must be called once before any calls to the decompression functions.

int tinf_uncompress(void *dest,
                    unsigned int *destLen,
                    const void *source,
                    unsigned int sourceLen);

Decompress data in deflate compressed format from source[] to dest[]. destLen is set to the length of the decompressed data. Returns TINF_OK on success, and TINF_DATA_ERROR on error.

int tinf_gzip_uncompress(void *dest,
                         unsigned int *destLen,
                         const void *source,
                         unsigned int sourceLen);

Decompress data in gzip compressed format from source[] to dest[]. destLen is set to the length of the decompressed data. Returns TINF_OK on success, and TINF_DATA_ERROR on error.

int tinf_zlib_uncompress(void *dest,
                         unsigned int *destLen,
                         const void *source,
                         unsigned int sourceLen);

Decompress data in zlib compressed format from source[] to dest[]. destLen is set to the length of the decompressed data. Returns TINF_OK on success, and TINF_DATA_ERROR on error.

unsigned int tinf_adler32(const void *data,
                          unsigned int length);

Computes the Adler-32 checksum of length bytes starting at data. Used by tinf_zlib_uncompress().

unsigned int tinf_crc32(const void *data,
                        unsigned int length);

Computes the CRC32 checksum of length bytes starting at data. Used by tinf_gzip_uncompress().

Source Code

The source code is ANSI C, and assumes that int is 32-bit. It has been tested on the x86 platform under Windows and Linux.

The decompression functions should be endian-neutral, and also reentrant and thread-safe (not tested).

In src/nasm there are 32-bit x86 assembler (386+) versions of some of the files.

Makefiles (GNU Make style) for a number of compilers are included.

Frequently Asked Questions

Q: Is it really free? Can I use it in my commercial ExpenZip software?

A: It's open-source software, available under the zlib license (see later), which means you can use it for free -- even in commercial products. If you do, please be kind enough to add an acknowledgement.

Q: Did you just strip stuff from the zlib source to make it smaller?

A: No, tinf was written from scratch, using the RFC documentation of the formats it supports.

Q: What do you mean by: 'the zlib library .. is more secure'?

A: The zlib decompression code checks the compressed data for validity while decompressing, so even on corrupted data it will not crash. The tinf code assumes it is given valid compressed data.

Q: I'm a Delphi programmer, can I use tinf?

A: Sure, the object files produced by both Borland C and Watcom C should be linkable with Delphi.

Q: Will tinf work on UltraSTRANGE machines running WhackOS?

A: I have no idea .. please try it out and let me know!

Q: Why are all the makefiles in GNU Make style?

A: I'm used to GNU Make, and it has a number of features that are missing in some of the other Make utilities.

Q: This is the first release, how can there be frequently asked questions?

A: Ok, ok .. I made the questions up ;-)

License

tinf - tiny inflate library

Copyright (c) 2003 Joergen Ibsen

This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software.

Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions:

  1. The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required.

  2. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software.

  3. This notice may not be removed or altered from any source distribution.

More Repositories

1

esp-open-sdk

Free and open (as much as possible) integrated SDK for ESP8266/ESP8285 chips
Makefile
1,971
star
2

picotui

Lightweight, pure-Python Text User Interface (TUI) widget toolkit with minimal dependencies. Dedicated to the Pycopy project.
Python
808
star
3

pycopy

Pycopy - a minimalist and memory-efficient Python dialect. Good for desktop, cloud, constrained systems, microcontrollers, and just everything.
C
764
star
4

awesome-smarthome

Curated list of awesome SmartHome/Home Automation things (open and leaving users in control)
487
star
5

picoweb

Really minimal web application framework for the Pycopy project (minimalist Python dialect) and its "uasyncio" async framework
Python
474
star
6

ScratchABit

Easily retargetable and hackable interactive disassembler with IDAPython-compatible plugin API
Python
392
star
7

pycopy-lib

Standard library of the Pycopy project, minimalist and light-weight Python language implementation
Python
247
star
8

awesome-micropython

Curated list of awesome MicroPython resources
175
star
9

awesome-program-analysis

Program Analisys and Transformation survey and links (particular focus on SSA)
153
star
10

blutunode

Autonomous wireless sensor/actuator node using bluetooth modules based on CSR BlueCore chipset. See wiki for more info.
C
149
star
11

foreign-dlopen

Small library allowing to use dlopen() from statically-linked applications (where statically-linked executable vs loaded shared library may use completely different libc's)
C
143
star
12

awesome-linux-android-hacking

List of hints and Q&As to get most of your Linux/Android device
127
star
13

ssabook

Mirror of InriaForge SSABook repository: https://gforge.inria.fr/projects/ssabook/ (was scheduled for retirement at the end of 2020, was still online as of 2021-03, but then gone by 2021-09).
TeX
125
star
14

yaota8266

Yet another OTA solution for ESP8266, this time supporting large (>512KB) firmwares even on 1MB devices (repo is rebased)
C
119
star
15

ScratchABlock

Yet another crippled decompiler project
Python
103
star
16

awesome-python-compilers

Retrospective of Python compilation efforts
62
star
17

utemplate

Micro template engine in Python with low memory usage, designed for Pycopy, a minimalist Python dialect, but also compatible with other Pythons.
Python
60
star
18

graph-llvm-ir

Visualization of LLVM IR
Python
55
star
19

PeripheralTemplateLibrary

Cross-platform, cross-MCU C++ template library for microcontrollers and peripheral devices
C++
54
star
20

android-platform-headers

Collection of platform headers and link libs for all versions of Android. (repo is rebased, see wiki)
C
54
star
21

visited_places

Simple, static, local visited places map (aka travel map) using D3 and D3 Datamaps
HTML
49
star
22

re1.5

re1, the unbloated regexp engine by Russ Cox, elaborated to be useful for real-world applications
C
41
star
23

python-compiler.old

Python2 compiler package ported to Python3. Compiles Python AST (as produced by the "ast" module) to bytecode assembly and code objects.
Python
41
star
24

python-imphook

Simple and clear import hooks for Python - import anything as if it were a Python module
Python
36
star
25

optware-android

Optware setup script for Android devices. Not actively maintained any more. You may want to look at https://github.com/Entware-ng/Entware-ng/wiki/Install-on-Android
Shell
35
star
26

notes-pico

Flask Notes sample application ported to the Pycopy's picoweb web micro-framework
Python
35
star
27

ida-xtensa2

IDAPython plugin for Tensilica Xtensa (as seen in ESP8266), version 2
Python
34
star
28

cortex-uni-startup

Unified startup code and link scripts for Cortex-M microcontrollers
C
32
star
29

pyflate

Pure Python zlib/gzip/bzip2 decompessor/bitstream dumper, plus additional utilities
Python
28
star
30

berkeley-db-1.xx

Berkeley DB 1.85 with patches and fixes applied
C
24
star
31

awesome-implicit-data-structures

Awesome implicit data structures
23
star
32

xtensa-subjects

Xtensa CPU architecture (ESP8266) binaries for ScratchABit interactive disassembler
HTML
22
star
33

esp-open-lwip

Superseded by https://github.com/pfalcon/lwip-esp8266. Was: Untangled build of vendor ESP8266 lwIP library
C
22
star
34

pymsasid3

Pure-Python x86 disassembler, ported to modern Python, with bugfixes
Python
21
star
35

lwip-esp8266

Upstream lwIP with complete history and cleaned up ESP8266 patchset on top
C
18
star
36

idapython

Automatically exported from code.google.com/p/idapython
Python
18
star
37

libperipha

Grand unified collection of headers to access various hardware chips and components
C
18
star
38

pyedit

Simple text editor widget for Python and Pycopy. Further developed in https://github.com/pfalcon/picotui
Python
18
star
39

squirrel-modules

General-purpose Squirrel language project. Includes collection of basic modules (including module loading infrastructure)
C++
17
star
40

pycopy-serial

pySerial-like interface for Pycopy (unix port)
Python
15
star
41

pyastinterp

Python AST interpreter (aka tree-walking interpreter) in Python. Aka meta-circular interpreter.
Python
15
star
42

opkg-static

Static self-contained opkg (the package manager) build for real-world embedded platforms like Android, Kindle, etc.
Shell
14
star
43

awesome-virtual-machines

Awesome Virtual Machines (VMs) and Language Runtimes (awesome from Computer Science perspective)
14
star
44

uorm

Sqlite3-based anti-ORM for Pycopy
Python
14
star
45

ctopy

Hacking on ESR's ctopy the C to Python conversion tool: http://www.catb.org/~esr/ctopy/
Python
14
star
46

papersman

Minimalist electronic documents/papers/publications manager/indexer/categorizer
Python
14
star
47

android-native

Android native (command-line) examples, utils and tools. See Wiki for more info.
C++
13
star
48

uart-bitbang

UART protocol for MCU using GPIO bitbang. Rate-adaptive and requires only clock counter to operate.
C
13
star
49

beap

Beap (bi-parental heap) algorithm reference implementation in Python
Python
13
star
50

esp-docs

Helping Espressif Systems, Inc. achieve 12-year product longevity (mirror of ESP8266, etc. documentation)
12
star
51

pycopy-projs

Various Pycopy-related proofs of concepts and hacks which don't deserve dedicated repos
Python
11
star
52

canterbury-corpus

The Canterbury compression corpus as a git repository
HTML
10
star
53

ullvm_c

Lightweight LLVM C API bindings for Pycopy and other Python implementations
Python
10
star
54

apps2org

Label/tag based application organizer and launcher for Android, fork of https://code.google.com/p/appsorganizer
Java
10
star
55

aes256_128

Code size efficient AES256/AES128 implementation
C
10
star
56

pycopy-jitgen

Generating machine code at runtime (aka JIT) using Pycopy (small Python dialect)
Python
10
star
57

uremi

Proof of concept of developing alike of https://github.com/dddomodossola/remi for Pycopy
Python
9
star
58

ppxml2db

Scripts to import PortfolioPerformance (https://github.com/portfolio-performance/portfolio) XML into a SQLite DB and export back
Python
9
star
59

sphinx_selective_exclude

Sphinx extension (plugin) to make ".only::" directive work like you expect. (Plus some other goodies for selective indexes.) [Unmaintained, may no longer work with latest Sphinx versions.]
Python
9
star
60

esp8266-re-wiki-mirror

Mirror of http://esp8266-re.foogod.com/wiki/
8
star
61

pycopy-filedb

Simple file-based ORM for Pycopy
Python
8
star
62

Chsmartbulb-led-bulb-speaker

Hacking "Chsmartbulb" BT/BLE LED bulb speaker
Python
8
star
63

llvm-codegen-py

(Machine) code generation experiments in Python, roughly centered around LLVM IR
Python
7
star
64

pymapfile

Python module to parse GNU ld/gcc map files
Python
7
star
65

arduino-hosted

Arduino Hosted Python module, allows to rapid-prototype simple Arduino microcontroller applications on desktop computer. Unlike many other Python wrappers, this provides syntax as close as possible to Arduino.
Python
7
star
66

pycopy-btreedb

Very simple ORM for Pycopy's btree module
Python
6
star
67

axtls

Pycopy fork of axTLS
C
6
star
68

pycopy-ffigen

FFI bindings generator for Pycopy
Python
6
star
69

simplejtag

Protocol and firmware to turn any low-cost board into JTAG/SWD adapater to use wiyh OpenOCD, PySWD, etc.
C++
6
star
70

picompile

Pico Compile, factored out "numpile" project, a small JIT compiler for Python with type inference
Python
6
star
71

android-depends

Tool to analyze Android module dependencies. Forked from http://code.google.com/p/rxwen-blog-stuff/source/browse/trunk/tools/
Python
6
star
72

awesome-python-projects

List of Python projects which either do something non-trivial, or are unbloated and lightweight. (See wiki)
6
star
73

parcopy

Implementation (in Python) of algorithm(s) for sequentializing of parallel copy
Python
6
star
74

udownmark

A dead simple parser/renderer for Markdown-like text markup (subset of Markdown). Dedicated to Pycopy, a minimalist Python implementation (https://github.com/pfalcon/micropython).
Python
5
star
75

change-control-manifesto

Change Control Manifesto for principled Open Source projects
5
star
76

esp-sdk-tools

Tools for hacking on Espressif ESP8266/ESP32 SDKs
Shell
4
star
77

py-runinpkg

Run Python scripts inside package directory as normal files, even if they use relative imports
Python
3
star
78

pseudoc-ir

Like LLVM, but simpler
Python
3
star
79

awesome-consumer-iot-hacks

A Collection of Hacks in Consumer IoT Space so that we can use those products (hopefully).
3
star
80

aygshell-win32

Implementation of WinCE AYGSHELL functions for pure Win32 API
C
3
star
81

ARM-CMSIS-BSD

BSD-licensed CMSIS core implementation from ARM, imported from CMSIS-SP-00300-r3p2-00rel1.zip
C
3
star
82

b43-tools

Mirror of git://git.bues.ch/b43-tools.git "Tools for the Broadcom 43xx series WLAN chip"
C
3
star
83

pycopy-uffmpeg

Pycopy bindings for FFmpeg
Python
3
star
84

cppmicroweb

Proof of concept C++ web micro framework for constrained devices inspired by Python microframeworks
C++
3
star
85

pycopy-dlman

Very simple download manager based on wget and Pycopy
Python
2
star
86

uaio_xiaomi_gw

Pycopy uasyncio module for Xiaomi Mi Home Gateway
Python
2
star
87

optware

Mirror of NSLU2 Optware + updates
C
2
star
88

openwrt-tools

Tools to automatically install/configure/manage OpenWRT
Shell
2
star
89

esp-open-headers

Headers for ESP8266 SDK internal functions and structures
C
2
star
90

awesome-stuff

Projects I'd like to record somewhere
2
star
91

dotfiles

Personal dotfiles
Shell
2
star
92

py-simple-serial

Very simple Python serial port module for Linux, roughly compatible with PySerial
Python
2
star
93

pycopy-libusb

libusb bindings for Pycopy
Python
2
star
94

b43-ucode

Mirror of git://git.bues.ch/b43-ucode.git "OpenSource firmware for Broadcom 43xx devices"
Assembly
2
star
95

meta-prefixdistro

OpenEmbedded layer for simlified distro rooted at atbitrary filesystem prefix
1
star
96

python-simple-mixin

Simple mixins for Python without multiple inheritance
Python
1
star
97

git-pynex

Proof of concept reimplementation of subset of git-annex in Python
Python
1
star
98

osmand-gpx-vis

Visualizing GPX tracks recorded by Osmand Android navigation software and other tools
Python
1
star
99

snapdragon410-mmio-gpio

Memory-mapped GPIO access on Snapdragon 410 SoC using MicroPython (and CPython)
Python
1
star
100

fdroidclient

Mirror (+local dev) of client app for F-Droid, FOSS Android package repository.
Java
1
star