• Stars
    star
    119
  • Rank 297,930 (Top 6 %)
  • Language
    C++
  • License
    Other
  • Created almost 5 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

πŸ—œοΈ An Arduino library to unpack/uncompress tar, gz, and tar.gz files on ESP32 and ESP8266

πŸ—œοΈ ESP32-targz

An ESP32/ESP8266/RP2040 Arduino library to provide decompression support for .tar, .gz and .tar.gz files

arduino-library-badge PlatformIO Registry

ES32-targz logo

This library is a wrapper for the following two great libraries:

This library enables the channeling of gz ➑️ tar ➑️ filesystem data without using an intermediate file (bug: see #4).

In order to reach this goal, TinyUntar was heavily modified to allow data streaming, uzlib is also customized.

Tradeoffs

When the output is the filesystem (e.g. NOT when streaming to TAR), gzip can work without the dictionary. Disabling the dictionary can cause huge slowdowns but saves ~36KB of ram.

TinyUntar requires 512bytes only so its memory footprint is negligible.

Limitations

ESP32-TarGz can only have one output filesystem (see Support Matrix), and it must be set at compilation time (see Usage). This limitation does not apply to the input filesystem/stream.

Scope

  • This library is only for unpacking / decompressing, no compression support is provided whatsoever
  • Although the examples use SPIFFS as default, it should work with any fs::FS filesystem (SD, SD_MMC, FFat, LittleFS) and streams (HTTP, HTTPS, UDP, CAN, Ethernet)
  • This is experimental, expect bugs!
  • Contributions and feedback are more than welcome :-)

Support Matrix

fs::FS SPIFFS LittleFS SD SD_MMC FFAT
ESP32 1.0 1.0.5 1.0.5 1.0 1.0
ESP8266 builtin 0.1.0 0.1.0 n/a n/a
RP2040 n/a n/a 2.0.0 n/a n/a

Usage

⚠️ Important note: setting the #define before including <ESP32-targz.h> is mandatory otherwise it will be ignored and the library will default to SPIFFS.

    // Set **destination** filesystem by uncommenting one of these:
    //#define DEST_FS_USES_SPIFFS
    //#define DEST_FS_USES_FFAT
    //#define DEST_FS_USES_SD
    //#define DEST_FS_USES_SD_MMC
    #define DEST_FS_USES_LITTLEFS
    #include <ESP32-targz.h>
    // filesystem object will be available as "tarGzFS"

Extract content from .gz file

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    GzUnpacker *GZUnpacker = new GzUnpacker();

    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity

    // expand one file
    if( !GZUnpacker->gzExpander(tarGzFS, "/gz_example.gz", tarGzFS, "/gz_example.jpg") ) {
      Serial.printf("gzExpander failed with return code #%d", GZUnpacker->tarGzGetError() );
    }

    // expand another file
    if( ! gzExpander(tarGzFS, "/blah.gz", tarGzFS, "/blah.jpg") ) {
      Serial.printf("operation failed with return code #%d", GZUnpacker->tarGzGetError() );
    }

Expand contents from .tar file to /tmp folder

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    TarUnpacker *TARUnpacker = new TarUnpacker();

    TARUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    if( !TARUnpacker->tarExpander(tarGzFS, "/tar_example.tar", tarGzFS, "/") ) {
      Serial.printf("tarExpander failed with return code #%d\n", TARUnpacker->tarGzGetError() );
    }

Expand contents from .tar.gz to /tmp folder

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    // using an intermediate file (default is /tmp/tmp.tar)
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/targz_example.tar.gz", tarGzFS, "/tmp") ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }

    // or without intermediate file
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/targz_example.tar.gz", tarGzFS, "/tmp", nullptr ) ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }

Flash the ESP with contents from .gz file

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    GzUnpacker *GZUnpacker = new GzUnpacker();

    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity

    if( ! GZUnpacker->gzUpdater(tarGzFS, firmwareFile, U_FLASH,/*don't restart after update*/false ) ) {
      Serial.printf("gzUpdater failed with return code #%d\n", GZUnpacker->tarGzGetError() );
    }

ESP32 Only: Flash the ESP with contents from .gz stream

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    fs::File file = tarGzFS.open( "/example_firmware.gz", "r" );

    if (!file) {
      Serial.println("Can't open file");
      return;
    }

    GzUnpacker *GZUnpacker = new GzUnpacker();

    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity

    if( !GZUnpacker->gzStreamUpdater( (Stream *)&file, UPDATE_SIZE_UNKNOWN ) ) {
      Serial.printf("gzStreamUpdater failed with return code #%d\n", GZUnpacker->tarGzGetError() );
    }

ESP32 Only: Direct expansion (no intermediate file) from .tar.gz. stream

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    fs::File file = tarGzFS.open( "/example_archive.tgz", "r" );

    if (!file) {
      Serial.println("Can't open file");
      return;
    }

    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    if( !TARGZUnpacker->tarGzStreamExpander( (Stream *)&file, tarGzFS ) ) {
      Serial.printf("tarGzStreamExpander failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }

ESP32 Only: Direct Update (no intermediate file) from .tar.gz. stream

    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( false ); // nothing to verify as we're writing a partition
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::targzNullProgressCallback ); // don't care about gz progress
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual partition
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( myTarMessageCallback/*BaseUnpacker::targzPrintLoggerCallback*/ ); // tar log verbosity

    // mount SD
    SD.begin();

    // this .tar.gz file has both the "app.ino.bin" and "app.spiffs.bin" partitions
    fs::File file = SD.open( "/bundle_firmware.tar.gz", "r" );

    if (!file) {
      Serial.println("Can't open file");
      return;
    }

    // this could also be a HTTP/HTTPS/UDP/Ethernet Stream
    Stream *streamptr = &file;

    if( !TARGZUnpacker->tarGzStreamUpdater( streamptr ) ) {
      Serial.printf("tarGzStreamUpdater failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    } else {
      Serial.println( "Flashing successful, now restarting" );
      ESP.restart();
    }

Callbacks

    // basic progress callback (valid for tar or gzip)
    void myBasicProgressCallback( uint8_t progress )
    {
      Serial.printf("Progress: %d\n", progress );
    }


    // complex progress callback (valid for tar or gzip)
    void myProgressCallback( uint8_t progress )
    {
      static int8_t myLastProgress = -1;
      if( myLastProgress != progress ) {
        if( myLastProgress == -1 ) {
          Serial.print("Progress: ");
        }
        myLastProgress = progress;
        switch( progress ) {
          case   0: Serial.print("0% β–“");  break;
          case  25: Serial.print(" 25% ");break;
          case  50: Serial.print(" 50% ");break;
          case  75: Serial.print(" 75% ");break;
          case 100: Serial.print("β–“ 100%\n"); myLastProgress = -1; break;
          default: if( progress < 100) Serial.print( "β–“" ); break;
        }
      }
    }


    // General Error/Warning/Info logger
    void myLogger(const char* format, ...)
    {
      va_list args;
      va_start(args, format);
      vprintf(format, args);
      va_end(args);
    }


    // status callback for TAR (fired at file creation)
    void myTarStatusProgressCallback( const char* name, size_t size, size_t total_unpacked )
    {
      Serial.printf("[TAR] %-64s %8d bytes - %8d Total bytes\n", name, size, total_unpacked );
    }

Return Codes

*Unpacker->tarGzGetError() returns a value when a problem occured:

  • General library error codes

    • 0 : Yay no error!
    • -1 : Filesystem error
    • -6 : Same a Filesystem error
    • -7 : Update not finished? Something went wrong
    • -38 : Logic error during deflating
    • -39 : Logic error during gzip read
    • -40 : Logic error during file creation
    • -100 : No space left on device
    • -101 : No space left on device
    • -102 : No space left on device
    • -103 : Not enough heap
    • -104 : Gzip dictionnary needs to be enabled
    • -105 : Gz Error when parsing header
    • -106 : Gz Error when allocating memory
    • -107 : General error, file integrity check fail
  • UZLIB: forwarding error values from uzlib.h as is (no offset)

    • -2 : Not a valid gzip file
    • -3 : Gz Error TINF_DATA_ERROR
    • -4 : Gz Error TINF_CHKSUM_ERROR
    • -5 : Gz Error TINF_DICT_ERROR
    • -41 : Gz error, can't guess filename
  • UPDATE: applying -20 offset to forwarded error values from Update.h

    • -8 : Updater Error UPDATE_ERROR_ABORT
    • -9 : Updater Error UPDATE_ERROR_BAD_ARGUMENT
    • -10 : Updater Error UPDATE_ERROR_NO_PARTITION
    • -11 : Updater Error UPDATE_ERROR_ACTIVATE
    • -12 : Updater Error UPDATE_ERROR_MAGIC_BYTE
    • -13 : Updater Error UPDATE_ERROR_MD5
    • -14 : Updater Error UPDATE_ERROR_STREAM
    • -15 : Updater Error UPDATE_ERROR_SIZE
    • -16 : Updater Error UPDATE_ERROR_SPACE
    • -17 : Updater Error UPDATE_ERROR_READ
    • -18 : Updater Error UPDATE_ERROR_ERASE
    • -19 : Updater Error UPDATE_ERROR_WRITE
  • TAR: applying -30 offset to forwarded error values from untar.h

    • 32 : Tar Error TAR_ERR_DATACB_FAIL
    • 33 : Tar Error TAR_ERR_HEADERCB_FAIL
    • 34 : Tar Error TAR_ERR_FOOTERCB_FAIL
    • 35 : Tar Error TAR_ERR_READBLOCK_FAIL
    • 36 : Tar Error TAR_ERR_HEADERTRANS_FAIL
    • 37 : Tar Error TAR_ERR_HEADERPARSE_FAIL
    • 38 : Tar Error TAR_ERROR_HEAP

Test Suite

Known bugs

  • tarGzStreamExpander hates SPIFFS
  • tarGzExpander/tarExpander: some formats aren't supported with SPIFFS (e.g contains symlinks or long filename/path)
  • tarGzExpander without intermediate file hates situations with low heap
  • tarGzExpander/gzExpander on ESP8266 : while the provided examples will work, the 32Kb dynamic allocation for gzip dictionary is unlikely to work in real world scenarios (e.g. with a webserver) and would probably require static allocation

- tarGzExpander: files smaller than 4K aren't processed

  • error detection isn't deferred efficiently, debugging may be painful
  • .tar files containing files smaller than 512 bytes aren't fully processed
  • reading/writing simultaneously on SPIFFS may induce errors

Debugging:

  • ESP32: use all of the "Debug level" values from the boards menu
  • ESP8266: Warning/Error when "Debug Port:Serial" is used, and Debug/Verbose when "Debug Level:Core" is selected from the boards menu

Resources

image

Alternate links

Credits:

More Repositories

1

M5Stack-SD-Updater

πŸ’Ύ Customizable menu system for M5Stack, M5Unified and ESP32-Chimera-Core - loads apps from the Micro SD card. Easily add you own apps
C++
310
star
2

ESP32-BLECollector

α›‘α›’ BLE Scanner + Data persistence on SD Card for M5Stack, M5Core2, Odroid-Go, ESP32-Wrover-Kit and other models
C
256
star
3

ESP32-USB-Soft-Host

An Arduino wrapper to @sdima1357's usb_soft_host esp-idf example
C
249
star
4

ESP32-Chimera-Core

ESP32-Chimera-Core πŸ‘Ύ is an arduino library for M5Stack, Odroid-Go, D-Duino-32-XS and other ESP32/TFT/SD bundles
C++
153
star
5

WUD-Ducky

An ESP32-S2 RubberDucky script parser, with Mouse/PenDrive support πŸ¦†
C
94
star
6

ESP32-BLEBeaconSpam

BLE Beacon spam on ESP32
C++
50
star
7

ESP32-PsRamFS

🐏 πŸ’Ύ RamDisk for ESP32-Arduino using PsRam and fs::FS
C
46
star
8

Rotatey_Cube

🧊 3D gyrocsopic rotating cube on Arduino using an MPU650 and SSD1306 OLED
C++
40
star
9

ESP32-3DPrinter-Bridge

A network <=> ESP32 <=> USB (FTDI) <=> 3D printer Bridge
C++
38
star
10

YAMLDuino

YAML <=> JSON converter for ESP32, ESP8266, RP2040 and possibly other devices
C
34
star
11

M5Tube

Video player for M5Stack with Docker conversion support
C++
30
star
12

ESP8266-Hobo-Clock

Self adjustable ESP8266 clock without RTC and NTP
Arduino
24
star
13

esp32-DCF77_Analyzer_Clock

⏰ An ESP32 TFT interpretation of the @deruiter's DCF77-Analyzer-Clock-V2.0
C
23
star
14

ESP8266SDUpdater

πŸ’Ύ ESP8266 Prequel to M5Stack-SD-Updater
C++
23
star
15

esp8266_deauther_serial

Serial version of spacehuhn/esp8266_deauther
C
22
star
16

WiFiChanViz

WiFi scanner with visual persistence, intended to find the idlest channel e.g. to assign to a ZigBee device
C
20
star
17

ESP32-ENC28J60

ENC28J60 Ethernet driver for ESP32-Arduino 2.0.5, lwip compliant
C
19
star
18

ESP32-Raytracer

An ESP32 study based on Dmitry V. Sokolov's tinyraytracer https://github.com/ssloy/tinyraytracer
C
19
star
19

Rotatey_Balls

🧊 3D animated gyro balls with ESP32, SSD1306 and MPU6050
Arduino
17
star
20

M5Stack-ESP12-Shield

ESP12 shield for M5Stack - Flash your ESP through the M5!
C
16
star
21

ImgurUploader

πŸ–ΌοΈ imgur.com πŸ“€ image/video uploader library for ESP32
C++
14
star
22

signal-logr

Wifi Location signal logger written in NodeJS for Raspberry Pi now with RTL-SDR support
JavaScript
14
star
23

ESP32-GifPlayer

GIF player Demo for M5Stack, Odroid-GO, ESP32-Wrover-Kit, LoLinD32-Pro, D-Duino32-XS, and more...
C++
13
star
24

ESP32-SIDView

SidWiz viewer for ESP32 [WORK IN PROGRESS]
C++
13
star
25

M5Rotatey_Cube

🧊 Modified M5Stack/MPU9250 gyro/accel demo sketch
C++
11
star
26

ESP32-Hector

M5Stack sprite+ULP-sound demo
C
10
star
27

ESP32-AmigaBoingBall

An interpretation of the famous Amiga Boing Ball Demo on ESP32 with the help of @bodmer's TFT_eSPI
C++
10
star
28

ESP32-Tesseract

4D 🧊 Hypercube animation demo for ESP32/TFT based on Space Dandy's Catherine
C
10
star
29

LGFXMeter

LGFX based Gauge Decoration and Animation library 🧭
C++
10
star
30

jqsheep

This jQuery plugin animates sheeps around page elements
JavaScript
10
star
31

M5Stack-Raytracer

An ESP32 study based on Dmitry V. Sokolov's tinyraytracer https://github.com/ssloy/tinyraytracer
C
9
star
32

TFT_GIF

Animated GIF on ST7735 with ESP8266
C
9
star
33

ESP32-MSGEQ7

πŸ”Š πŸ“ˆ MSGEQ7 based AudioSpectrum Visualizer for ESP32 and TFT
C++
8
star
34

M5Stack-Rickroll

Offline rickrolling for your M5Stack
C
8
star
35

ESP32-BLETimeServer

Time server module for ESP32-BLECollector
C++
7
star
36

M5Stack-NyanCat

A NyanCat for your M5Stack
C
6
star
37

YOURLS-GeoShortURL

πŸ”— YOURLS plugin to add per-country support to existing short URLs
PHP
6
star
38

kde-thumbnailer-qoi

KDE thumbnailer for QOI (Quite OK Image Format)
CMake
6
star
39

esp32-qemu-sim

Github action to run ESP32 compiled binaries in QEmu and collects output logs
Shell
6
star
40

tobozo.github.io

HTML
5
star
41

node-deauther

A HTTP to Serial bridge for esp8266_deauther
JavaScript
5
star
42

WiFiManagerTz

A NTP/Timezone extension to @tzapu's WiFiManager
C++
5
star
43

SerialServer

ESP8266WebServer to Serial syntactic substitute
C++
4
star
44

ESPWifiBench

ESP8266 / ESP32 Wifi bench
Arduino
4
star
45

ghetto_blaster

Rotary controls for OLED Ghetto Blaster
C
4
star
46

SpinningRat

Horizontally spinning rat πŸ€ for M5Unified
C
3
star
47

rpi-camsentry

Pan / Tilt controls + head tracking + video streaming over websocket from RaspiCam
JavaScript
3
star
48

M5Core2-SoundBouncer

M5Stack / M5Core2 demo animating 16 metronomes
C
3
star
49

ESP32-reSID

reSID is a Vice MOS6581/MOS8580 Sound Interface Device Emulator for ESP32
C
2
star
50

ESP32-Flocker

πŸ—οΈ App-suite builder for ESP32
Shell
1
star
51

php-tor-control-port

CSS
1
star
52

dotfiles

1
star