Awesome Open Source
Awesome Open Source

🗜️ ESP32-targz

An ESP32/ESP8266 Arduino library to provide decompression support for .tar, .gz and .tar.gz files

arduino-library-badge

ES32-targz logo

This library is a wrapper for the following two great libraries:

This library enables the channeling of gz ➡️ tar ➡️ filesystem data without using an intermediate file (bug: see #4).

In order to reach this goal, TinyUntar was heavily modified to allow data streaming, uzlib is also customized.

Tradeoffs

When the output is the filesystem (e.g. NOT when streaming to TAR), gzip can work without the dictionary. Disabling the dictionary can cause huge slowdowns but saves ~36KB of ram.

TinyUntar requires 512bytes only so its memory footprint is negligible.

Limitations

ESP32-TarGz can only have one output filesystem (see Support Matrix), and it must be set at compilation time (see Usage). This limitation does not apply to the input filesystem/stream.

Scope

  • This library is only for unpacking / decompressing, no compression support is provided whatsoever
  • Although the examples use SPIFFS as default, it should work with any fs::FS filesystem (SD, SD_MMC, FFat, LittleFS) and streams (HTTP, HTTPS, UDP, CAN, Ethernet)
  • This is experimental, expect bugs!
  • Contributions and feedback are more than welcome :-)

Support Matrix

fs::FS SPIFFS LittleFS SD SD_MMC FFAT
ESP32 1.0 1.0.5 1.0.5 1.0 1.0
ESP8266 builtin 0.1.0 0.1.0 n/a n/a

Usage

⚠️ Important note: setting the #define before including <ESP32-targz.h> is mandatory otherwise it will be ignored and the library will default to SPIFFS.

    // Set **destination** filesystem by uncommenting one of these:
    //#define DEST_FS_USES_SPIFFS
    //#define DEST_FS_USES_FFAT
    //#define DEST_FS_USES_SD
    //#define DEST_FS_USES_SD_MMC
    #define DEST_FS_USES_LITTLEFS
    #include <ESP32-targz.h>
    // filesystem object will be available as "tarGzFS"

Extract content from .gz file


    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    GzUnpacker *GZUnpacker = new GzUnpacker();

    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity

    // expand one file
    if( !GZUnpacker->gzExpander(tarGzFS, "/gz_example.gz", tarGzFS, "/gz_example.jpg") ) {
      Serial.printf("gzExpander failed with return code #%d", GZUnpacker->tarGzGetError() );
    }

    // expand another file
    if( ! gzExpander(tarGzFS, "/blah.gz", tarGzFS, "/blah.jpg") ) {
      Serial.printf("operation failed with return code #%d", GZUnpacker->tarGzGetError() );
    }


Expand contents from .tar file to /tmp folder


    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    TarUnpacker *TARUnpacker = new TarUnpacker();

    TARUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    if( !TARUnpacker->tarExpander(tarGzFS, "/tar_example.tar", tarGzFS, "/") ) {
      Serial.printf("tarExpander failed with return code #%d\n", TARUnpacker->tarGzGetError() );
    }


Expand contents from .tar.gz to /tmp folder


    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    // using an intermediate file (default is /tmp/tmp.tar)
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/targz_example.tar.gz", tarGzFS, "/tmp") ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }

    // or without intermediate file
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/targz_example.tar.gz", tarGzFS, "/tmp", nullptr ) ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }


Flash the ESP with contents from .gz file


    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    GzUnpacker *GZUnpacker = new GzUnpacker();

    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity

    if( ! GZUnpacker->gzUpdater(tarGzFS, firmwareFile, /*don't restart after update*/false ) ) {
      Serial.printf("gzUpdater failed with return code #%d\n", GZUnpacker->tarGzGetError() );
    }


ESP32 Only: Flash the ESP with contents from .gz stream


    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    fs::File file = tarGzFS.open( "/example_firmware.gz", "r" );

    if (!file) {
      Serial.println("Can't open file");
      return;
    }

    GzUnpacker *GZUnpacker = new GzUnpacker();

    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity

    if( !GZUnpacker->gzStreamUpdater( (Stream *)&file, UPDATE_SIZE_UNKNOWN ) ) {
      Serial.printf("gzStreamUpdater failed with return code #%d\n", GZUnpacker->tarGzGetError() );
    }


ESP32 Only: Direct expansion (no intermediate file) from .tar.gz. stream


    // mount spiffs (or any other filesystem)
    tarGzFS.begin();

    fs::File file = tarGzFS.open( "/example_archive.tgz", "r" );

    if (!file) {
      Serial.println("Can't open file");
      return;
    }

    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback ); // tar log verbosity

    if( !TARGZUnpacker->tarGzStreamExpander( (Stream *)&file, tarGzFS ) ) {
      Serial.printf("tarGzStreamExpander failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }


ESP32 Only: Direct Update (no intermediate file) from .tar.gz. stream


    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();

    TARGZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARGZUnpacker->setTarVerify( false ); // nothing to verify as we're writing a partition
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::targzNullProgressCallback ); // don't care about gz progress
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual partition
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARGZUnpacker->setTarMessageCallback( myTarMessageCallback/*BaseUnpacker::targzPrintLoggerCallback*/ ); // tar log verbosity

    // mount SD
    SD.begin();

    // this .tar.gz file has both the "app.ino.bin" and "app.spiffs.bin" partitions
    fs::File file = SD.open( "/bundle_firmware.tar.gz", "r" );

    if (!file) {
      Serial.println("Can't open file");
      return;
    }

    // this could also be a HTTP/HTTPS/UDP/Ethernet Stream
    Stream *streamptr = &file;

    if( !TARGZUnpacker->tarGzStreamUpdater( streamptr ) ) {
      Serial.printf("tarGzStreamUpdater failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    } else {
      Serial.println( "Flashing successful, now restarting" );
      ESP.restart();
    }


Callbacks



    // basic progress callback (valid for tar or gzip)
    void myBasicProgressCallback( uint8_t progress )
    {
      Serial.printf("Progress: %d\n", progress );
    }


    // complex progress callback (valid for tar or gzip)
    void myProgressCallback( uint8_t progress )
    {
      static int8_t myLastProgress = -1;
      if( myLastProgress != progress ) {
        if( myLastProgress == -1 ) {
          Serial.print("Progress: ");
        }
        myLastProgress = progress;
        switch( progress ) {
          case   0: Serial.print("0% ▓");  break;
          case  25: Serial.print(" 25% ");break;
          case  50: Serial.print(" 50% ");break;
          case  75: Serial.print(" 75% ");break;
          case 100: Serial.print("▓ 100%\n"); myLastProgress = -1; break;
          default: if( progress < 100) Serial.print( "▓" ); break;
        }
      }
    }


    // General Error/Warning/Info logger
    void myLogger(const char* format, ...)
    {
      va_list args;
      va_start(args, format);
      vprintf(format, args);
      va_end(args);
    }


    // status callback for TAR (fired at file creation)
    void myTarStatusProgressCallback( const char* name, size_t size, size_t total_unpacked )
    {
      Serial.printf("[TAR] %-64s %8d bytes - %8d Total bytes\n", name, size, total_unpacked );
    }



Return Codes

*Unpacker->tarGzGetError() returns a value when a problem occured:

  • General library error codes

    • 0 : Yay no error!
    • -1 : Filesystem error
    • -6 : Same a Filesystem error
    • -7 : Update not finished? Something went wrong
    • -38 : Logic error during deflating
    • -39 : Logic error during gzip read
    • -40 : Logic error during file creation
    • -100 : No space left on device
    • -101 : No space left on device
    • -102 : No space left on device
    • -103 : Not enough heap
    • -104 : Gzip dictionnary needs to be enabled
    • -105 : Gz Error when parsing header
    • -106 : Gz Error when allocating memory
    • -107 : General error, file integrity check fail
  • UZLIB: forwarding error values from uzlib.h as is (no offset)

    • -2 : Not a valid gzip file
    • -3 : Gz Error TINF_DATA_ERROR
    • -4 : Gz Error TINF_CHKSUM_ERROR
    • -5 : Gz Error TINF_DICT_ERROR
    • -41 : Gz error, can't guess filename
  • UPDATE: applying -20 offset to forwarded error values from Update.h

    • -8 : Updater Error UPDATE_ERROR_ABORT
    • -9 : Updater Error UPDATE_ERROR_BAD_ARGUMENT
    • -10 : Updater Error UPDATE_ERROR_NO_PARTITION
    • -11 : Updater Error UPDATE_ERROR_ACTIVATE
    • -12 : Updater Error UPDATE_ERROR_MAGIC_BYTE
    • -13 : Updater Error UPDATE_ERROR_MD5
    • -14 : Updater Error UPDATE_ERROR_STREAM
    • -15 : Updater Error UPDATE_ERROR_SIZE
    • -16 : Updater Error UPDATE_ERROR_SPACE
    • -17 : Updater Error UPDATE_ERROR_READ
    • -18 : Updater Error UPDATE_ERROR_ERASE
    • -19 : Updater Error UPDATE_ERROR_WRITE
  • TAR: applying -30 offset to forwarded error values from untar.h

    • 32 : Tar Error TAR_ERR_DATACB_FAIL
    • 33 : Tar Error TAR_ERR_HEADERCB_FAIL
    • 34 : Tar Error TAR_ERR_FOOTERCB_FAIL
    • 35 : Tar Error TAR_ERR_READBLOCK_FAIL
    • 36 : Tar Error TAR_ERR_HEADERTRANS_FAIL
    • 37 : Tar Error TAR_ERR_HEADERPARSE_FAIL
    • 38 : Tar Error TAR_ERROR_HEAP

Test Suite

Known bugs

  • tarGzStreamExpander hates SPIFFS
  • tarGzExpander/tarExpander: some formats aren't supported with SPIFFS (e.g contains symlinks or long filename/path)
  • tarGzExpander without intermediate file hates situations with low heap
  • tarGzExpander/gzExpander on ESP8266 : while the provided examples will work, the 32Kb dynamic allocation for gzip dictionary is unlikely to work in real world scenarios (e.g. with a webserver) and would probably require static allocation

- tarGzExpander: files smaller than 4K aren't processed

  • error detection isn't deferred efficiently, debugging may be painful
  • .tar files containing files smaller than 512 bytes aren't fully processed
  • reading/writing simultaneously on SPIFFS may induce errors

Debugging:

  • ESP32: use all of the "Debug level" values from the boards menu
  • ESP8266: Warning/Error when "Debug Port:Serial" is used, and Debug/Verbose when "Debug Level:Core" is selected from the boards menu

Resources

image

Alternate links

Credits:


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
C Plus Plus (371,093) 
Hacktoberfest (35,643) 
Arduino (21,584) 
Esp8266 (2,852) 
Esp32 (2,579) 
Arduino Library (1,399) 
Tar (224) 
Zlib (147) 
Related Projects
Advertising 📦 9
All Projects
Application Programming Interfaces 📦 120
Applications 📦 181
Artificial Intelligence 📦 72
Blockchain 📦 70
Build Tools 📦 111
Cloud Computing 📦 79
Code Quality 📦 28
Collaboration 📦 30
Command Line Interface 📦 48
Community 📦 81
Companies 📦 60
Compilers 📦 60
Computer Science 📦 74
Configuration Management 📦 39
Content Management 📦 167
Control Flow 📦 197
Data Formats 📦 77
Data Processing 📦 266
Data Storage 📦 132
Economics 📦 60
Frameworks 📦 198
Games 📦 122
Graphics 📦 103
Hardware 📦 148
Integrated Development Environments 📦 47
Learning Resources 📦 147
Legal 📦 28
Libraries 📦 119
Lists Of Projects 📦 21
Machine Learning 📦 336
Mapping 📦 61
Marketing 📦 15
Mathematics 📦 55
Media 📦 228
Messaging 📦 97
Networking 📦 304
Operating Systems 📦 84
Operations 📦 120
Package Managers 📦 52
Programming Languages 📦 229
Runtime Environments 📦 96
Science 📦 42
Security 📦 375
Social Media 📦 26
Software Architecture 📦 70
Software Development 📦 68
Software Performance 📦 57
Software Quality 📦 127
Text Editors 📦 45
Text Processing 📦 131
User Interface 📦 310
User Interface Components 📦 465
Version Control 📦 29
Virtualization 📦 68
Web Browsers 📦 38
Web Servers 📦 25
Web User Interface 📦 194