• Stars
    star
    142
  • Rank 252,420 (Top 6 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created almost 5 years ago
  • Updated 12 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Pure rust zip & zip64 reading and writing

rc-zip

test pipeline Coverage Status (codecov.io)

Motivation

Have a pure rust, highly compatible, I/O-model independent, zip reading and writing library.

Design decisions

This crate does not perform I/O directly. Instead, it uses a state machine, and asks for reads at specific offsets. This allows it to work under different I/O models: blocking, non-blocking, and async. It has no expectations of the zip archive being present on disk (ie. it doesn't assume std::fs), just that random access is possible.

This crate relies fully on the central directory, not on local headers:

[local file header 1] // <---------------- ignored
[file data 1]
[local file header 2]
[file data 2]
[central directory header 1] // <--------- used
[central directory header 2]
[end of central directory record]

The reason for that is that the central directory is the canonical list of entries in a zip. Archives that have been repacked may contain duplicate local file headers (and data), along with headers for entries that have been removed. Only the central directory is authoritative when it comes to the contents of a zip archive.

This crate accepts what is known as "trailing zips" - for example, files that are valid ELF or PE executables, and merely have a valid zip archive appended. This covers some forms of self-extracting archives and installers.

This crate recognizes and uses zip64 metadata. This allows for a large number of entries (above 65536) and large entries (above 4GiB). This crate attempts to forgives some non-standard behavior from common tools. Such behavior has been observed in the wild and is, whenever possible, tested.

This crate attempts to recognize as much metadata as possible, and normalize it. For example, MSDOS timestamps, NTFS timestamps, Extended timestamps and Unix timestamps are supported, and they're all converted to a chrono DateTime.

Although the normalized version of metadata (names, timestamps, UID, GID, etc.) is put front and center, this crate attempts to expose a "raw" version of that same metadata whenever the authors felt it was necessary.

Whenever the zip archive doesn't explicitly specify UTF-8 encoding, this crate relies on encoding detection to decide between CP-437 and Shift-JIS. It uses encoding_rs to deal with Shift-JIS.

Due to the history of the zip format, some compatibility issues are to be expected: for example, for archives with only MSDOS timestamps, the results might be in the wrong timezone. For archive with very few files and non-UTF8 names, the encoding might not be detected properly, and thus decoding may fail.

As much as possible, nom is used to parse the various data structures used in the zip archive format. This allows a semi-declarative style that is easier to write, read, and amend if needed. Some (hygienic) macros are used to avoid repetition.

API design

The design of the API is constrained by several parameters:

  • A compliant zip reader must first read the central directory, located near the end of the zip archive. This means simply taking an Read won't do.
  • Multiple I/O models must be supported. Whereas other crates focus on taking a Read, a Read + Seek, or simply a byte slice, this crate aims to support synchronous and asynchronous I/O.

As a result, the structs in this crate are state machines, that advertise their need to read (and from where), to process data, or to write. As a result, I/O errors are cleanly separated from the rest, and calls to this crate never block.

See the inline rustdoc comments for more details on API design.

License

This project is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE and LICENSE-MIT for details.

More Repositories

1

jsmad

🎷 Javascript MPEG-1 Audio Layer III (mp3) and ID3v2 decoder
JavaScript
763
star
2

mevi

A memory visualizer in Rust (ptrace + userfaultfd)
Rust
576
star
3

jooc-legacy

🐊 The historical, initial implementation of an ooc compiler in Java
Java
115
star
4

axact

A resource monitor in your browser, in Rust
Rust
96
star
5

isaac-paper

πŸƒ A fan remake of The Binding of Isaac - with a paper look
ooc
65
star
6

pegviz

PEG trace visualizer
Rust
58
star
7

rue

A bad version of strace in Rust
Rust
44
star
8

shin

⚠️ (def shin (dissoc clojurescript :jvm :google_closure)) (deprecated)
Ruby
35
star
9

homebrew-mingw

⚠️ brew for Windows (legacy)
Ruby
33
star
10

oc

⚠️ An experimental, next-generation, coroutine-based, modular compiler for a subset of ooc. (deprecated)
ooc
21
star
11

stringsearch

Collection of techniques for (binary) string searching
Rust
18
star
12

lavish-compiler

Compiler for the Lavish IDL
Rust
18
star
13

refresh

Silly Fast Fresh Deploys with Rust
Rust
18
star
14

teeworlds-ai

An attempt to add bots to teeworlds-0.5.1 =)
C
16
star
15

hello-sdl2-ios6

C, SDL2, iOS 6.0, cmake = Great success!
C
15
star
16

ooc-gtk

GTK2 bindings for ooc
ooc
15
star
17

stop-optimizing-me

How NOT to optimize something
Go
15
star
18

feedback

An issue tracker for articles/series/videos at https://fasterthanli.me/
13
star
19

yajit

A small JIT-compiler written in ooc, for ooc (providing closures and partials)
ooc
12
star
20

fasterthanlime

My GitHub profile page
12
star
21

inception-engine

flexible game engine in ooc, with an ingame console to adjust everything in realtime
ooc
12
star
22

read-raw-ext4

Code sample for "Reading files the hard way Part 3"
Rust
11
star
23

ziptest-rs

Wasm + Rust test
Rust
11
star
24

git-mirror

πŸ”œ Push to a git remote when receiving a github webhook
Go
10
star
25

dotfiles

GitHub Codespaces made me do this
Shell
10
star
26

more-jpeg

Mooooooore JPEG.
Rust
10
star
27

ooc-docs

Official documentation for ooc 0.3+
Python
10
star
28

stream-example

Quick example of axum + async-stream
Rust
10
star
29

rc-deflate

Pure rust deflate (& related formats) reading
Rust
9
star
30

fallacious

Torrent client daemon + user-friendly web application to facilitate content search, download and streaming
ooc
9
star
31

netherquote

Repo to profile pathological compile times with knuffel / chumsky
Rust
8
star
32

ooc.vim

ooc highlighting for VIM
Vim Script
8
star
33

hring-tls

Playing around with hring & kTLS
Rust
8
star
34

scissors

Running ooc with scissors
ooc
8
star
35

small-str-bench

Small string benchmark
Rust
7
star
36

mincore

FFI advice
Rust
7
star
37

wrt

πŸ¦€ WinRT/Rust experiments
Rust
7
star
38

bootcamp

ooc compatibility kit - test suite and tools to verify the compliance of an ooc implementation
ooc
7
star
39

ooc-zeromq

ØMQ bindings for ooc
ooc
7
star
40

cork

Another try at an ooc compiler
C
6
star
41

scb2obj

Converter for League of Legends' SCB (binary scene objects) to Wavefront OBJ
C
6
star
42

rustc-superlinear-sadness

Trying to reproduce build time issues with tower
Rust
6
star
43

whatever

Rust / C / Cgo example
Rust
6
star
44

supertasball

Super TASball, the new and improved version of the Ludum Dare 41 entry TASball!
HTML
6
star
45

telegram-bot-tide

Demo of `telegram-bot` and `tide` crates used together
Rust
6
star
46

ooczmq

A czmq wrapper for ooc, allowing ØMQ 2.0 and 3.0 compatibility
ooc
5
star
47

decadence

And thus, all llamaistas went their own ways, creating their own languages, leaving the mothership which they loved and lived for.
ooc
5
star
48

ooc-coroutine

Coroutines for ooc! using Steve Dekorte's libcoroutine
ooc
5
star
49

peg-leg

[deprecated, see greg] A fork of Ian Piumarta's peg/leg with adjustable YY_STACK_SIZE
5
star
50

morning

Morning language
Rust
5
star
51

otto

Oh hi there! What a nice day to BURN FUCKING AUTOTOOLS TO THE GROUND
Go
5
star
52

bamb

2-player reverse sudoku
TypeScript
5
star
53

epfl-wifi

Everything you need to connect to the Wifi at EPFL reliably
5
star
54

toml-edit-js

Just trying formatting-preserving TOML edits in javascript
JavaScript
5
star
55

ooc-duktape

Duktape bindings for ooc
C
5
star
56

2manydistros

Operating System for web developers, with sensible defaults. Arch-based.
Shell
5
star
57

ooc-curl

CURL bindings for ooc
ooc
4
star
58

bleep

Sound library for ooc
ooc
4
star
59

alone

A Ludum Dare #22 entry in ooc
ooc
4
star
60

zealot

A card game for Ludum Dare 40
TypeScript
4
star
61

ooci

experimental ooc repl
ooc
4
star
62

manual-future

Fun with futures and pin!
Rust
4
star
63

astronvim_config

My astronvim user config, I guess
Lua
4
star
64

ld42

Ludum Dare 42 entry
Lua
3
star
65

ubio

ubiquitous I/O - Implements ara traits for HTTP resources and local files
Rust
3
star
66

slow-max-by-key

Showcasing Iterator::max_by_key being slower than a manual implementation
Rust
3
star
67

pablo

crossbeam scoped threads / rayon iterators example
Rust
3
star
68

alarmoo

A cool alarm clock written in ooc, which launches mplayer with your favorite music on a given hour
ooc
3
star
69

libmad

My libmad debugging version
C
3
star
70

msgpack-play

Playing around with msgpack
Rust
3
star
71

meta

The upcoming lexing/parsing/macro-ing system for rock, the ooc compiler in ooc.
3
star
72

sdk

🚨 DEPRECATED 🚨 An SDK written from scratch? Possibly?
ooc
3
star
73

ooc-gdk

GDK bindings for ooc
ooc
3
star
74

jsw

Client-side Wiki, jQuery/Sinatra, generates static content on the server
JavaScript
3
star
75

thebootstrap

Hi! The world is yours.
Ruby
3
star
76

johnqadamant

The 10 seconds to John Q in the 1824 election are back - and they're pissed.
ooc
3
star
77

swiff

Given an HTTP(S) URL, return width and height for an .swf file
Go
3
star
78

tasball

Tool-assisted speed pinball
HTML
3
star
79

pug

My new language project, I guess.
C
3
star
80

crate-universe-sample

Using bazel to build some rust
Starlark
3
star
81

agora2

Modern forum software written on top of Sammy.js + node.js + mongodb
JavaScript
3
star
82

ooc-uv

ooc bindings for libuv
ooc
3
star
83

surviving

Surviving Rust async interfaces
Rust
3
star
84

the-scary-slope

TypeScript
3
star
85

herge

A parser generator
ooc
3
star
86

github-language-rankings

Ranks the languages by popularity on GitHub
Python
3
star
87

oocarray

ooc arrays tests
ooc
2
star
88

ara

Async random access (Rust traits)
Rust
2
star
89

dangerzone

A reddit super fill up clone!
ooc
2
star
90

ultipoly

7DRTS entry (MiniLD #44) - real-time monopoly on steroids
ooc
2
star
91

Marmot.Linux

An installation and update framework for Linux desktop apps
2
star
92

hopemage

Uses nagaqueen to generate ooc docs
C
2
star
93

pkgcost

Compute the cost of a go package in LOCs (including deduplicated dependencies)
Go
2
star
94

openvolt-notes

Notes from early discussions
2
star
95

ooc-ti

allows ooc programs to run on TI89/TI92+/TIv200 graphic calculators, using tigcc
2
star
96

deaddrop

Keep out ( <3 )
2
star
97

coke

Build tool for ooc
2
star
98

cs322

CS-323 project
Ruby
2
star
99

filexp

File experiments in Go
Go
2
star
100

lavish-rs

Lavish RPC Runtime (rust)
Rust
2
star