• Stars
    star
    251
  • Rank 161,862 (Top 4 %)
  • Language
    Python
  • License
    MIT License
  • Created over 3 years ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Python function to stream unzip all the files in a ZIP archive on the fly

stream-unzip

PyPI version CircleCI Test Coverage

Python function to stream unzip all the files in a ZIP archive, without loading the entire ZIP file into memory or any of its uncompressed files.

To create ZIP files on the fly try stream-zip.

Features

In addition to being memory efficient, stream-unzip supports:

  • Deflate-compressed ZIPs. The is the historical standard for ZIP files.

  • Deflate64-compressed ZIPs. These are created by certain versions of Windows Explorer in some circumstances. Python's zipfile module cannot open Deflate64-compressed ZIPs.

  • Zip64 ZIP files. These are ZIP files that allow sizes far beyond the approximate 4GiB limit of the original ZIP format.

  • WinZip-style AES-encrypted ZIPs. Python's zipfile module cannot open AES-encrypted ZIPs.

  • Legacy-encrypted ZIP files. This is also known as ZipCrypto/Zip 2.0.

  • ZIP files created by Java's ZipOutputStream that are larger than 4GiB. At the time of writing libarchive-based stream readers cannot read these without error.

  • BZip2-compressed ZIPs.


Visit the stream-unzip documentation for usage instructions.

More Repositories

1

cypress-image-diff

Visual regression test with cypress
Handlebars
233
star
2

mbtiles-s3-server

Python server to on-the-fly extract and serve vector tiles from an mbtiles file on S3
Python
136
star
3

sqlite-s3vfs

Python writable virtual filesystem for SQLite on S3
Python
112
star
4

stream-zip

Python function to construct a ZIP archive on the fly
Python
86
star
5

mobius3

Continuously sync folder to S3, using inotify under the hood
Python
42
star
6

data-workspace-frontend

An open source data analysis platform with features for users with a range of technical skills
Python
42
star
7

fargatespawner

Spawns JupyterHub single user servers in Docker containers running in AWS Fargate
Python
38
star
8

pg-bulk-ingest

Python utility function to ingest data into a SQLAlchemy-defined PostgreSQL table
Python
34
star
9

dns-rewrite-proxy

A DNS proxy server that conditionally rewrites and filters A record requests
Python
28
star
10

stream-sqlite

Python function to extract rows from a SQLite file while iterating over its bytes
Python
23
star
11

tidy-json-to-csv

Convert JSON to a set of tidy CSV files
Python
20
star
12

tamato

The Tariff Management Tool (TaMaTo) stores and manages the tariffs and controls that are applied on imports and exports at the UK border. 🍅
Python
18
star
13

jupyters3

Jupyter Notebook Contents Manager for AWS S3
Python
17
star
14

dit-clamav-rest

A minimal ClamD REST interface
Python
15
star
15

great-cms

great.gov.uk CMS
Python
15
star
16

django-workflow-engine

Configurable Django workflow engine
Python
13
star
17

stream-read-xbrl

Python package to parse Companies House accounts data in a streaming way
Python
13
star
18

mock-sso

A mock SSO server to test an applications SSO integration
JavaScript
12
star
19

data-hub-api

Django API for Data Hub frontend
Python
12
star
20

data-hub-frontend

The frontend rendering application for Data Hub
JavaScript
10
star
21

streampq

Python PostgreSQL adapter to stream results of multi-statement queries without a server-side cursor
Python
8
star
22

export-opportunities

UK - Great.gov - Export Opportunities - Find and apply for overseas opportunities from businesses looking for products or services like yours.
HTML
7
star
23

pii-secret-check-hooks

PII (Personal Identifiable Information) and secret check hooks for pre-commit
Python
7
star
24

iterable-subprocess

Python context manager to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed
Python
7
star
25

market-access-python-frontend

Python
7
star
26

to-file-like-obj

Python utility function to convert an iterable of bytes or str to a readable file-like object
Python
7
star
27

directory-cms

Content Management System service for directory apps
Python
6
star
28

django-chunk-s3-av-upload-handlers

Chunking Django file handlers for S3 and ClamAV service uploads
Python
6
star
29

platform-tools

Tooling to assist with setting up and managing AWS copilot infrastructure
Python
6
star
30

tariff-data-manual

Technical documentation about how the UK tariff data works.
Ruby
6
star
31

s3chunkuploader

A multi threaded S3 file chunk uploader which bypasses local file system and pipes the file directly to S3
Python
6
star
32

trade-elements

Front end pattern library for Department of International Trade
HTML
5
star
33

streamlit-gov-uk-components

A collection of Streamlit components that use or are inspired by the GOV.UK Design System
Shell
5
star
34

data-hub-components

Collection of React Components used by DIT's Data Hub.
JavaScript
5
star
35

digital-workspace-v2

DBT's intranet Django/Wagtail
Python
5
star
36

django-log-formatter-ecs

An ECS formatter for Django logs
Python
5
star
37

jwt-postgresql-proxy

Stateless JWT authentication in front of PostgreSQL
Python
5
star
38

trade-tariff-api

Taric file API for Trade Tariff Management application
Python
5
star
39

data-engineering

A collection of information and learning material relating to data engineering at the Department for International Trade
5
star
40

data-hub-prototype

JavaScript
4
star
41

pg-force-execute

Context manager to run PostgreSQL queries with SQLAlchemy, terminating any other clients that block it
Python
4
star
42

terraform-module-eks-base

HCL
4
star
43

directory-tests

Smoke, integration and functional tests for DIT's services
Python
4
star
44

trade-tariff-management

Ruby
4
star
45

countries-of-interest-service

Lightweight API service for querying for companies that have expressed interest in exporting to specific countries
Python
4
star
46

icms

Import Case Management System - Python/Django port on PostgreSQL database
Python
4
star
47

govuk-frontend-django

GOV.UK Frontend Django
Python
4
star
48

directory-ui-buyer

Python
3
star
49

help-desk-service

Python
3
star
50

terraform-prometheus-module

HCL
3
star
51

terraform-modules

HCL
3
star
52

data-hub-api-spec

Swagger files and test server for the Data Hub CRM project
Python
3
star
53

great-international-ui

Python
3
star
54

stream-write-ods

Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk
Python
3
star
55

directory-api

Directory of Exporters API
Python
3
star
56

dnb-service

A Dunn & Bradstreet microservice
Python
3
star
57

export-support

Python
3
star
58

data-workspace-tools

HTML
3
star
59

dit-thumber

A simple django app for soliciting user feedback on django views
Python
3
star
60

lite-api

Service for handling backend calls in LITE.
Python
3
star
61

infrastructure

Vagrant, Docker and Ansible configuration repository for DIT Digital
Shell
3
star
62

clamav-s3-mirror

Python
3
star
63

legal-basis-api

Legal Basis for Consent Service API Server
Python
3
star
64

dit-helpdesk

HTML
3
star
65

django-zenslackchat

A bot which synchronises support requests from Slack to Zendesk and back.
Python
3
star
66

docker-overlay-network-benchmark

Shell
3
star
67

data-hub-helpcentre

Central point for articles, advice and how-tos for the Data Hub
CSS
3
star
68

market-access-api

The API for Market Access
Python
3
star
69

selling-online-overseas

Department of International Trade marketplace navigator.
Python
3
star
70

flower-runner

Celery Flower runner for GOV.UK PaaS
Python
3
star
71

django-staff-sso-client

A client for staff-sso
Python
3
star
72

vulnerability-priority-list

A command line report on a GitHub organisation's repositories, ordered by priority, and including time-to-SLA for each severity level
Python
3
star
73

global-uk-tariff

GOV.UK service displaying tariffs to trade with the UK from 1 January 2021
CSS
3
star
74

ci-pipeline

DIT CD pipeline
Ruby
2
star
75

investment-opps

JavaScript
2
star
76

aioftps3

FTP in front of AWS S3, powered by asyncio and aiohttp
Python
2
star
77

django-ga-measurement-protocol

Python
2
star
78

copilot-poc

Co Pilot PoC
Python
2
star
79

public-data-api

The source for the Department for International Trade's Public Data API
HTML
2
star
80

directory-form-data

Python
2
star
81

company-matching-service

Python
2
star
82

dit-contact-forms

DIT Contact Forms
HTML
2
star
83

great-styles

Styles for great.gov.uk
SCSS
2
star
84

markdown-gds

Python
2
star
85

directory-ui-supplier

deprecated. Incorporated into https://github.com/uktrade/great-international-ui
Python
2
star
86

data-workspace-visualisation

Template repository for Data Workspace visualisations
R
2
star
87

terraform-module-openshift

HCL
2
star
88

mirror-git-to-s3

Python functions and CLI to mirror git repositories to S3
Python
2
star
89

directory-sso

Python
2
star
90

prototypes-on-paas

HTML
2
star
91

invest-pir-api

Personalised Investment Report
Python
2
star
92

terraform-module-aws_account

HCL
2
star
93

directory-companies-house-search

Python
2
star
94

iigb-beta-content

Content for IIGB website
2
star
95

great-pattern-library

Great Pattern library
CSS
2
star
96

gds-technical-documentation-template

mkdocs template using GOV.UK Design System styles
JavaScript
2
star
97

statement-of-works

Automatic generation for SOW forms
Python
2
star
98

dit-ip

Python
2
star
99

lite-internal-frontend

Application for handling internal information in LITE.
Python
2
star
100

dit-classification-matcher

A microservice developed in Python + Flask to map between CPV <-> HS <-> TARIC codes
Python
2
star