• Stars
    star
    793
  • Rank 57,419 (Top 2 %)
  • Language
    TypeScript
  • License
    MIT License
  • Created almost 10 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Fast, resilient, reproducible builds with npm install.

shrinkpack

Fast, resilient, reproducible builds with npm install.

NPM version NPM downloads Follow JamieMason on GitHub Follow fold_left on Twitter

What

shrinkpack points your package-lock.json at npm tarballs checked into your project's source control, so you can install while offline, during a registry outage, or during the next left-pad incident.

How

  1. Read package-lock.json or npm-shrinkwrap.json.
  2. Download the exact same .tgz files that npm install fetches from registry.npmjs.org.
  3. Decompress the .tgz files into .tar files. This avoids storing binary files in Git and removes the cost of decompression during npm install.
  4. Store the .tar files in your project at node_shrinkpack/*.tar.
  5. Rewrite package-lock.json to point at those instead of the registry.

Now your project can be installed while completely offline:

- npm install
+ npm ci --offline

The rest of the npm installation process is exactly the same. The only difference is that no network activity is necessary when installing and building your project. The node_shrinkpack directory can be ignored in your editor (much like is done with the node_modules directory), but is instead checked into source control.

Why

For context, please see the target problem and justification sections of this README.

Installation

Requires npm@7 or higher.

npm install --global shrinkpack

Usage

Run shrinkpack every time you have modified and installed your dependencies to produce a new package-lock.json.

Usage: shrinkpack [options] [directory]

Options:
  -V, --version  output the version number
  -h, --help     display help for command

Icons:
  + Added
  - Removed
  i Information
  12:34 Time Taken

History

  1. Feb 2015: shrinkpack was created.
  2. Mar 2016: The left-pad incident happened (shrinkpack users were unaffected!).
  3. Jun 2016: yarn added "offline mirror" support.
  4. May 2017: npm@5 added package-lock.json but npm@5 broke support for installing local files from a lockfile. Subsequent fixes were released for npm, but the issue was not resolved.
  5. Apr 2018: npm announced plans to integrate shrinkpack functionality into npm.
  6. May 2018: Work on shrinkpack is abandoned after the regression in npm@5 is still not fixed after a year.
  7. Dec 2021: Jack Franklin wrote why you should check-in your node dependencies and I'm reminded of why I wrote shrinkpack in the first place.
  8. Dec 2021: Work resumes on shrinkpack and support is added for npm >= 7.

Target Problem

Back in 2015 I was working at skysports.com. Each time we pushed code, our continuous integration environment created a clean workspace, installed, configured, and built the latest version of the project, then ran various tests and tasks.

We were happy with this process and the convenience of npm in particular, but the phase of our builds where npm install listed a huge amount of network traffic would always raise the same concerns:

  • This seems slow, wasteful, and inefficient.
  • We really depend on registry.npmjs.org, what do we do if it goes down?

The first suggestion was always to check in the node_modules directory, but the idea of large and chatty commits whenever we chose to upgrade or change a dependency put us off.

Other teams felt they could live with that and decided to proceed, only to find that packages such as phantomjs and node-sass will helpfully install the appropriate binary depending on which operating system you're running.

This meant that if Chris added phantomjs or node-sass to the project on his Mac and checked it into the repository, Helen wouldn't be able to use it on her Windows Machine.

The remaining alternatives were caching proxies or self-hosted registry mirrors, and caches-of-sorts. None of which appealed to us and, grudgingly, we continued as we were until later creating shrinkpack.

Justification

Note: This section was first written in 2015, before lockfiles were the default in npm, pnpm, and yarn. You had to opt-in to using a lockfile by running npm shrinkwrap to generate an npm-shrinkwrap.json file.

This text has been updated to reflect the situation today, where the need for lockfiles is more widely understood.

Whenever we add, remove, or update an npm dependency — we should test our application for regressions before locking down our dependencies with a lockfile. A tagged release should be a locked-down, frozen snapshot of the codebase which has been tested sufficiently enough that it is approved and trusted. When fed into a repeatable, automated deployment process it should always result in the same output.

  • Without a lockfile your dependency graph will mutate on a regular basis.
  • Checking in node_modules fixes this, but there are some issues which we discussed earlier.
  • You can be reasonably sure your dependency graph will remain consistent with a lockfile.
  • You can be completely sure with a lockfile and an offline cache.

A lockfile is something I would recommend you use anyway, even if you don't decide to use shrinkpack. It increases (but doesn't guarantee) certainty and confidence over exactly what versions of every nested dependency you've tested against and approved.

Without a lockfile and an offline cache, that's not guaranteed.

Consider this snippet from the package.json of a nested dependency in your project as an example. It's not even a package you directly control, it's a dependency of a dependency of a dependency:

"dependencies": {
  "lolwut": ">=0.1.0"
}

If [email protected] contains a regression and you're not using a lockfile, your project will contain that regression the next time you install it.

shrinkpack

With you hopefully convinced of the merits of lockfiles, shrinkpack will hopefully be seen as a small and complementary addition.

shrinkpack takes the tarballs of the specific dependency graph described by your lockfile and stores them within your project.

This means;

  • No need for repeated requests to registry.npmjs.org.
  • Each package/version pair can be checked in as a single tarball, avoiding commits with all kinds of noisy diffs.
  • Packages can be checked in, while still being installed by members of the team on different operating systems.

Suitability to your project

shrinkpack is best suited to a project which is the root consumer of dependencies and not a dependency itself. If your project is intended to be installed as a dependency of another project using npm install, let those downstream projects make their own decisions on bundling.

That said, if you're developing an npm package and want to use shrinkpack to speed up and harden your development and CI environments, adding package-lock.json and node_shrinkpack to your .npmignore file will allow you to do that, without publishing your shrinkpacked dependencies to the registry.

It's not recommended to publish a project with bundled or shrinkpacked dependencies to the registry, which would become bloated with duplicate copies of packages, bundled amongst various other ones.

Getting Help

More Repositories

1

ImageOptim-CLI

Make optimisation of images part of your automated build process
TypeScript
3,452
star
2

syncpack

Consistent dependency versions in large JavaScript Monorepos.
TypeScript
1,426
star
3

Jasmine-Matchers

Write Beautiful Specs with Custom Matchers for Jest and Jasmine
JavaScript
586
star
4

grunt-imageoptim

Make ImageOptim, ImageAlpha and JPEGmini part of your automated build process
JavaScript
479
star
5

expect-more

Curried Type Testing library, and Test Matchers for Jest
TypeScript
172
star
6

astexplorer.app

An https://astexplorer.net wrapper which adds module bundling and hot reloading.
TypeScript
120
star
7

karma-benchmark

A Karma plugin to run Benchmark.js over multiple browsers with CI compatible output.
TypeScript
90
star
8

codemods

A collection of transforms for use with JSCodeshift
JavaScript
59
star
9

blurhash-to-css

Convert BlurHashes to CSS Objects using TypeScript, Rust, and WebAssembly.
TypeScript
56
star
10

eslint-plugin-prefer-arrow-functions

Auto-fix plain Functions into Arrow Functions, in all cases where conversion would result in the same behaviour
TypeScript
52
star
11

karma-jasmine-matchers

A Karma plugin to inject Jasmine-Matchers for Jasmine and Jest.
JavaScript
46
star
12

nextjs-typescript-tailwind-critical-css

Next.js 9.3 with TypeScript, tailwindcss, and inlined Critical CSS.
TypeScript
41
star
13

eslint-formatter-git-log

ESLint Formatter featuring Git Author, Date, and Hash
TypeScript
41
star
14

ts-import-types-cli

Autofix TypeScript types to be imported using `import type`
TypeScript
40
star
15

self-help

Interactive Q&A Guides for Web and the Command Line.
TypeScript
37
star
16

image-optimisation-tools-comparison

A Benchmarking Suite for popular Image Optimisation Tools
TypeScript
28
star
17

Unreadable

An intelligent/CSS-aware HTML Minifier and Optimizer
JavaScript
26
star
18

custom-linkedin-cv

LinkedIn profile JSON + AngularJS == Custom Résumé
CSS
23
star
19

add-matchers

Write useful test matchers compatible with Jest and Jasmine.
JavaScript
18
star
20

syncpack-github-action

A GitHub Action to synchronise monorepo dependency versions with syncpack.
14
star
21

karma-nested-reporter

Easy to read test output with nested describe and it blocks
JavaScript
14
star
22

devtools-extension-tab-colours

Adds colours to the tabs in Chrome DevTools.
CSS
11
star
23

glob-bus

249 byte event emitter / pubsub with namespaced wildcards.
TypeScript
10
star
24

phantomjs-test-starter

A boilerplate / starter template for testing PhantomJS ‘Applications’ with Jasmine, Grunt and Istanbul
JavaScript
8
star
25

eslint-plugin-move-files

Move and rename files while keeping imports up to date
TypeScript
8
star
26

logservable

git log as an observable stream of JSON
TypeScript
8
star
27

subclass.js

Inherit, extend, decorate, and override classes and instances.
JavaScript
7
star
28

Jasmine-Matchers-Snippets

Snippets for Jest & Jasmine Matchers.
6
star
29

react-xstate-pull-to-refresh

Created with CodeSandbox
TypeScript
6
star
30

giterator

`git log` as a JavaScript generator
TypeScript
5
star
31

grunt-rewrite

A Grunt plugin to edit & replace file contents.
JavaScript
5
star
32

conventional-recommended-version

Determine the semantic version number of your project
TypeScript
5
star
33

csslib

A customised fork of inuitcss which I use on most projects.
CSS
5
star
34

node-es6-starter

babel, code climate, commitizen, coveralls, easystatic, ghooks, istanbul, jasmine, rollup, travis, xo.
JavaScript
4
star
35

peach.js

A precompiled forEach, unrolled for faster runtime performance.
JavaScript
4
star
36

doei

Experiment: Remove unused CSS/JS using RUM Code Coverage
TypeScript
4
star
37

Tim

Node.js, Express & MongoDB app starter template with multi-language support, user registration and authentication
JavaScript
4
star
38

jest-fail-on-console-reporter

Disallow untested console output produced during tests
TypeScript
2
star
39

treeify

Format an indented text file in the style of the Linux tree command
JavaScript
2
star
40

valueless

Replace values with paths, for use in test fixtures
JavaScript
2
star
41

lastfm-to-itunes

JavaScript
2
star
42

generator-jekyll-inuit

A Jekyll Website with Inuit.css generator for Yeoman
CSS
2
star
43

shrinkpack-examples

Examples of monorepos using shrinkpack to install while offline.
2
star
44

get-time-between

Measure the amount of time during work hours between two dates
TypeScript
2
star
45

is-office-hours

Determine whether a given date is within office hours
TypeScript
2
star
46

Jasmine-Matchers-ES6-Snippets

ES6 Snippets for Jest & Jasmine Matchers.
2
star
47

next-optimized-head

Next.js `<Head/>` which orders elements to improve (perceived) page performance.
TypeScript
2
star
48

commit-release

Commit and tag a conventional changelog release
TypeScript
1
star
49

front-end-engineer-interview-exercise

How do other Developers approach interview exercises?
1
star
50

github-actions-temporary-repo

JavaScript
1
star