• Stars
    star
    190
  • Rank 203,739 (Top 5 %)
  • Language
    JavaScript
  • License
    Other
  • Created over 5 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

making fetch happen for npm

make-fetch-happen

npm version license Travis Coverage Status

make-fetch-happen is a Node.js library that wraps minipass-fetch with additional features minipass-fetch doesn't intend to include, including HTTP Cache support, request pooling, proxies, retries, and more!

Install

$ npm install --save make-fetch-happen

Table of Contents

Example

const fetch = require('make-fetch-happen').defaults({
  cachePath: './my-cache' // path where cache will be written (and read)
})

fetch('https://registry.npmjs.org/make-fetch-happen').then(res => {
  return res.json() // download the body as JSON
}).then(body => {
  console.log(`got ${body.name} from web`)
  return fetch('https://registry.npmjs.org/make-fetch-happen', {
    cache: 'no-cache' // forces a conditional request
  })
}).then(res => {
  console.log(res.status) // 304! cache validated!
  return res.json().then(body => {
    console.log(`got ${body.name} from cache`)
  })
})

Features

  • Builds around minipass-fetch for the core fetch API implementation
  • Request pooling out of the box
  • Quite fast, really
  • Automatic HTTP-semantics-aware request retries
  • Cache-fallback automatic "offline mode"
  • Proxy support (http, https, socks, socks4, socks5)
  • Built-in request caching following full HTTP caching rules (Cache-Control, ETag, 304s, cache fallback on error, etc).
  • Node.js Stream support
  • Transparent gzip and deflate support
  • Subresource Integrity support
  • Literally punches nazis
  • Built in DNS cache
  • (PENDING) Range request caching and resuming

Contributing

The make-fetch-happen team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The Contributor Guide outlines the process for community interaction and contribution. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.

All participants and maintainers in this project are expected to follow the npm Code of Conduct, and just generally be excellent to each other.

Please refer to the Changelog for project history details, too.

Happy hacking!

API

> fetch(uriOrRequest, [opts]) -> Promise<Response>

This function implements most of the fetch API: given a uri string or a Request instance, it will fire off an http request and return a Promise containing the relevant response.

If opts is provided, the minipass-fetch-specific options will be passed to that library. There are also additional options specific to make-fetch-happen that add various features, such as HTTP caching, integrity verification, proxy support, and more.

Example
fetch('https://google.com').then(res => res.buffer())

> fetch.defaults([defaultUrl], [defaultOpts])

Returns a new fetch function that will call make-fetch-happen using defaultUrl and defaultOpts as default values to any calls.

A defaulted fetch will also have a .defaults() method, so they can be chained.

Example
const fetch = require('make-fetch-happen').defaults({
  cachePath: './my-local-cache'
})

fetch('https://registry.npmjs.org/make-fetch-happen') // will always use the cache

> minipass-fetch options

The following options for minipass-fetch are used as-is:

  • method
  • body
  • redirect
  • follow
  • timeout
  • compress
  • size

These other options are modified or augmented by make-fetch-happen:

  • headers - Default User-Agent set to make-fetch happen. Connection is set to keep-alive or close automatically depending on opts.agent.
  • agent
    • If agent is null, an http or https Agent will be automatically used. By default, these will be http.globalAgent and https.globalAgent.
    • If opts.proxy is provided and opts.agent is null, the agent will be set to an appropriate proxy-handling agent.
    • If opts.agent is an object, it will be used as the request-pooling agent argument for this request.
    • If opts.agent is false, it will be passed as-is to the underlying request library. This causes a new Agent to be spawned for every request.

For more details, see the documentation for minipass-fetch itself.

> make-fetch-happen options

make-fetch-happen augments the minipass-fetch API with additional features available through extra options. The following extra options are available:

> opts.cachePath

A string Path to be used as the cache root for cacache.

NOTE: Requests will not be cached unless their response bodies are consumed. You will need to use one of the res.json(), res.buffer(), etc methods on the response, or drain the res.body stream, in order for it to be written.

The default cache manager also adds the following headers to cached responses:

  • X-Local-Cache: Path to the cache the content was found in
  • X-Local-Cache-Key: Unique cache entry key for this response
  • X-Local-Cache-Mode: Always stream to indicate how the response was read from cacache
  • X-Local-Cache-Hash: Specific integrity hash for the cached entry
  • X-Local-Cache-Status: One of miss, hit, stale, revalidated, updated, or skip to signal how the response was created
  • X-Local-Cache-Time: UTCString of the cache insertion time for the entry

Using cacache, a call like this may be used to manually fetch the cached entry:

const h = response.headers
cacache.get(h.get('x-local-cache'), h.get('x-local-cache-key'))

// grab content only, directly:
cacache.get.byDigest(h.get('x-local-cache'), h.get('x-local-cache-hash'))
Example
fetch('https://registry.npmjs.org/make-fetch-happen', {
  cachePath: './my-local-cache'
}) // -> 200-level response will be written to disk

> opts.cache

This option follows the standard fetch API cache option. This option will do nothing if opts.cachePath is null. The following values are accepted (as strings):

  • default - Fetch will inspect the HTTP cache on the way to the network. If there is a fresh response it will be used. If there is a stale response a conditional request will be created, and a normal request otherwise. It then updates the HTTP cache with the response. If the revalidation request fails (for example, on a 500 or if you're offline), the stale response will be returned.
  • no-store - Fetch behaves as if there is no HTTP cache at all.
  • reload - Fetch behaves as if there is no HTTP cache on the way to the network. Ergo, it creates a normal request and updates the HTTP cache with the response.
  • no-cache - Fetch creates a conditional request if there is a response in the HTTP cache and a normal request otherwise. It then updates the HTTP cache with the response.
  • force-cache - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it creates a normal request and updates the HTTP cache with the response.
  • only-if-cached - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it returns a network error. (Can only be used when request’s mode is "same-origin". Any cached redirects will be followed assuming request’s redirect mode is "follow" and the redirects do not violate request’s mode.)

(Note: option descriptions are taken from https://fetch.spec.whatwg.org/#http-network-or-cache-fetch)

Example
const fetch = require('make-fetch-happen').defaults({
  cachePath: './my-cache'
})

// Will error with ENOTCACHED if we haven't already cached this url
fetch('https://registry.npmjs.org/make-fetch-happen', {
  cache: 'only-if-cached'
})

// Will refresh any local content and cache the new response
fetch('https://registry.npmjs.org/make-fetch-happen', {
  cache: 'reload'
})

// Will use any local data, even if stale. Otherwise, will hit network.
fetch('https://registry.npmjs.org/make-fetch-happen', {
  cache: 'force-cache'
})

> opts.cacheAdditionalHeaders

The following headers are always stored in the cache when present:

  • cache-control
  • content-encoding
  • content-language
  • content-type
  • date
  • etag
  • expires
  • last-modified
  • link
  • location
  • pragma
  • vary

This option allows a user to store additional custom headers in the cache.

Example
fetch('https://registry.npmjs.org/make-fetch-happen', {
  cacheAdditionalHeaders: ['my-custom-header'],
})

> opts.proxy

A string or new url.URL()-d URI to proxy through. Different Proxy handlers will be used depending on the proxy's protocol.

Additionally, process.env.HTTP_PROXY, process.env.HTTPS_PROXY, and process.env.PROXY are used if present and no opts.proxy value is provided.

(Pending) process.env.NO_PROXY may also be configured to skip proxying requests for all, or specific domains.

Example
fetch('https://registry.npmjs.org/make-fetch-happen', {
  proxy: 'https://corporate.yourcompany.proxy:4445'
})

fetch('https://registry.npmjs.org/make-fetch-happen', {
  proxy: {
    protocol: 'https:',
    hostname: 'corporate.yourcompany.proxy',
    port: 4445
  }
})

> opts.noProxy

If present, should be a comma-separated string or an array of domain extensions that a proxy should not be used for.

This option may also be provided through process.env.NO_PROXY.

> opts.ca, opts.cert, opts.key, opts.strictSSL

These values are passed in directly to the HTTPS agent and will be used for both proxied and unproxied outgoing HTTPS requests. They mostly correspond to the same options the https module accepts, which will be themselves passed to tls.connect(). opts.strictSSL corresponds to rejectUnauthorized.

> opts.localAddress

Passed directly to http and https request calls. Determines the local address to bind to.

> opts.maxSockets

Default: 15

Maximum number of active concurrent sockets to use for the underlying Http/Https/Proxy agents. This setting applies once per spawned agent.

15 is probably a pretty good value for most use-cases, and balances speed with, uh, not knocking out people's routers. πŸ€“

> opts.retry

An object that can be used to tune request retry settings. Retries will only be attempted on the following conditions:

  • Request method is NOT POST AND
  • Request status is one of: 408, 420, 429, or any status in the 500-range. OR
  • Request errored with ECONNRESET, ECONNREFUSED, EADDRINUSE, ETIMEDOUT, or the fetch error request-timeout.

The following are worth noting as explicitly not retried:

  • getaddrinfo ENOTFOUND and will be assumed to be either an unreachable domain or the user will be assumed offline. If a response is cached, it will be returned immediately.

If opts.retry is false, it is equivalent to {retries: 0}

If opts.retry is a number, it is equivalent to {retries: num}

The following retry options are available if you want more control over it:

  • retries
  • factor
  • minTimeout
  • maxTimeout
  • randomize

For details on what each of these do, refer to the retry documentation.

Example
fetch('https://flaky.site.com', {
  retry: {
    retries: 10,
    randomize: true
  }
})

fetch('http://reliable.site.com', {
  retry: false
})

fetch('http://one-more.site.com', {
  retry: 3
})

> opts.onRetry

A function called with the response or error which caused the retry whenever one is attempted.

Example
fetch('https://flaky.site.com', {
  onRetry(cause) {
    console.log('we will retry because of', cause)
  }
})

> opts.integrity

Matches the response body against the given Subresource Integrity metadata. If verification fails, the request will fail with an EINTEGRITY error.

integrity may either be a string or an ssri Integrity-like.

Example
fetch('https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
  integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // -> ok

fetch('https://malicious-registry.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
  integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // Error: EINTEGRITY

> opts.dns

An object that provides options for the built-in DNS cache. The following options are available:

Note: Due to limitations in the current proxy agent implementation, users of proxies will not benefit from the DNS cache.

  • ttl: Milliseconds to keep cached DNS responses for. Defaults to 5 * 60 * 1000 (5 minutes)
  • lookup: A custom lookup function, see dns.lookup() for implementation details. Defaults to require('dns').lookup.

More Repositories

1

npm

This repository is moving to: https://github.com/npm/cli
17,473
star
2

cli

the package manager for JavaScript
JavaScript
8,303
star
3

node-semver

The semver parser for node (the one npm uses)
JavaScript
4,772
star
4

npm-expansions

Send us a pull request by editing expansions.txt
JavaScript
2,209
star
5

tink

a dependency unwinder for javascript
JavaScript
2,153
star
6

ini

An ini parser/serializer in JavaScript
JavaScript
733
star
7

npx

npm package executor
JavaScript
731
star
8

rfcs

Public change requests/proposals & ideation
JavaScript
726
star
9

npm-registry-couchapp

couchapp bits of registry.npmjs.org
JavaScript
615
star
10

nopt

Node/npm Option Parsing
JavaScript
527
star
11

registry

npm registry documentation
449
star
12

npmlog

The logger that npm uses
JavaScript
423
star
13

marky-markdown

npm's markdown parser
JavaScript
405
star
14

arborist

npm's tree doctor
JavaScript
369
star
15

documentation

Documentation for the npm registry, website, and command-line interface.
MDX
369
star
16

pacote

npm fetcher
JavaScript
349
star
17

download-counts

Background jobs and a minimal service for collecting and delivering download counts
JavaScript
328
star
18

gauge

A terminal based horizontal guage aka, a progress bar
JavaScript
319
star
19

node-which

Like which(1) unix command. Find the first instance of an executable in the PATH.
JavaScript
305
star
20

init-package-json

A node module to get your node module started
JavaScript
284
star
21

validate-npm-package-name

Is the given string an acceptable npm package name?
JavaScript
282
star
22

cacache

npm's content-addressable cache
JavaScript
274
star
23

npm-merge-driver

git merge driver for resolving conflicts in npm-related files
JavaScript
271
star
24

npm-registry-client

JavaScript
264
star
25

lockfile

A very polite lock file utility, which endeavors to not litter, and to wait patiently for others.
JavaScript
259
star
26

registry-issue-archive

An archive of the old npm registry issue tracker
250
star
27

write-file-atomic

Write files in an atomic fashion w/configurable ownership
JavaScript
217
star
28

read-package-json

The thing npm uses to read package.json files with semantics and defaults and validation and stuff
JavaScript
214
star
29

roadmap

Public roadmap for npm
213
star
30

hosted-git-info

Provides metadata and conversions from repository urls for Github, Bitbucket and Gitlab
JavaScript
206
star
31

fstream

Advanced FS Streaming for Node
JavaScript
205
star
32

read

read(1) for node.
JavaScript
187
star
33

normalize-package-data

normalizes package metadata, typically found in package.json file.
JavaScript
184
star
34

ndm

ndm allows you to deploy OS-specific service-wrappers directly from npm-packages.
JavaScript
181
star
35

are-we-there-yet

Track complex hiearchies of asynchronous task completion statuses.
JavaScript
173
star
36

abbrev-js

Like ruby's Abbrev module
JavaScript
158
star
37

statusboard

Public monitor/status/health board for @npm/cli-team's maintained projects
JavaScript
152
star
38

security-holder

An npm package that holds a spot.
145
star
39

osenv

Look up environment settings specific to different operating systems.
JavaScript
137
star
40

feedback

Public feedback discussions for npm
133
star
41

npm-registry-fetch

like fetch() but for the npm registry
JavaScript
130
star
42

npm-package-arg

Parse the things that can be arguments to `npm install`
JavaScript
116
star
43

libnpm

programmatic npm API
JavaScript
113
star
44

npm-collection-staff-picks

JavaScript
112
star
45

npm-packlist

Walk through a folder and figure out what goes in an npm package
JavaScript
110
star
46

promzard

A prompting json thingie
JavaScript
101
star
47

npm-remote-ls

Examine a package's dependency graph before you install it
JavaScript
89
star
48

npmconf

npm config thing
JavaScript
75
star
49

cmd-shim

The cmd-shim used in npm
JavaScript
75
star
50

npm-tips

A collection of short (5 words or so) tips and tricks that can be sprinkled about the npm site.
JavaScript
73
star
51

www

community space for the npm website
68
star
52

policies

Privacy policy, code of conduct, license, and other npm legal stuff
Shell
67
star
53

package-json

Programmatic API to update package.json
JavaScript
65
star
54

git

a util for spawning git from npm CLI contexts
JavaScript
62
star
55

npm_conf

A conference about npm, maybe. Not to be confused with npmconf.
59
star
56

ignore-walk

Nested/recursive `.gitignore`/`.npmignore` parsing and filtering.
JavaScript
57
star
57

registry-follower-tutorial

write you a registry follower for great good
JavaScript
56
star
58

ssri

subresource integrity for npm
JavaScript
55
star
59

minipass-fetch

An implementation of window.fetch in Node.js using Minipass streams
JavaScript
54
star
60

ci-detect

Detect what kind of CI environment the program is in
JavaScript
53
star
61

run-script

Run a lifecycle script for a package (descendant of npm-lifecycle)
JavaScript
53
star
62

read-installed

Read all the installed packages in a folder, and return a tree structure with all the data.
JavaScript
52
star
63

mute-stream

Bytes go in, but they don't come out (when muted).
JavaScript
49
star
64

fs-write-stream-atomic

Like `fs.createWriteStream(...)`, but atomic.
JavaScript
48
star
65

read-package-json-fast

Like read-package-json, but faster
JavaScript
48
star
66

libnpmpublish

programmatically publish and unpublish npm packages
JavaScript
46
star
67

logical-tree

Calculates a nested logical tree using a package.json and a package lock.
JavaScript
44
star
68

read-package-tree

Read the contents of node_modules
JavaScript
42
star
69

jobs

41
star
70

unique-filename

Generate a unique filename for use in temporary directories or caches.
JavaScript
40
star
71

lock-verify

Report if your package.json is out of sync with your package-lock.json
JavaScript
38
star
72

fstream-ignore

JavaScript
37
star
73

benchmarks

The npm CLI's benchmark suite
JavaScript
36
star
74

npm-lifecycle

npm lifecycle script runner
JavaScript
36
star
75

wombat-cli

The wombat cli tool.
JavaScript
35
star
76

npme-installer

npm Enterprise installer
JavaScript
35
star
77

couch-login

A module for doing logged-in requests against a couchdb server
JavaScript
33
star
78

libnpmexec

npm exec (npx) Programmatic API
JavaScript
33
star
79

npm-audit-report

npm audit security report
JavaScript
33
star
80

ansible-nagios

Ansible role for building Nagios 4.
Perl
32
star
81

config

Configuration management for https://github.com/npm/cli
JavaScript
32
star
82

parse-conflict-json

Parse a JSON string that has git merge conflicts, resolving if possible
JavaScript
32
star
83

npm-profile

Make changes to your npmjs.com profile via cli or library
JavaScript
31
star
84

unique-slug

Generate a unique character string suitible for use in files and URLs.
JavaScript
31
star
85

fstream-npm

fstream class for creating npm packages
JavaScript
30
star
86

redsess

Yet another redis session thing for node.
JavaScript
30
star
87

concurrent-couch-follower

a couch follower wrapper that you can use to be sure you don't miss any documents even if you process them asynchronously.
JavaScript
28
star
88

npm-registry-mock

mock the npm registry
JavaScript
27
star
89

lint

lint the npmcli way
JavaScript
26
star
90

fs

filesystem helper functions, wrappers, and promisification for the npm cli
JavaScript
25
star
91

template-oss

a template package for npm CLI team development
JavaScript
25
star
92

libnpmsearch

programmatic API for the shiny new npm search endpoint
JavaScript
25
star
93

bin-links

.bin/ script linker
JavaScript
24
star
94

libnpmaccess

programmatic api for `npm access`
JavaScript
24
star
95

logos

official logos for npm, Inc
22
star
96

types

Typescript definitions for npm registry content
JavaScript
22
star
97

public-api

21
star
98

deprecate-holder

An npm package that holds a spot.
21
star
99

map-workspaces

Retrieves a name:pathname Map for a given workspaces config
JavaScript
20
star
100

libnpmversion

library to do the things that 'npm version' does
JavaScript
20
star