• Stars
    star
    110
  • Rank 316,770 (Top 7 %)
  • Language
    JavaScript
  • License
    MIT License
  • Created about 5 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This is an Express middleware that makes developing javascript in a monorepo easier.

kevin-middleware

(ง° ͜ʖ°)ง 📦 📦 📦

Kevin is an Express-style middleware that makes developing with Webpack in a monorepo a lot simpler. It's loosely based off of Webpack's dev middleware, and it is intended to be used as a replacement for it. Only use this middleware in development please!

Using Webpack in development in a monorepo can be challenging because, by default, it will try to keep your entire JavaScript codebase in memory. This can be remarkably resource-intensive. Kevin addresses this problem by allowing you to create separate Webpack configs for different parts of your codebase; it then manages these configs by having Webpack only watch and build relevant files.

How does it do that?

When Kevin receives a request for an asset, it determines which config is responsible for building that particular asset and spins up an instance of Webpack to handle building it. Kevin will keep any compiler running as long as you regularly use it, up to a configurable limit. It automatically turns off any unused compilers in order to conserve your resources. It blocks on requests, but it will render a loading modal for newly-initialized compilers (since initial builds can take a bit of time).

Requirements

To use this middleware, you must:

  • be using Webpack version 4.0.0 or later (but not Webpack 5 yet!),
  • use it with a server that accepts Express-style middleware,
  • and have a name property specified in all of your Webpack configs.

How do I use it?

First, install kevin-middleware in your project:

npm install --save-dev kevin-middleware

Then, add kevin-middleware to your server. For example:

const express = require("express");
const Kevin = require("kevin-middleware");

// This is an array of webpack configs. Each config **must** be named so that we can
// uniquely identify each one consistently. A regular ol' webpack config should work just
// fine as well.
const webpackConfigs = require("path/to/webpack.config.js");

// Setup your server and configure Kevin
const app = express();

const kevin = new Kevin(webpackConfigs, {
    kevinPublicPath: "http://localhost:3000",
});
app.use(kevin.getMiddleware());

// Serve static files as needed. This is required if you generate async chunks; Kevin
// only knows about the entrypoints in your configs, so it has to assume that everything
// else is handled by a different middleware.
app.use("/ac/webpack/js", express.static(webpackConfigs[0].output.path));

// Let 'er rip
app.listen(3000);

For a complete example, check out this repository.


Kevin is initialized with two arguments: your webpack config (or more probably your array of configs) and an options object:

const kevin = new Kevin(webpackConfigs [, options ] );

Once you've instantiated a new instance of Kevin, call getMiddleware to get access to an Express-style middleware function:

app.use(kevin.getMiddleware());

Options

The Kevin constructor accepts an options object. All of these are optional and have reasonable defaults.

maxCompilers

  • Type: Integer
  • Default: 3

The maximum number of compilers you want to have running at any point in time. Set this to 0 to never evict anything (but that will probably make you run out of memory).

buildOnly

  • Type: Boolean
  • Default: false

Only build assets; don't handle serving them. This is useful if you want to do something with the built asset before serving it, in which case you'd handle that logic yourself after the Kevin middleware does its thing.

kevinPublicPath

  • Type: String
  • Default: null

Root path for Kevin's internal API to be exposed through. This is used to tell the loading modal where to look for data on the status of builds. This should be set to the path that this middleware is bound to. For now, this path can not end in a slash. If set to null, auto-refresh is disabled.

kevinApiPrefix

  • Type: String
  • Default: "/__kevin"

This is a prefix for Kevin's internal API. You probably don't need to change this unless you have an asset being served that's named __kevin or something.

getAssetName

  • Type: Function
  • Default: (requestPath, req, res) => requestPath.replace(/^\//, "").replace(/\.js$/, "")

Given a request path, req object, and res object, return the name of the asset we're trying to serve. Useful if you have entries that don't map to the filenames they render.

selectConfigName

  • Type: Function
  • Default:
(requestPath, configNames) => {
    if (!configNames) {
        return null;
    }
    if (configNames.length > 1) {
        logError(
            `Multiple configNames found for ${reqPath}: ${configNames.join(
                ","
            )}. Using first one.`
        );
    }
    return configNames[0];
};

Given a request path and a list of configNames, return the name of the config to use. Useful if you know more about the request URI and if there are multiple configs that claim to serve it.

additionalOverlayInfo

  • Type: String
  • Default: ""

This is a string that's inserted into the overlay, in order to provide users with additional information. It's useful if you'd like to provide feedback to users of your server, like "If you run into issues, try running restart_server_please.sh". This string may contain valid HTML.

Hooks

To further extend Kevin's capabilities, we used Webpack's Tapable framework to provide access to some of Kevin's core functionality. You can use a hook much like you would with Webpack:

// Tap into hooks first...
kevin.hooks.compilerStart.tap("MySweetLoggingPlugin", (compilerName) => {
    CustomLogger.log(`The ${compilerName} compiler just started up`);
});

// before adding Kevin to your server.
app.use(kevin.getMiddleware());

Here are the hooks you can take advantage of:

start (SyncHook)

This hook is run just after the middleware starts and before any requests are handled. You may find this useful to eagerly start a compiler as soon as Kevin starts, or to attach file watchers to your configs to restart a compiler when its config has changed on disk. Callback parameters:

  • configs: an array of the configs Kevin is responsible for
  • configManager : PublicConfigManager: an object containing methods for understanding and managing configs and their compilers
    • buildConfig(configName : string) => Promise<bool> — given the name of a managed webpack config, starts a compiler for it (if one doesn't already exist). Returns a promise that resolves to true immediately if this is the first build for that config, or false after a rebuild has finished.
    • closeCompiler(configName : string) => Promise<string|null> — given a config name, close the compiler responsible for it, if it exists. Returns a promise that resolves to the name of the config once the compiler responsible for it has closed, or null if no such compiler could be found.
    • isCompilerActive(configName : string) => bool — given a config name, returns true if and only if there is a compiler running for that config.
    • getActiveCompilerNames() => Array<string> — returns a list of all active compiler names.

compilerStart (SyncHook)

This hook is run immediately before a webpack compiler is started. It has one parameter:

  • compilerName — The name of the compiler that we're gonna start.

compilerClose (SyncHook)

This hook is run just before a compiler is about to close. It has one parameter:

  • evictionOptions — An object containing two properties:
    • compilerToEvict : string — the name of the compiler we're about to evict.
    • compilerStats — an object containing metadata about the compiler, including its current build status, a measure of its' frequency of use and its frecency, a list of times it's been used, whether or not its pinned, and any errors it currently has.

handleRequest (SyncHook)

This hook is called after we know Kevin is responsible for a request, but before anything actually happens (like closing or starting compilers). It has a few parameters:

  • request — the Express request object.
  • assetName : string — the name of the asset that we're going to build.
  • compilerName : string — the name of the compiler that we're planning on using (or spinning up) to handle the request.

Internal API

If you'd like to access additional details about the status of Kevin (and the compilers it manages), you can hit Kevin't internal web API. By default, this is hosted at [kevinPublicPath][kevinApiPrefix]. So, for example, if you had the following configuration:

{
    kevinPublicPath: "http://your.webpack.server.dev",
    kevinApiPrefix: "/__kevin"
}

the internal API would be hosted at http://your.webpack.server.dev/__kevin/[route].

The following routes are available:

/build-status

This endpoint shows the state of each compiler. The overlay uses this endpoint to know whether or not to reload the page.

/compiler-info

This endpoint shows general details about each compiler, particularly metrics around its use and whether it may be eligible for eviction.

/memory-usage

This enpoint lists memory stats for the process in which kevin is being run.

Why did you do all this?

Webpack is an awesome JavaScript build system. It's powerful, flexible, and widely adopted. However, using it to build and manage a monorepo can be tough for a couple of reasons:

  1. Webpack retains a lot of data in memory to allow for fast iterative builds and other performance-related optimizations.
  2. In order to identify optimizations across entire projects (like identifying groups of modules to bundle into their own asset), Webpack needs to build the entirety of your project at once.

The combination of these things makes it very resource-intensive to build large projects on reasonable computers. This is particularly problematic at Etsy, where most of our web code lives in one large repository. Most of the current solutions involve manually running an instance of webpack with a configuration for a particular part of your codebase, but having to constantly start and stop compilers just to browse around the site can be frustrating. We needed something that would be mindful of our resource limitations, while still being essentially maintenance-free; as long as the server is running, built JavaScript should just show up in the browser.

Our solution was to separate different parts of our site into their own configs, placing assets in the same config when they belonged to the same experience or flow. To glue it all together, we use Kevin to ensure that regions were automatically started and stopped as we browsed arond the site, with as little interaction from developers as possible. We named this middleware Kevin because it was the best name we could think of at the time.

More Repositories

1

AndroidStaggeredGrid

An Android staggered grid view which supports multiple columns with rows of varying sizes.
Java
4,756
star
2

skyline

It'll detect your anomalies! Part of the Kale stack.
Python
2,135
star
3

logster

Parse log files, generate metrics for Graphite and Ganglia
Python
1,968
star
4

deployinator

Deployinate!
Ruby
1,878
star
5

morgue

post mortem tracker
PHP
1,017
star
6

411

An Alert Management Web Application
PHP
971
star
7

feature

Etsy's Feature flagging API used for operational rampups and A/B testing.
PHP
869
star
8

MIDAS

Mac Intrusion Detection Analysis System
833
star
9

opsweekly

On call alert classification and reporting
JavaScript
761
star
10

oculus

The metric correlation component of Etsy's Kale system
Java
707
star
11

mctop

a top like tool for inspecting memcache key values in realtime
Ruby
507
star
12

supergrep

realtime log streamer
JavaScript
411
star
13

Conjecture

Scalable Machine Learning in Scalding
Java
361
star
14

statsd-jvm-profiler

Simple JVM Profiler Using StatsD and Other Metrics Backends
Java
330
star
15

nagios-herald

Add context to Nagios alerts
Ruby
322
star
16

dashboard

JavaScript
308
star
17

boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
Python
262
star
18

Testing101

Etsy's educational materials on testing and design
PHP
262
star
19

DebriefingFacilitationGuide

Leading Groups at Etsy to Learn From Accidents
247
star
20

phpunit-extensions

Etsy PHPUnit Extensions
PHP
228
star
21

nagios_tools

Tools for use with Nagios
Python
173
star
22

open-api

We are working on a new version of Etsy’s Open API and want feedback from developers like you.
166
star
23

TryLib

TryLib is a simple php library that helps you generate a diff of your working copy and send it to Jenkins to run the test suite(s) on the latest code patched with your changes.
PHP
155
star
24

BugHunt-iOS

Objective-C
148
star
25

mod_realdoc

Apache module to support atomic deploys - http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/
C
128
star
26

ab

Etsy's little framework for A/B testing, feature ramp up, and more.
128
star
27

wpt-script

Scripts to generate WebPagetest tests and download results
PHP
121
star
28

applepay-php

A PHP extension that verifies and decrypts Apple Pay payment tokens
C
118
star
29

foodcritic-rules

Etsy's foodcritic rules
Ruby
115
star
30

mixer

a tool to initiate meetings by randomly pairing individuals
Go
100
star
31

cloud-jewels

Estimate energy consumption using GCP Billing Data
TSQL
96
star
32

jenkins-master-project

Jenkins Plugin: Master Project. Jenkins project type that allows for selection of sub-jobs to execute, watch, and report worst status of all sub-projects.
Java
83
star
33

Sahale

A Cascading Workflow Visualizer
JavaScript
83
star
34

PushBot

An IRC Bot for organizing code pushes
Java
79
star
35

cdncontrol

CLI tool for working with multiple CDNs
Ruby
79
star
36

rules_grafana

Bazel rules for building Grafana dashboards
Starlark
70
star
37

chef-whitelist

Simple library to enable host based rollouts of changes
Ruby
68
star
38

rfid-checkout

Low Frequency RFID check out/in client for Raspberry Pi
Python
64
star
39

Etsy-Engineering-Career-Ladder

Etsy's Engineering Career Ladder
HTML
61
star
40

Evokit

Rust
60
star
41

ELK-utils

Utilities for working with the ELK (Elasticsearch, Logstash, Kibana) stack
Ruby
59
star
42

incpath

PHP extension to support atomic deploys
C
52
star
43

arbiter

A utility for generating Oozie workflows from a YAML definition
Java
48
star
44

VIPERBuilder

Scaffolding for building apps in a clean way with VIPER architecture
Swift
41
star
45

chef-handlers

Chef handlers we use at Etsy
Ruby
40
star
46

sbt-checkstyle-plugin

SBT Plugin for Running Checkstyle on Java Sources
Scala
32
star
47

es-restlog

Plugin for logging Elasticsearch REST requests
Java
29
star
48

yubigpgkeyer

Script to make RSA authentication key generation on Yubikeys differently painful
Python
28
star
49

Apotheosis

Python
28
star
50

jenkins-deployinator

Jenkins Plugin: Deployinator. Links key deployinator information to Jenkins builds via the CLI.
Java
25
star
51

sbt-compile-quick-plugin

SBT Plugin for Compiling a Single File
Scala
25
star
52

geonames

Scripts for using Geonames
PHP
24
star
53

jading

cascading.jruby build and execution tool
16
star
54

etsy.github.com

Etsy! on Github!
HTML
16
star
55

divertsy-client

The Android client for running DIVERTsy, a waste stream recording tool to help track diversion rates.
Java
13
star
56

cdncontrol_ui

A web UI for Etsy's cdncontrol tool
CSS
13
star
57

terraform-demux

A user-friendly launcher (à la bazelisk) for Terraform.
Go
12
star
58

logstash-plugins

Ruby
11
star
59

jenkins-triggering-user

Jenkins Plugin: Triggering User. Populates a $TRIGGERING_USER environment variable from the build cause and other sources, a best guess.
10
star
60

EtsyCompositionalLayoutBridge

iOS framework that allows for simultaneously leveraging flow layout and compositional layout in collection views
Swift
3
star
61

consulkit

Ruby API for interacting with HashiCorp's Consul.
Ruby
1
star
62

soft-circuits-workshop

Etsy Soft Circuits Workshop
Arduino
1
star