• Stars
    star
    791
  • Rank 57,558 (Top 2 %)
  • Language
    JavaScript
  • License
    MIT License
  • Created over 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Low-overhead lexer dedicated to ES module parsing for fast analysis

ES Module Lexer

Build Status

A JS module syntax lexer used in es-module-shims.

Outputs the list of exports and locations of import specifiers, including dynamic import and import meta handling.

A very small single JS file (4KiB gzipped) that includes inlined Web Assembly for very fast source analysis of ECMAScript module syntax only.

For an example of the performance, Angular 1 (720KiB) is fully parsed in 5ms, in comparison to the fastest JS parser, Acorn which takes over 100ms.

Comprehensively handles the JS language grammar while remaining small and fast. - ~10ms per MB of JS cold and ~5ms per MB of JS warm, see benchmarks for more info.

Built with Chomp

Usage

npm install es-module-lexer

For use in CommonJS:

const { init, parse } = require('es-module-lexer');

(async () => {
  // either await init, or call parse asynchronously
  // this is necessary for the Web Assembly boot
  await init;

  const source = 'export var p = 5';
  const [imports, exports] = parse(source);
  
  // Returns "p"
  source.slice(exports[0].s, exports[0].e);
  // Returns "p"
  source.slice(exports[0].ls, exports[0].le);
})();

An ES module version is also available:

import { init, parse } from 'es-module-lexer';

(async () => {
  await init;

  const source = `
    import { name } from 'mod\\u1011';
    import json from './json.json' assert { type: 'json' }
    export var p = 5;
    export function q () {

    };
    export { x as 'external name' } from 'external';

    // Comments provided to demonstrate edge cases
    import /*comment!*/ (  'asdf', { assert: { type: 'json' }});
    import /*comment!*/.meta.asdf;
  `;

  const [imports, exports] = parse(source, 'optional-sourcename');

  // Returns "modထ"
  imports[0].n
  // Returns "mod\u1011"
  source.slice(imports[0].s, imports[0].e);
  // "s" = start
  // "e" = end

  // Returns "import { name } from 'mod'"
  source.slice(imports[0].ss, imports[0].se);
  // "ss" = statement start
  // "se" = statement end

  // Returns "{ type: 'json' }"
  source.slice(imports[1].a, imports[1].se);
  // "a" = assert, -1 for no assertion

  // Returns "external"
  source.slice(imports[2].s, imports[2].e);

  // Returns "p"
  source.slice(exports[0].s, exports[0].e);
  // Returns "p"
  source.slice(exports[0].ls, exports[0].le);
  // Returns "q"
  source.slice(exports[1].s, exports[1].e);
  // Returns "q"
  source.slice(exports[1].ls, exports[1].le);
  // Returns "'external name'"
  source.slice(exports[2].s, exports[2].e);
  // Returns -1
  exports[2].ls;
  // Returns -1
  exports[2].le;

  // Dynamic imports are indicated by imports[2].d > -1
  // In this case the "d" index is the start of the dynamic import bracket
  // Returns true
  imports[2].d > -1;

  // Returns "asdf" (only for string literal dynamic imports)
  imports[2].n
  // Returns "import /*comment!*/ (  'asdf', { assert: { type: 'json' } })"
  source.slice(imports[3].ss, imports[3].se);
  // Returns "'asdf'"
  source.slice(imports[3].s, imports[3].e);
  // Returns "(  'asdf', { assert: { type: 'json' } })"
  source.slice(imports[3].d, imports[3].se);
  // Returns "{ assert: { type: 'json' } }"
  source.slice(imports[3].a, imports[3].se - 1);

  // For non-string dynamic import expressions:
  // - n will be undefined
  // - a is currently -1 even if there is an assertion
  // - e is currently the character before the closing )

  // For nested dynamic imports, the se value of the outer import is -1 as end tracking does not
  // currently support nested dynamic immports

  // import.meta is indicated by imports[3].d === -2
  // Returns true
  imports[4].d === -2;
  // Returns "import /*comment!*/.meta"
  source.slice(imports[4].s, imports[4].e);
  // ss and se are the same for import meta
})();

CSP asm.js Build

The default version of the library uses Wasm and (safe) eval usage for performance and a minimal footprint.

Neither of these represent security escalation possibilities since there are no execution string injection vectors, but that can still violate existing CSP policies for applications.

For a version that works with CSP eval disabled, use the es-module-lexer/js build:

import { parse } from 'es-module-lexer/js';

Instead of Web Assembly, this uses an asm.js build which is almost as fast as the Wasm version (see benchmarks below).

Escape Sequences

To handle escape sequences in specifier strings, the .n field of imported specifiers will be provided where possible.

For dynamic import expressions, this field will be empty if not a valid JS string.

Facade Detection

Facade modules that only use import / export syntax can be detected via the third return value:

const [,, facade] = parse(`
  export * from 'external';
  import * as ns from 'external2';
  export { a as b } from 'external3';
  export { ns };
`);
facade === true;

Environment Support

Node.js 10+, and all browsers with Web Assembly support.

Grammar Support

  • Token state parses all line comments, block comments, strings, template strings, blocks, parens and punctuators.
  • Division operator / regex token ambiguity is handled via backtracking checks against punctuator prefixes, including closing brace or paren backtracking.
  • Always correctly parses valid JS source, but may parse invalid JS source without errors.

Limitations

The lexing approach is designed to deal with the full language grammar including RegEx / division operator ambiguity through backtracking and paren / brace tracking.

The only limitation to the reduced parser is that the "exports" list may not correctly gather all export identifiers in the following edge cases:

// Only "a" is detected as an export, "q" isn't
export var a = 'asdf', q = z;

// "b" is not detected as an export
export var { a: b } = asdf;

The above cases are handled gracefully in that the lexer will keep going fine, it will just not properly detect the export names above.

Benchmarks

Benchmarks can be run with npm run bench.

Current results for a high spec machine:

Wasm Build

Module load time
> 5ms
Cold Run, All Samples
test/samples/*.js (3123 KiB)
> 18ms

Warm Runs (average of 25 runs)
test/samples/angular.js (739 KiB)
> 3ms
test/samples/angular.min.js (188 KiB)
> 1ms
test/samples/d3.js (508 KiB)
> 3ms
test/samples/d3.min.js (274 KiB)
> 2ms
test/samples/magic-string.js (35 KiB)
> 0ms
test/samples/magic-string.min.js (20 KiB)
> 0ms
test/samples/rollup.js (929 KiB)
> 4.32ms
test/samples/rollup.min.js (429 KiB)
> 2.16ms

Warm Runs, All Samples (average of 25 runs)
test/samples/*.js (3123 KiB)
> 14.16ms

JS Build (asm.js)

Module load time
> 2ms
Cold Run, All Samples
test/samples/*.js (3123 KiB)
> 34ms

Warm Runs (average of 25 runs)
test/samples/angular.js (739 KiB)
> 3ms
test/samples/angular.min.js (188 KiB)
> 1ms
test/samples/d3.js (508 KiB)
> 3ms
test/samples/d3.min.js (274 KiB)
> 2ms
test/samples/magic-string.js (35 KiB)
> 0ms
test/samples/magic-string.min.js (20 KiB)
> 0ms
test/samples/rollup.js (929 KiB)
> 5ms
test/samples/rollup.min.js (429 KiB)
> 3.04ms

Warm Runs, All Samples (average of 25 runs)
test/samples/*.js (3123 KiB)
> 17.12ms

Building

This project uses Chomp for building.

With Chomp installed, download the WASI SDK 12.0 from https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-12.

Locate the WASI-SDK as a sibling folder, or customize the path via the WASI_PATH environment variable.

Emscripten emsdk is also assumed to be a sibling folder or via the EMSDK_PATH environment variable.

Example setup:

git clone https://github.com:guybedford/es-module-lexer
git clone https://github.com/emscripten-core/emsdk
cd emsdk
git checkout 1.40.1-fastcomp
./emsdk install 1.40.1-fastcomp
cd ..
wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-linux.tar.gz
gunzip wasi-sdk-12.0-linux.tar.gz
tar -xf wasi-sdk-12.0-linux.tar
mv wasi-sdk-12.0-linux.tar wasi-sdk-12.0
cargo install chompbuild
cd es-module-lexer
chomp test

For the asm.js build, git clone emsdk from is assumed to be a sibling folder as well.

License

MIT

More Repositories

1

es-module-shims

Shims for new ES modules features on top of the basic modules support in browsers
JavaScript
1,514
star
2

require-css

A RequireJS CSS loader plugin to allow CSS requires and optimization
JavaScript
983
star
3

chomp

'JS Make' - parallel task runner for the frontend ecosystem with a JS extension system.
Rust
141
star
4

require-less

LESS loader plugin for RequireJS with builds, based on RequireCSS
JavaScript
139
star
5

wasm-intro

Egghead.io WASM Introduction Examples
HTML
119
star
6

wasm-demo

Egghead.io WASM Introduction Demo Project
HTML
107
star
7

isomorphic-browser-modules

A workflow for using <script type="module"> with a fallback for older browsers
JavaScript
51
star
8

require-is

Conditional loader plugin for RequireJS with code branch build support
JavaScript
51
star
9

amd-loader

RequireJS loader plugin helper for transpiler and template plugins
JavaScript
34
star
10

import-maps-extensions

Extensions to the WICG import maps proposal
HTML
30
star
11

systemjs-webpack-plugin

Webpack bundling for SystemJS
JavaScript
28
star
12

wasm-stdlib-hack

Copy of libc and libcxx includes from Emscripten with some wast linking helpers
C++
25
star
13

markdown-component

JavaScript
21
star
14

proposal-weak-imports

Experimental Weak Imports Proposal for ECMAScript
HTML
20
star
15

cjs

CommonJS loader plugin for RequireJS
JavaScript
17
star
16

jspm-react-component-demo

JavaScript
14
star
17

react-jspm-es6-gulp-example

a work in progress example showing react / react-nested-router / jspm / gulp / es6
JavaScript
14
star
18

system-md

jspm Markdown plugin
JavaScript
12
star
19

es6

ES6 Loader Plugin for RequireJS
JavaScript
10
star
20

packagemap-polyfill

Experimental polyfill approach for package name maps in browsers
JavaScript
10
star
21

js-components

WebAssembly
9
star
22

require-inline

A small add-on for RequireJS providing inline sync load support. Useful for dynamic widgets that need JavaScript attachments to occur instantly during the page load.
JavaScript
8
star
23

amdquery

A selector plugin for RequireJS with a dynamic query selector polyfill, supporting modular DOM utilities and builds.
JavaScript
6
star
24

jspm-beta-guide

5
star
25

proposal-export-star-default

Proposal to include default export in `export * from 'module'`
HTML
5
star
26

chomp-extensions

Chomp template collection
JavaScript
5
star
27

proposal-pkg-entries

package.json "entries" field proposal
4
star
28

systemjs-istanbul

SystemJS Istanbul Coverage Helper
JavaScript
4
star
29

chomp-action

GitHub Action to Install Chomp
JavaScript
4
star
30

wasi_unstable

WASI Unstable Experimental ES Module Integration
JavaScript
4
star
31

node-resolve-hook

Draft proposal for NodeJS ES modules pipeline hooks
4
star
32

karma-systemjs-experiment

JavaScript
3
star
33

jspm-demo

JavaScript
3
star
34

extract-requires

Extract CommonJS requires from a JavaScript source string with simple tokenizing
JavaScript
3
star
35

modules-caching

JavaScript
2
star
36

cjs-named-exports-loader

JavaScript
2
star
37

jspm-test-demo

jspm test demo
JavaScript
2
star
38

proposal-pkg-targets

package.json proposal for differential target resolution
2
star
39

hbs

JavaScript
2
star
40

babel-modules-system-source-maps-bug

JavaScript
1
star
41

plugin-traceur

SystemJS Traceur Plugin (wip)
1
star
42

voxel-demo

JavaScript
1
star
43

json

AMD wrapper module for JSON parsing. Feature detects JSON.stringify and JSON.parse, dynamically requesting json2 polyfill when necessary.
JavaScript
1
star
44

asset-plugin-experiment

JavaScript
1
star
45

module-property-refined-proposal

Yet another proposal for ES module distinction in Node
1
star
46

esm-resolver-test

JavaScript
1
star
47

amd-version

RequireJS semver-compatible versioning plugin
JavaScript
1
star
48

chrome-modules-bug

ES modules circular test case
JavaScript
1
star
49

separate-css-test

JavaScript
1
star
50

styletab

JavaScript
1
star
51

systemjs-sandbox

Prototype sandbox demo for SystemJS
JavaScript
1
star
52

require-cs

JavaScript
1
star
53

modules-link-preload

Testing link rel=preload for module scripts in browsers
JavaScript
1
star
54

modules-benchmarks

Module loading benchmarks
JavaScript
1
star