• Stars
    star
    309
  • Rank 135,306 (Top 3 %)
  • Language
    Lua
  • License
    MIT License
  • Created over 4 years ago
  • Updated 10 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Async Await in 90 lines of code.

Lua Async Await

Async Await in 90 lines of code.

Originally written for Neovim, because it uses the same libuv eventloop from NodeJS.

Works for any LUA code

Special Thanks

svermeulen for fixing inability to return functions.

Preface

This tutorial assumes that you are familiar with the concept of async await

You will also need to read through the first 500 words of how coroutines work in lua.

Luv

Neovim use libuv for async, the same monster that is the heart of NodeJS.

The libuv bindings are exposed through luv for lua, this is accessed using vim.loop.

Most of the luv APIs are similar to that of NodeJS, ie in the form of

API :: (param1, param2, callback)

Our goal is avoid the dreaded calback hell.

Preview

local a = require "async"

local do_thing = a.sync(function (val)
  local o = a.wait(async_func())
  return o + val
end)

local main = a.sync(function ()
  local thing = a.wait(do_thing()) -- composable!

  local x = a.wait(async_func())
  local y, z = a.wait_all{async_func(), async_func()}
end)

main()

Coroutines

If you don't know how coroutines work, go read the section on generators on MDN.

It is in js, but the idea is identical, and the examples are much better.


Here is an example of coroutines in Lua:

Note that in Lua code coroutine is not a coroutine, it is an namespace.

To avoid confusion, I will follow the convention used in the Lua book, and use thread to denote coroutines in code.

local co = coroutine

local thread = co.create(function ()
  local x, y, z = co.yield(something)
  return 12
end)

local cont, ret = co.resume(thread, x, y, z)

Notice the similarities with async await

In both async await and coroutines, the LHS of the assignment statements receives values from the RHS.

This is how it works in all synchronous assignments. Except, we can defer the transfer of the values from RHS.

The idea is that we will make RHS send values to LHS, when RHS is ready.

Synchronous Coroutines

To warm up, we will do a synchronous version first, where the RHS is always ready.

Here is how you send values to a coroutine:

co.resume(thread, x, y, z)

The idea is that we will repeat this until the coroutine has been "unrolled"

local pong = function (thread)
  local nxt = nil
  nxt = function (cont, ...)
    if not cont
      then return ...
      else return nxt(co.resume(thread, ...))
    end
  end
  return nxt(co.resume(thread))
end

if we give pong some coroutine, it will recursively run the coroutine until completion

local thread = co.create(function ()
  local x = co.yield(1)
  print(x)
  local y, z = co.yield(2, 3)
  print(y)
end)

pong(thread)

We can expect to see 1, 2 3 printed.

Thunk

Once you understand how the synchronous pong works, we are super close!

But before we make the asynchronous version, we need to learn one more simple concept.

For our purposes a Thunk is function whose purpose is to invoke a callback.

i.e. It adds a transformation of (arg, callback) -> void to arg -> (callback -> void) -> void

local read_fs = function (file)
  local thunk = function (callback)
    fs.read(file, callback)
  end
  return thunk
end

This too, is a process that can be automated:

local wrap = function (func)
  local factory = function (...)
    local params = {...}
    local thunk = function (step)
      table.insert(params, step)
      return func(unpack(params))
    end
    return thunk
  end
  return factory
end

local thunk = wrap(fs.read)

So why do we need this?

Async Await

The answer is simple! We will use thunks for our RHS!


With that said, we will still need one more magic trick, and that is to make a step function.

The sole job of the step funciton is to take the place of the callback to all the thunks.

In essence, on every callback, we take 1 step forward in the coroutine.

local pong = function (func, callback)
  assert(type(func) == "function", "type error :: expected func")
  local thread = co.create(func)
  local step = nil
  step = function (...)
    local stat, ret = co.resume(thread, ...)
    assert(stat, ret)
    if co.status(thread) == "dead" then
      (callback or function () end)(ret)
    else
      assert(type(ret) == "function", "type error :: expected func")
      ret(step)
    end
  end
  step()
end

Notice that we also make pong call a callback once it is done.


We can see it in action here:

local echo = function (...)
  local args = {...}
  local thunk = function (step)
    step(unpack(args))
  end
  return thunk
end

local thread = co.create(function ()
  local x, y, z = co.yield(echo(1, 2, 3))
  print(x, y, z)
  local k, f, c = co.yield(echo(4, 5, 6))
  print(k, f, c)
end)

pong(thread)

We can expect this to print 1 2 3 and 4 5 6

Note, we are using a synchronous echo for illustration purposes. It doesn't matter when the callback is invoked. The whole mechanism is agnostic to timing.

You can think of async as the more generalized version of sync.

You can run an asynchronous version in the last section.

Await All

One more benefit of thunks, is that we can use them to inject arbitrary computation.

Such as joining together many thunks.

local join = function (thunks)
  local len = table.getn(thunks)
  local done = 0
  local acc = {}

  local thunk = function (step)
    if len == 0 then
      return step()
    end
    for i, tk in ipairs(thunks) do
      local callback = function (...)
        acc[i] = {...}
        done = done + 1
        if done == len then
          step(unpack(acc))
        end
      end
      tk(callback)
    end
  end
  return thunk
end

This way we can perform await_all on many thunks as if they are a single one.

More Sugar

All this explicit handling of coroutines are abit ugly. The good thing is that we can completely hide the implementation detail to the point where we don't even need to require the coroutine namespace!

Simply wrap the coroutine interface with some friendly helpers

local pong = function (func, callback)
  local thread = co.create(func)
  ...
end

local await = function (defer)
  return co.yield(defer)
end

local await_all = function (defer)
  return co.yield(join(defer))
end

Composable

At this point we are almost there, just one more step!

local sync = wrap(pong)

We wrap pong into a thunk factory, so that calling it is no different than yielding other thunks. This is how we can compose together our async await.

It's thunks all the way down.

Tips and Tricks

In Neovim, we have something called textlock, which prevents many APIs from being called unless you are in the main event loop.

This will prevent you from essentially modifying any Neovim states once you have invoked a vim.loop funciton, which run in a seperate loop.

Here is how you break back to the main loop:

local main_loop = function (f)
  vim.schedule(f)
end
a.sync(function ()
  -- do something in other loop
  a.wait(main_loop)
  -- you are back!
end)()

Plugin!

I have bundle up this tutorial as a vim plugin, you can install it the usual way.

Plug 'ms-jpq/lua-async-await', {'branch': 'neo'}

and then call the test functions like so:

:LuaAsyncExample

:LuaSyncExample

:LuaTextlockFail

:LuaTextLockSucc

More Repositories

1

coq_nvim

Fast as FUCK nvim completion. SQLite, concurrent scheduler, hundreds of hours of optimization.
Python
3,435
star
2

sad

CLI search and replace | Space Age seD
Rust
1,681
star
3

chadtree

File manager for Neovim. Better than NERDTree.
Python
1,618
star
4

noact

Minimalist React (< 70 lines)
TypeScript
317
star
5

gay

Colour your text / terminal to be more gay. πŸ³οΈβ€πŸŒˆ
Python
305
star
6

isomorphic_copy

Cross platform clipboard | networkless! remote copy
Python
206
star
7

kde-in-docker

So you can run KDE inside a browser
Dockerfile
109
star
8

docker-time-machine

Zero Configuration, time machine for MacOS | Docker 🐳
Shell
83
star
9

markdown-live-preview

markdown live preview | works with any editor
Python
73
star
10

windows-in-docker

Manage Windows in Docker (easy)
Python
61
star
11

coq.thirdparty

Module lua sources for `coq.nvim`, first & third party
Lua
61
star
12

windows-in-docker-old

WHY? So you can run Windows "as a container"
Shell
51
star
13

nvim_rc

Python
49
star
14

simple-traefik-dash

Zero conf service dashboard for Traefik v2 Ingress Controller
F#
33
star
15

sync-dockerhub-readme

Github action to sync <readme.md> to Dockerhub
Python
22
star
16

simple-traefik-identity

Simple & Configurable -- single sign-on auth, for Traefik.
F#
22
star
17

coq.artifacts

Python
16
star
18

docker-home-router

Python
13
star
19

vim.bench

Reproducible nvim completion framework benchmarks.
Python
13
star
20

vim.conf-2023

SCSS
10
star
21

download-windows-iso

Python script to download official windows iso
Python
9
star
22

pynvim_pp

Python
8
star
23

snips

Vim Snippet
7
star
24

tmux_rc

Python
6
star
25

forechan

Go style CSP for Python
Python
6
star
26

lab

We have cloud at home - mom
Shell
6
star
27

kaleidoscope

Interactive & arbitrary Chaos Game fractal visualization tool
TypeScript
5
star
28

vim.benchmarks

5
star
29

nda

NPM - Domain Agnostic :: use with NPM dependency manager
TypeScript
5
star
30

nodebook-docker

Docker in Docker version of Nodebook (multilang repl)
Dockerfile
5
star
31

shell_rc

Shell
5
star
32

simple-markdown-showcase

Responsive personal portfolio pages using Github APIs.
Python
5
star
33

bing-daily-images

A docker container that downloads bing daily images for you.
Python
4
star
34

simple-reeducks

Minimalist Classless Redux in Swift
Swift
4
star
35

auto-github-push

Github action to refresh various Github inactivity countdowns.
Python
4
star
36

sortd

sorting commands for stdin -> stdout
Python
4
star
37

ms-jpq

4
star
38

iterm2

Python
3
star
39

ms-jpq.github.io

https://ms-jpq.github.io
HTML
3
star
40

std2

Python
3
star
41

vim.conf

TeX
3
star
42

defcon604-2023

Shell
2
star
43

emacs_rc

Emacs Lisp
2
star
44

fda

Fsharp - Domain Agnostic :: use with Paket dependency manager
F#
2
star
45

py-dev

Python
2
star
46

deb

2
star
47

kaleidoscope-page

Page for Kaleidoscope
HTML
1
star
48

languagetool-cli

Python
1
star
49

homebrew-sad

Ruby
1
star
50

cda

SASS - Domain Agnostic :: use with NPM dependency manager
SCSS
1
star
51

noact-page

Page for Noact
CSS
1
star
52

deb-src

PPA builder, hosted on Github pages
Makefile
1
star
53

mol

Minimal hardware accelerated `curl -- linux.iso | qemu-system-aarch64` scripts.
Shell
1
star