The problem

Deploying your app to multiple regions along with your data is a great way to make your app really fast, but there are two issues:

1. Read replica instances can only read from the database, they cannot write to it.
2. There's an edge case where the user could write to the primary instance and then read from a replica instance before replication is finished.

The first problem is as simple as making sure you use a special fly-replay response so Fly can pass the request to the primary instance:

But the second problem is a little harder. Here's how we visualize that:

This solution

This module comes with several utilities to help you work around these issues. Specifically, it allows you an easy way to add a special cookie to the client that identifies the client's "transaction number" which is then used by read replicas to compare to their local transaction number and force the client to wait until replication has finished if necessary (with a timeout).

Here's how we visualize that:

The even better (experimental) proxy solution

At the time of this writing, LiteFS just released experimental support for a proxy server that will handle much of this stuff for you. You simply configure the proxy server in your litefs.yml and then you don't need to bother with the tx number cookie or ensuring primary on non-get requests at all. The litefs-js module is still useful for one-off situations where you're making mutations in GET requests for example, or if you need to know more about the running instances of your application, but for most of the use cases, you can get away with using the proxy. Learn more about using the proxy from this PR.

Installation

This module is distributed via npm which is bundled with node and should be installed as one of your project's dependencies:

npm install --save litefs-js

Unless you plan on using lower-level utilities, you'll need to set two environment variables on your server:

• LITEFS_DIR - the directory where the .primary file is stored. This should be what you set your fuse.dir config to in the litefs.yml config.
• DATABASE_FILENAME - the filename of your sqlite database. This is used to determine the location of the -pos file which LiteFS uses to track the transaction number.
• INTERNAL_PORT - the port set in the fly.toml (can be different from PORT if you're using the litefs proxy). This is useful for the getInternalInstanceDomain utility.

Usage

The best way to use this is with the latest version of LiteFS which supports the proxy server:

# litefs.yml
# ...
proxy:
  # matches the internal_port in fly.toml
  addr: ':${INTERNAL_PORT}'
  target: 'localhost:${PORT}'
  db: '${DATABASE_FILENAME}'
# ...

From there all you really need litefs-js for is when you run your database migrations. For example:

// start.js
const fs = require('fs')
const { spawn } = require('child_process')
const os = require('os')
const path = require('path')
const { getInstanceInfo } = require('litefs-js')

async function go() {
	const { currentInstance, currentIsPrimary, primaryInstance } =
		await getInstanceInfo()

	if (currentIsPrimary) {
		console.log(
			`Instance (${currentInstance}) in ${process.env.FLY_REGION} is primary. Deploying migrations.`,
		)
		await exec('npx prisma migrate deploy')
	} else {
		console.log(
			`Instance (${currentInstance}) in ${process.env.FLY_REGION} is not primary (the primary instance is ${primaryInstance}). Skipping migrations.`,
		)
	}

	console.log('Starting app...')
	await exec('node ./build')
}
go()

async function exec(command) {
	const child = spawn(command, { shell: true, stdio: 'inherit' })
	await new Promise((res, rej) => {
		child.on('exit', code => {
			if (code === 0) {
				res()
			} else {
				rej()
			}
		})
	})
}

The only other thing you need to worry about is if you ever perform a mutation as a part of a GET request. The proxy will handle forwarding non-GET requests for you automatically, but if you need to write to a non-primary in a GET, you'll need to use some of the utilities to ensure the server replays the request to the primary.

Lower Level Usage

Integrating this with your existing server requires integration in two places:

1. Setting the transaction number cookie on the client after mutations have finished
2. Waiting for replication to finish before responding to requests

Low-level utilities are exposed, but higher level utilities are also available for express and remix.

Additionally, any routes that trigger database mutations will need to ensure they are running on the primary instance, which is where ensurePrimary comes in handy.

Express

import express from 'express'
import {
	getSetTxNumberMiddleware,
	getTransactionalConsistencyMiddleware,
	getEnsurePrimaryMiddleware,
} from 'litefs-js/express'

const app = express()
// this should appear before any middleware that mutates the database
app.use(getEnsurePrimaryMiddleware())

// this should appear before any middleware that retrieves something from the database
app.use(getTransactionalConsistencyMiddleware())

// ... other middleware that might mutate the database here
app.use(getSetTxNumberMiddleware())

// ... middleware that send the response here

The tricky bit here is that often your middleware that mutates the database is also responsible for sending the responses, so you may need to use a lower-level utility like setTxCookie to set the cookie after mutations.

Remix

Until we have proper middleware support in Remix, you'll have to use the express or other lower-level utilities. You cannot currently use this module with the built-in Remix server because there's no way to force the server to wait before calling your loaders. Normally, you just need to use getTransactionalConsistencyMiddleware in express, and then you can use appendTxNumberCookie as shown below.

Of course, instead of using express with getTransactionalConsistencyMiddleware, you could use await handleTransactionalConsistency(request) to the top of every loader if you like:

// app/root.tsx (and app/routes/*.tsx... and every other loader in your app)
export function loader({ request }: DataFunctionArgs) {
	await handleTransactionalConsistency(request)
	// ... your loader code here
}

The same thing applies to getEnsurePrimaryMiddleware as well. If you need or like, you can use await ensurePrimary() in every action call or any loaders that mutate the database (of which, there should be few because you should avoid mutations in loaders).

We're umm... really looking forward to Remix middleware...

The appendTxNumberCookie utility should be used in the entry.server.ts file in both the default export (normally people call this handleDocumentRequest or handleRequest) and the handleDataRequest export.

// app/entry.server.ts
import { appendTxNumberCookie } from 'litefs-js/remix'

export default async function handleRequest(
	request: Request,
	responseStatusCode: number,
	responseHeaders: Headers,
	remixContext: EntryContext,
) {
	// Most of the time, all mutations are finished by now, but just make sure
	// you're finished with all mutations before this line:
	await appendTxNumberCookie(request, responseHeaders)
	// send the response
}

export async function handleDataRequest(
	response: Response,
	{ request }: Parameters<HandleDataRequestFunction>[1],
) {
	// Most of the time, all mutations are finished by now, but just make sure
	// you're finished with all mutations before this line:
	await appendTxNumberCookie(request, response.headers)
	return response
}

Other

There are several other lower-level utilities that you can use. They allow for more customization and are documented via jsdoc. Utilities you may find helpful:

• ensurePrimary - Use this to ensure that the server that's handling the request is the primary server. This is useful if you know you need to do a mutation for that request.
• getInstanceInfo - get the currentInstance and primaryInstance hostnames from the filesystem.
• waitForUpToDateTxNumber - wait for the local transaction number to match the one you give it
• getTxNumber - read the transaction number from the filesystem.
• getTxSetCookieHeader - get the Set-Cookie header value for the transaction number
• checkCookieForTransactionalConsistency - the logic used to check the transaction number cookie for consistency and wait for replication if necessary.
• getAllInstances - get all the instances of your app currently running
• getInternalInstanceDomain - get the internal domain for the current instance so you can communicate between instances of your app (ensure you've set the INTERNAL_PORT environment variable to what appears in your fly.toml).

How it works

This module uses the special .primary directory in your Fuse filesystem to determine the primary (litefs primary docs), and the -pos file to determine the transaction number (litefs transaction number docs).

When necessary, replay requests are made by responding with a 409 status code and a fly-replay header (docs on dynamic request routing).

Inspiration

This was built to make it much easier for people to take advantage of distributed SQLite with LiteFS on Fly.io. The bulk of the logic was extracted from kentcdodds/kentcdodds.com.

LICENSE

MIT