readysettech/readyset

ReadySet is a SQL caching engine designed to help developers enhance the performance and scalability of their existing database-backed applications.

• What is ReadySet?
• Tutorial
• Documentation
• Join the Community
• Development
• License

What is ReadySet?

ReadySet is a lightweight SQL caching engine that sits between your application and Postgres or MySQL database and turns even the most complex SQL reads into lightning-fast lookups. Unlike other caching solutions, ReadySet keeps the cache up-to-date automatically and requires no changes to your application code.

Based on years of dataflow research at MIT¹, ReadySet stores the results of queries in-memory and automatically keeps these results up-to-date as the underlying database changes. ReadySet can do this automatically because it listens to your database's replication stream.

This means:

• No extra code to keep your cache and database in sync
• No extra code to evict stale records
• No TTLs to set - your cache is as up-to-date as your replication lag

ReadySet is wire-compatible with MySQL and Postgres.

Tutorial

These instructions show you how to run ReadySet against a new Postgres or MySQL database using Docker. You can follow along in the docs for the ReadySet Quickstart.

Before you begin

• Install and start Docker for your OS.
• Install the psql shell or the mysql shell.

Step 1. Start the database

Create a Docker container and start the database inside it.

For Postgres:

docker run -d \
--name=postgres \
--publish=5432:5432 \
-e POSTGRES_PASSWORD=readyset \
-e POSTGRES_DB=testdb \
postgres:14 \
-c wal_level=logical

For MySQL:

docker run -d \
--name=mysql \
--publish=3306:3306 \
-e MYSQL_ROOT_PASSWORD=readyset \
-e MYSQL_DATABASE=testdb \
mysql

Step 2. Load sample data

Download a sample data file and load it into the database via the SQL shell.

For Postgres:

curl -O https://raw.githubusercontent.com/readysettech/docs/main/docs/assets/quickstart-data-postgres.sql

PGPASSWORD=readyset psql \
--host=127.0.0.1 \
--port=5432 \
--username=postgres \
--dbname=testdb

\i quickstart-data-postgres.sql

\q

For MySQL:

curl -O https://raw.githubusercontent.com/readysettech/docs/main/docs/assets/quickstart-data-mysql.sql

mysql \
--host=127.0.0.1 \
--port=3306 \
--user=root \
--password=readyset \
--database=testdb

source quickstart-data-mysql.sql;

\q

Step 3. Start ReadySet

Create a Docker container and start ReadySet inside it, connecting ReadySet to the database via the connection string in --upstream-db-url:

For Postgres:

docker run -d \
--name=readyset \
--publish=5433:5433 \
--platform=linux/amd64 \
--volume='readyset:/state' \
--pull=always \
-e DEPLOYMENT_ENV=quickstart_github \
public.ecr.aws/readyset/readyset:beta-2023-04-19 \
--standalone \
--deployment='github-postgres' \
--database-type=postgresql \
--upstream-db-url=postgresql://postgres:[email protected]:5432/testdb \
--address=0.0.0.0:5433 \
--username='postgres' \
--password='readyset' \
--db-dir='/state'

For MySQL:

docker run -d \
--name=readyset \
--publish=3307:3307 \
--platform=linux/amd64 \
--volume='readyset:/state' \
--pull=always \
-e DEPLOYMENT_ENV=quickstart_github \
public.ecr.aws/readyset/readyset:beta-2023-04-19 \
--standalone \
--deployment='github-mysql' \
--database-type=mysql \
--upstream-db-url=mysql://root:[email protected]:3306/testdb \
--address=0.0.0.0:3307 \
--username='root' \
--password='readyset' \
--db-dir='/state'

Next steps

• Connect an existing database

  The tutorial above runs ReadySet against a sample database. To see the power of ReadySet against your own database, swap the upstream-db-url argument to point at an existing database for testing.

• Connect your application

  Once you have a ReadySet instance up and running, the next step is to connect your application by swapping out your database connection string to point to ReadySet instead. The specifics of how to do this vary by database client library, ORM, and programming language. See Connect an App for examples.

  Note: By default, ReadySet will proxy all queries to the database, so changing your app to connect to ReadySet should not impact performance. You will explicitly tell ReadySet which queries to cache.

• Cache queries

  Once ReadySet is proxying queries, connect a database SQL shell to ReadySet and use the custom SHOW PROXIED QUERIES SQL command to view the queries that ReadySet has proxied to your upstream database and identify which queries are supported by ReadySet. Then use the custom CREATE CACHE SQL command to cache supported queries.

  Note: To successfully cache the results of a query, ReadySet must support the SQL features and syntax in the query. For more details, see SQL Support.

• Tear down

  When you are done testing, stop and remove the Docker resources:

  docker rm -f readyset postgres mysql \
  && docker volume rm readyset

• Additional deployment options In addition to Docker, you can deploy ReadySet using binaries. For Kubernetes environments, check out how to deploy ReadySet to Kubernetes.

FAQs

Q: How does ReadySet work under the hood?

A: The heart of ReadySet is a query engine based on partially-stateful, streaming data flow. To learn more about how it works, see our docs.

Q: How does ReadySet keep cached state up to date?

A: ReadySet receives updates about data changes from your backing database via binlog replication and uses those updates to automatically update its internal state.

Q: Do I have to send all of my database traffic to ReadySet?

A: You can if you want to, but it’s not required. You can manually route a subset of your traffic to ReadySet (as you would with a traditional database read replica), or you can send all of it to ReadySet. It's important to note that not all of the queries sent to ReadySet need to be cached– you have fine-grained control over what is cached vs. what is proxied.

Q: Does ReadySet automatically determine which queries will be cached?

A: No - by default, ReadySet proxies queries to your upstream database so you can profile your application. Only after you run CREATE CACHE will ReadySet begin caching a query.

Documentation

For more information, check out our full docs site here.

Join the Community

For questions or support, join us on the ReadySet Community Discord to chat with our team.

Development

This section is for developers who want to build ReadySet from source as they work on the ReadySet codebase.

Install prerequisites

1. Install ReadySet dependencies.

   macOS with homebrew:

   brew install lz4 [email protected]

   Add the following to your cargo config to make it discover lz4:

   [env]
   LIBRARY_PATH = "/opt/homebrew/lib"

   Ubuntu:

   sudo apt update && sudo apt install -y build-essential libssl-dev pkg-config llvm clang liblz4-dev
   sudo apt-get -y install cmake

   Arch:

   sudo pacman -S base-devel clang lz4

   CentOS/Amazon Linux:

   sudo yum -y update
   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install clang lz4-devel openssl-devel

   Nix:

   nix-shell

2. Install Rust via rustup.rs.

   ReadySet is written entirely in Rust.

3. Install Docker via Get Docker and Docker Compose via Install Docker Compose.

   ReadySet runs alongside a backing Postgres or MySQL database and, when run in distributed fashion, uses Consul for leader election, failure detection, and internal cluster state management. You'll use Docker Compose to create and manage these resources for local development.

Build and run ReadySet

1. Clone the repo using git and navigate into it:

   git clone https://github.com/readysettech/readyset.git
   cd readyset

2. Start a backing database:

   docker-compose up -d

   This starts both Postgres and MySQL in containers, although you will run ReadySet against only one at a time. If you don't want to run both databases, edit docker-compose.yml and comment out the mysql or postgres fields.

3. Compile and run ReadySet, replacing <deployment name> with a unique identifier for the deployment.

   Run against Postgres:

   cargo run --bin readyset --release -- --standalone --database-type=postgresql --upstream-db-url=postgresql://postgres:[email protected]:5432/testdb --username=postgres --password=readyset --address=0.0.0.0:5433 --deployment=<deployment name> --prometheus-metrics --query-log --query-log-ad-hoc

   Run against MySQL:

   cargo run --bin readyset --release -- --standalone --database-type=mysql --upstream-db-url=mysql://root:[email protected]:3306/testdb --username=root --password=readyset --address=0.0.0.0:3307 --deployment=<deployment name> --prometheus-metrics --query-log --query-log-ad-hoc

   This runs the ReadySet Server and Adapter as a single process, a simple, standard way to run ReadySet that is also the easiest approach when developing. For production deployments, however, it is possible to run the Server and Adapter as separate processes. See the scale out deployment pattern in the docs.

   For details about ReadySet options, see the CLI docs (coming soon).

4. With ReadySet up and running, you can now connect the Postgres or MySQL shell:

   Postgres:

   PGPASSWORD=readyset psql
   --host=127.0.0.1 \
   --port=5433 \
   --username=postgres \
   --dbname=testdb

   MySQL:

   mysql \
   --host=127.0.0.1 \
   --port=3307 \
   --user=root \
   --password=readyset \
   --database=testdb

Testing

To run tests for the project, run the following command:

cargo test --skip integration_serial

Note:

• Certain tests cannot be run in parallel with others, and these tests are typically in files postfixed with _serial. Running the entire set of tests for a package (e.g., cargo test -p readyset-server) may fail if serial tests are included.

• Running tests may require increasing file descriptor limits. You can do so by running ulimit -Sn 65535.

Performance

When running ReadySet in a performance-critical setting, make sure you compile with the --release flag.

Note: This repository contains a snapshot of the current state of the ReadySet internal repository. Our team is currently working on getting our live codebase, including all development history, ready to be hosted on GitHub, at which point we'll force push to this repository.

License

ReadySet is licensed under the BSL 1.1 license, converting to the open-source Apache 2.0 license after 4 years. It is free to use on any number of nodes.