• Stars
    star
    456
  • Rank 95,985 (Top 2 %)
  • Language
    TypeScript
  • License
    MIT License
  • Created about 1 year ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A self-hosted backend for Deno KV

denokv

A self-hosted backend for Deno KV, the JavaScript first key-value database:

  • Seamlessly integrated JavaScript APIs
  • ACID transactions
  • Multiple consistency levels for optimal performance for every usecase

Diagram showing how a denokv setup looks Diagram showing how a denokv setup looks

Deno KV can be used with the built-in single instance database in the CLI, useful for testing and development, with a hosted and scalable backend on Deno Deploy, or with this self-hostable Deno KV backend.

To run denokv, just run:

docker run -it --init -p 4512:4512 -v ./data:/data ghcr.io/denoland/denokv --sqlite-path /data/denokv.sqlite serve --access-token <random-token>

Then run your Deno program and specify the access token in the DENO_KV_ACCESS_TOKEN environment variable:

const kv = await Deno.openKv("http://localhost:4512");

The self-hosted denokv backend is built on the same robust SQLite backend as the built-in single instance database in the CLI. It is designed to be run on a VPS or Kubernetes cluster statefully, with Deno processes connecting via the network using KV Connect.

The standalone denokv binary is designed to handle thousands of concurrent requests, from hundreds of different Deno processes. It is built on top of the robust SQLite database, and uses non-blocking IO to ensure excellent performance even in the face of hundreds of concurrent connections.

Just like the Deno CLI, denokv is MIT licensed, free and open source.

Read more in the announcement of self-hosted Deno KV.

When should I use this?

If you need more than a single Deno process to access the same KV database, and you are ok with running a server, keeping denokv updated, handling backups, and performing regular maintenance, then this is for you.

You can use a hosted KV database on Deno Deploy if you don't want to self-host and manage a denokv server.

If you are just need a backend for local development or testing, you can use the Deno KV backend built into the Deno CLI. You can open a temporary in memory KV database with Deno.openKv(":memory:") or a persistent database by specifying a path like Deno.openKv("./my-database.sqlite").

How to run

Docker on a VPS

Ensure that you are running on a service that supports persistent storage, and does not perform auto-scaling beyond a single instance. This means you can not run denokv on Google Cloud Run or AWS Lambda.

Install Docker on your VPS and create a directory for the database to store data in.

$ mkdir -p /data

Then run the denokv Docker image, mounting the /data directory as a volume and specifying a random access token.

docker run -it --init -p 4512:4512 -v ./data:/data ghcr.io/denoland/denokv --sqlite-path /data/denokv.sqlite serve --access-token <random-token>

You can now access the database from your Deno programs by specifying the access token in the DENO_KV_ACCESS_TOKEN environment variable, and the host and port of your VPS in the URL passed to Deno.openKv.

You should additionally add a HTTPS terminating proxy or loadbalancer in front of denokv to ensure that all communication happens over TLS. Not using TLS can pose a significant security risk. The HTTP protocol used by Deno KV is compatible with any HTTP proxy, such as caddy, nginx, or a loadbalancer.

Fly.io

You can easily host denokv on https://fly.io.

Note: Fly.io is a paid service. You will need to add a credit card to your account to use it.

Sign up to Fly.io and install the flyctl CLI.

Sign into the CLI with flyctl auth login.

Create a new app with flyctl apps create.

Create a fly.toml file with the following contents. Make sure to replace the <your-app-name> and <region> placeholders with your app name and the region you want to deploy to.

app = "<your-app-name>"
primary_region = "<region>"

[build]
  image = "ghcr.io/denoland/denokv:latest"

[http_service]
  internal_port = 4512
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0

[env]
  DENO_KV_SQLITE_PATH="/data/denokv.sqlite3"
  # access token is set via `flyctl secrets set`

[mounts]
  destination = "/data"
  source = "denokv_data"

Run flyctl volumes create denokv_data to create a volume to store the database in.

Run flyctl secrets set DENO_KV_ACCESS_TOKEN=<random-token> to set the access token. Make sure to replace <random-token> with a random string. Keep this token secret, and don't share it with anyone. You will need this token to connect to your database from Deno.

Run flyctl deploy to deploy your app.

You can now access the database from your Deno programs by specifying the access token in the DENO_KV_ACCESS_TOKEN environment variable, and the URL provided by flyctl deploy in the URL passed to Deno.openKv.

Be aware that with this configuration, your database can scale to 0 instances when not in use. This means that the first request to your database after a period of inactivity will be slow, as the database needs to be started. You can avoid this by setting min_machines_running to 1, and setting auto_stop_machines = false.

Install binary

You can download a prebuilt binary from the releases page and place it in your PATH.

You can also compile from source by running cargo install denokv --locked.

How to connect

Deno

To connect to a denokv server from Deno, use the Deno.openKv API:

const kv = await Deno.openKv("http://localhost:4512");

Make sure to specify your access token in the DENO_KV_ACCESS_TOKEN environment variable.

Advanced setup

Running as a replica of a hosted KV database

denokv has a mode for running as a replica of a KV database hosted on Deno Deploy through the S3 backup feature.

To run as a replica:

docker run -it --init -p 4512:4512 -v ./data:/data \
  -e AWS_ACCESS_KEY_ID="<aws-access-key-id>" \
  -e AWS_SECRET_ACCESS_KEY="<aws-secret-access-key>" \
  -e AWS_REGION="<aws-region>" \
  ghcr.io/denoland/denokv --sqlite-path /data/denokv.sqlite serve \
  --access-token <random-token> --sync-from-s3 --s3-bucket your-bucket --s3-prefix some-prefix/6aea9765-2b1e-41c7-8904-0bdcd70b21d3/

To sync the local database from S3, without updating the snapshot:

denokv --sqlite-path /data/denokv.sqlite pitr sync --s3-bucket your-bucket --s3-prefix some-prefix/6aea9765-2b1e-41c7-8904-0bdcd70b21d3/

To list recoverable points:

denokv --sqlite-path /data/denokv.sqlite pitr list

To checkout the snapshot at a specific recoverable point:

denokv --sqlite-path /data/denokv.sqlite pitr checkout 0100000002c0f4c10000

Continuous backup using LiteFS

TODO

Other things in this repo

This repository contains two crates:

  • denokv_proto (/proto): Shared interfaces backing KV, like definitions of Key, Database, and Value.
  • denokv_sqlite (/sqlite): An implementation of Database backed by SQLite.
  • denokv_remote (/remote): An implementation of Database backed by a remote KV database, acessible via the KV Connect protocol.

These crates are used by the deno_kv crate in the Deno repository to provide a JavaScript API for interacting with Deno KV.

The Deno KV Connect protocol used for communication between Deno and a remote KV database is defined in /proto/kv-connect.md.

More Repositories

1

deno

A modern runtime for JavaScript and TypeScript.
Rust
93,907
star
2

fresh

The next-gen web framework.
TypeScript
12,234
star
3

rusty_v8

Rust bindings for the V8 JavaScript engine
Rust
3,094
star
4

deno_std

Deno standard library
TypeScript
2,705
star
5

deno_lint

Blazing fast linter for JavaScript and TypeScript written in Rust
Rust
1,514
star
6

vscode_deno

Visual Studio Code plugin for Deno
TypeScript
1,468
star
7

dnt

Deno to npm package build tool.
Rust
1,211
star
8

saaskit

A modern SaaS template built on Fresh.
TypeScript
1,195
star
9

dotland

[Archived] deno.land website
TypeScript
958
star
10

deno_install

Deno Binary Installer
PowerShell
945
star
11

deno_docker

Latest dockerfiles and images for Deno - alpine, centos, debian, ubuntu
Dockerfile
876
star
12

fastwebsockets

A fast RFC6455 WebSocket implementation
Rust
837
star
13

deno-lambda

A deno runtime for AWS Lambda. Deploy deno via docker, SAM, serverless, or bundle it yourself.
TypeScript
836
star
14

deno_blog

Minimal boilerplate blogging.
TypeScript
459
star
15

deployctl

Command line tool for Deno Deploy
TypeScript
331
star
16

showcase_chat

TypeScript
302
star
17

roll-your-own-javascript-runtime

Rust
298
star
18

merch

The Deno shop!
TypeScript
283
star
19

deno_bindgen

Write high-level Deno FFI libraries in Rust.
Rust
275
star
20

wasmbuild

Build tool to use Rust code in Deno and the browser.
TypeScript
263
star
21

deno_core

The core engine at the heart of Deno
Rust
262
star
22

deno_doc

Documentation generator for Deno
Rust
251
star
23

setup-deno

Set up your GitHub Actions workflow with a specific version of Deno
JavaScript
248
star
24

meet-me

A calendly clone in Deno and hosted on Deno Deploy
TypeScript
245
star
25

deno_kv_oauth

High-level OAuth 2.0 powered by Deno KV.
TypeScript
245
star
26

eszip

A compact file format to losslessly serialize an ECMAScript module graph into a single file
Rust
220
star
27

deno-gfm

Server-side GitHub Flavored Markdown rendering for Deno
TypeScript
220
star
28

deno_emit

Transpile and bundle JavaScript and TypeScript under Deno and Deno Deploy
TypeScript
217
star
29

webgpu-examples

TypeScript
214
star
30

doc_website

Archived. New version at https://github.com/denoland/docland
TypeScript
195
star
31

manual

Deprecated - find these resources on docs.deno.com instead
TypeScript
162
star
32

node_shims

npm packages providing shims for the Deno namespace and other globals. Useful for running Deno-first programs on Node.
TypeScript
149
star
33

deno_ast

Source text parsing, lexing, and AST related functionality for Deno
Rust
148
star
34

denobyexample

[Archived] Deno by example - short examples showcasing how to use Deno. Now the examples have been moved to https://github.com/denoland/deno-docs / https://docs.deno.com/examples
TypeScript
146
star
35

fresh_charts

A server-side-rendered charting library for Fresh
TypeScript
136
star
36

deploy_examples

Examples for Deno Deploy
TypeScript
125
star
37

docland

The documentation generation website for Deno
TypeScript
120
star
38

deno_graph

The module graph logic for Deno CLI
Rust
111
star
39

deno_task_shell

Cross-platform shell for deno task.
Rust
104
star
40

deno_registry2

The backend for the deno.land/x service
TypeScript
92
star
41

showcase_todo

Collaborative todo-list app built with Deno and Fresh
TypeScript
82
star
42

deno_third_party

TypeScript
78
star
43

monch

Inspired by nom, but specifically for strings.
Rust
77
star
44

examples

A simple todo app using Deno and React.
TypeScript
77
star
45

tic-tac-toe

A global, real-time multiplayer TicTacToe game for Deno πŸ¦•
TypeScript
75
star
46

deploy_feedback

For reporting issues with Deno Deploy
74
star
47

deno-astro-adapter

A Deno adapter for running Astro applications on the Deno runtime.
TypeScript
67
star
48

pixelpage

Pixel page is an r/place style shared pixel art canvas πŸŽ¨πŸ¦•
TypeScript
64
star
49

v8

floating patches for rusty_v8
TypeScript
60
star
50

rust-urlpattern

Rust implementation of the `URLPattern` web API
Rust
58
star
51

docs

Deno documentation, examples and API Reference. Powered by Lume.
TypeScript
57
star
52

fresh-wordpress-themes

https://wp-blog-example.deno.dev/ https://wp-sweets-co.deno.dev/
TypeScript
53
star
53

apiland

The API server for deno.land
TypeScript
52
star
54

deno-astro-template

Template repo for an Astro site, preconfigured to run with Deno and Deno Deploy
Astro
49
star
55

sui

Embed custom RO data into precompiled executables
Rust
47
star
56

wanted_modules

Is there a missing deno module that is preventing you from building something? Let us know here.
46
star
57

cargo_gn

Cargo GN integration
Rust
40
star
58

wasmbuild_example

Example of using wasmbuild.
JavaScript
38
star
59

deno_cache_dir

Deno CLI's module cache
Rust
38
star
60

ga

Utilities for server side processing of Google Analytics in Deno CLI and Deploy
TypeScript
38
star
61

serde_v8

Moved to https://github.com/denoland/deno
Rust
36
star
62

import_map

An implementation of WICG Import Maps specification
Rust
30
star
63

flaky_test

atttribute macro for running a flaky test multiple times
Rust
30
star
64

chromium_build

Deno floats patches to //build here (they will be sent upstream eventually)
Python
29
star
65

fresh-blog-example

An example for building a blog with Fresh.
TypeScript
28
star
66

libffi-rs

Fork of libffi-rs which corrects autotools usage
C
22
star
67

deno_npm

npm registry client and dependency resolver used in the Deno CLI.
Rust
22
star
68

deno-sveltekit-template

A starter template for running SvelteKit on Deno Deploy
JavaScript
22
star
69

chatspace

Real-time, collaborative GPT frontend built with Deno KV
TypeScript
22
star
70

doc_components

A set of components for rendering deno_doc doc nodes
TypeScript
21
star
71

rustls-tokio-stream

AsyncRead/AsyncWrite interface for rustls-on-Tokio
Rust
21
star
72

subhosting_ide_starter

Basic starter app for a browser-based IDE using the Deno Subhosting API
JavaScript
21
star
73

deno-vue-example

An example of using Vue with Deno.
Vue
20
star
74

terraform-deploy-provider

Terraform provider for Deno Deploy
Go
20
star
75

deploy_lume_example

An example demonstrating using static site generators on Deno Deploy
TypeScript
20
star
76

monaco-nextjs-demo

A demo Next.js app that features an in-browser IDE built with Monaco.
JavaScript
20
star
77

image-resizing-api

A simple image resizing API written in Deno.
TypeScript
19
star
78

kv_api

WORK IN PROGRESS: Attach a flexible REST API to your Deno web app to manage data in Deno KV
TypeScript
19
star
79

deno-kv-hackathon

Rules, details, and place to submit your project for the Deno KV hackathon.
18
star
80

automation

Automation scripts used for denoland org repos
TypeScript
17
star
81

react18-with-deno

A starter app and tutorial with React18 and Deno.
TypeScript
17
star
82

chromium_buildtools

forked from chromium to use git submodules instead of gclient
Python
17
star
83

benchmark_data

TypeScript
16
star
84

terraform-provider-deno

Terraform provider for hosted Deno APIs
Go
15
star
85

deno-nuxt-template

A template repo for a Nuxt project preconfigured for Deno Deploy
TypeScript
14
star
86

fresh-deno-kv-oauth-demo

Fresh + Deno KV OAuth demo
TypeScript
14
star
87

x-to-jsr

TypeScript
14
star
88

serverless-coldstart-benchmarks

Configuration, benchmarking scripts, and raw data for serverless cold start benchmarks.
Jupyter Notebook
14
star
89

icu

For floating patches on top of https://chromium.googlesource.com/chromium/deps/icu.git
C++
12
star
90

fresh_template

template repository for a Fresh project
12
star
91

deno-vite-plugin

Vite plugin to enable Deno resolution inside vite.
TypeScript
12
star
92

experimental-deno-specifiers-example

TypeScript
12
star
93

deno_lockfile

Rust
12
star
94

fresh-auth-example

TypeScript
11
star
95

deno_config

Rust
11
star
96

deno_media_type

Media type used in Deno.
Rust
11
star
97

notebook

TypeScript
10
star
98

nextgen-install

HCL
10
star
99

v8_valueserializer

A Rust implementation of V8's ValueSerializer and ValueDeserializer
Rust
10
star
100

website_feedback

For reporting issues & suggestions for deno.com and deno.land
9
star