• Stars
    star
    380
  • Rank 109,028 (Top 3 %)
  • Language
    TypeScript
  • License
    MIT License
  • Created 7 months ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A self-hosted backend for Deno KV

denokv

A self-hosted backend for Deno KV, the JavaScript first key-value database:

  • Seamlessly integrated JavaScript APIs
  • ACID transactions
  • Multiple consistency levels for optimal performance for every usecase

Diagram showing how a denokv setup looks Diagram showing how a denokv setup looks

Deno KV can be used with the built-in single instance database in the CLI, useful for testing and development, with a hosted and scalable backend on Deno Deploy, or with this self-hostable Deno KV backend.

To run denokv, just run:

docker run -it --init -p 4512:4512 -v ./data:/data ghcr.io/denoland/denokv --sqlite-path /data/denokv.sqlite serve --access-token <random-token>

Then run your Deno program and specify the access token in the DENO_KV_ACCESS_TOKEN environment variable:

const kv = await Deno.openKv("http://localhost:4512");

The self-hosted denokv backend is built on the same robust SQLite backend as the built-in single instance database in the CLI. It is designed to be run on a VPS or Kubernetes cluster statefully, with Deno processes connecting via the network using KV Connect.

The standalone denokv binary is designed to handle thousands of concurrent requests, from hundreds of different Deno processes. It is built on top of the robust SQLite database, and uses non-blocking IO to ensure excellent performance even in the face of hundreds of concurrent connections.

Just like the Deno CLI, denokv is MIT licensed, free and open source.

Read more in the announcement of self-hosted Deno KV.

When should I use this?

If you need more than a single Deno process to access the same KV database, and you are ok with running a server, keeping denokv updated, handling backups, and performing regular maintenance, then this is for you.

You can use a hosted KV database on Deno Deploy if you don't want to self-host and manage a denokv server.

If you are just need a backend for local development or testing, you can use the Deno KV backend built into the Deno CLI. You can open a temporary in memory KV database with Deno.openKv(":memory:") or a persistent database by specifying a path like Deno.openKv("./my-database.sqlite").

How to run

Docker on a VPS

Ensure that you are running on a service that supports persistent storage, and does not perform auto-scaling beyond a single instance. This means you can not run denokv on Google Cloud Run or AWS Lambda.

Install Docker on your VPS and create a directory for the database to store data in.

$ mkdir -p /data

Then run the denokv Docker image, mounting the /data directory as a volume and specifying a random access token.

docker run -it --init -p 4512:4512 -v ./data:/data ghcr.io/denoland/denokv --sqlite-path /data/denokv.sqlite serve --access-token <random-token>

You can now access the database from your Deno programs by specifying the access token in the DENO_KV_ACCESS_TOKEN environment variable, and the host and port of your VPS in the URL passed to Deno.openKv.

You should additionally add a HTTPS terminating proxy or loadbalancer in front of denokv to ensure that all communication happens over TLS. Not using TLS can pose a significant security risk. The HTTP protocol used by Deno KV is compatible with any HTTP proxy, such as caddy, nginx, or a loadbalancer.

Fly.io

You can easily host denokv on https://fly.io.

Note: Fly.io is a paid service. You will need to add a credit card to your account to use it.

Sign up to Fly.io and install the flyctl CLI.

Sign into the CLI with flyctl auth login.

Create a new app with flyctl apps create.

Create a fly.toml file with the following contents. Make sure to replace the <your-app-name> and <region> placeholders with your app name and the region you want to deploy to.

app = "<your-app-name>"
primary_region = "<region>"

[build]
  image = "ghcr.io/denoland/denokv:latest"

[http_service]
  internal_port = 4512
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0

[env]
  DENO_KV_SQLITE_PATH="/data/denokv.sqlite3"
  # access token is set via `flyctl secrets set`

[mounts]
  destination = "/data"
  source = "denokv_data"

Run flyctl volumes create denokv_data to create a volume to store the database in.

Run flyctl secrets set DENO_KV_ACCESS_TOKEN=<random-token> to set the access token. Make sure to replace <random-token> with a random string. Keep this token secret, and don't share it with anyone. You will need this token to connect to your database from Deno.

Run flyctl deploy to deploy your app.

You can now access the database from your Deno programs by specifying the access token in the DENO_KV_ACCESS_TOKEN environment variable, and the URL provided by flyctl deploy in the URL passed to Deno.openKv.

Be aware that with this configuration, your database can scale to 0 instances when not in use. This means that the first request to your database after a period of inactivity will be slow, as the database needs to be started. You can avoid this by setting min_machines_running to 1, and setting auto_stop_machines = false.

Install binary

You can download a prebuilt binary from the releases page and place it in your PATH.

You can also compile from source by running cargo install denokv --locked.

How to connect

Deno

To connect to a denokv server from Deno, use the Deno.openKv API:

const kv = await Deno.openKv("http://localhost:4512");

Make sure to specify your access token in the DENO_KV_ACCESS_TOKEN environment variable.

Advanced setup

Running as a replica of a hosted KV database

denokv has a mode for running as a replica of a KV database hosted on Deno Deploy through the S3 backup feature.

To run as a replica:

docker run -it --init -p 4512:4512 -v ./data:/data \
  -e AWS_ACCESS_KEY_ID="<aws-access-key-id>" \
  -e AWS_SECRET_ACCESS_KEY="<aws-secret-access-key>" \
  -e AWS_REGION="<aws-region>" \
  ghcr.io/denoland/denokv --sqlite-path /data/denokv.sqlite serve \
  --access-token <random-token> --sync-from-s3 --s3-bucket your-bucket --s3-prefix some-prefix/6aea9765-2b1e-41c7-8904-0bdcd70b21d3/

To sync the local database from S3, without updating the snapshot:

denokv --sqlite-path /data/denokv.sqlite pitr sync --s3-bucket your-bucket --s3-prefix some-prefix/6aea9765-2b1e-41c7-8904-0bdcd70b21d3/

To list recoverable points:

denokv --sqlite-path /data/denokv.sqlite pitr list

To checkout the snapshot at a specific recoverable point:

denokv --sqlite-path /data/denokv.sqlite pitr checkout 0100000002c0f4c10000

Continuous backup using LiteFS

TODO

Other things in this repo

This repository contains two crates:

  • denokv_proto (/proto): Shared interfaces backing KV, like definitions of Key, Database, and Value.
  • denokv_sqlite (/sqlite): An implementation of Database backed by SQLite.
  • denokv_remote (/remote): An implementation of Database backed by a remote KV database, acessible via the KV Connect protocol.

These crates are used by the deno_kv crate in the Deno repository to provide a JavaScript API for interacting with Deno KV.

The Deno KV Connect protocol used for communication between Deno and a remote KV database is defined in /proto/kv-connect.md.

More Repositories

1

deno

A modern runtime for JavaScript and TypeScript.
Rust
92,633
star
2

fresh

The next-gen web framework.
TypeScript
11,819
star
3

rusty_v8

Rust bindings for the V8 JavaScript engine
Rust
2,976
star
4

deno_std

Deno standard library
TypeScript
2,705
star
5

deno_lint

Blazing fast linter for JavaScript and TypeScript written in Rust
Rust
1,499
star
6

vscode_deno

Visual Studio Code plugin for Deno
TypeScript
1,453
star
7

dnt

Deno to npm package build tool.
Rust
1,147
star
8

saaskit

A modern SaaS template built on Fresh.
TypeScript
1,071
star
9

dotland

[Archived] deno.land website
TypeScript
966
star
10

deno_install

Deno Binary Installer
PowerShell
945
star
11

deno_docker

Latest dockerfiles and images for Deno - alpine, centos, debian, ubuntu
Dockerfile
838
star
12

deno-lambda

A deno runtime for AWS Lambda. Deploy deno via docker, SAM, serverless, or bundle it yourself.
TypeScript
836
star
13

fastwebsockets

A fast RFC6455 WebSocket implementation
Rust
741
star
14

deno_blog

Minimal boilerplate blogging.
TypeScript
435
star
15

deployctl

Command line tool for Deno Deploy
TypeScript
321
star
16

showcase_chat

TypeScript
303
star
17

merch

The Deno shop!
TypeScript
282
star
18

roll-your-own-javascript-runtime

Rust
261
star
19

deno_bindgen

Write high-level Deno FFI libraries in Rust.
Rust
257
star
20

wasmbuild

Build tool to use Rust code in Deno and the browser.
TypeScript
251
star
21

deno_doc

Documentation generator for Deno
Rust
247
star
22

meet-me

A calendly clone in Deno and hosted on Deno Deploy
TypeScript
243
star
23

setup-deno

Set up your GitHub Actions workflow with a specific version of Deno
JavaScript
230
star
24

deno_kv_oauth

High-level OAuth 2.0 powered by Deno KV.
TypeScript
221
star
25

deno-gfm

Server-side GitHub Flavored Markdown rendering for Deno
TypeScript
209
star
26

eszip

A compact file format to losslessly serialize an ECMAScript module graph into a single file
Rust
208
star
27

webgpu-examples

TypeScript
201
star
28

doc_website

Archived. New version at https://github.com/denoland/docland
TypeScript
195
star
29

deno_core

The core engine at the heart of Deno
Rust
179
star
30

deno_emit

Transpile and bundle JavaScript and TypeScript under Deno and Deno Deploy
TypeScript
177
star
31

manual

Deprecated - find these resources on docs.deno.com instead
TypeScript
163
star
32

denobyexample

Deno by example - short examples showcasing how to use Deno
TypeScript
142
star
33

node_shims

npm packages providing shims for the Deno namespace and other globals. Useful for running Deno-first programs on Node.
TypeScript
140
star
34

deno_ast

Source text parsing, lexing, and AST related functionality for Deno
Rust
134
star
35

fresh_charts

A server-side-rendered charting library for Fresh
TypeScript
126
star
36

deploy_examples

Examples for Deno Deploy
TypeScript
124
star
37

docland

The documentation generation website for Deno
TypeScript
120
star
38

deno_task_shell

Cross-platform shell for deno task.
Rust
100
star
39

deno_graph

The module graph logic for Deno CLI
Rust
94
star
40

deno_registry2

The backend for the deno.land/x service
TypeScript
93
star
41

deno_third_party

TypeScript
78
star
42

monch

Inspired by nom, but specifically for strings.
Rust
76
star
43

examples

A simple todo app using Deno and React.
TypeScript
75
star
44

showcase_todo

Collaborative todo-list app built with Deno and Fresh
TypeScript
74
star
45

tic-tac-toe

A global, real-time multiplayer TicTacToe game for Deno 🦕
TypeScript
73
star
46

deploy_feedback

For reporting issues with Deno Deploy
72
star
47

v8

floating patches for rusty_v8
TypeScript
57
star
48

pixelpage

Pixel page is an r/place style shared pixel art canvas 🎨🦕
TypeScript
54
star
49

rust-urlpattern

Rust implementation of the `URLPattern` web API
Rust
54
star
50

apiland

The API server for deno.land
TypeScript
53
star
51

fresh-wordpress-themes

https://wp-blog-example.deno.dev/ https://wp-sweets-co.deno.dev/
TypeScript
51
star
52

deno-astro-adapter

A Deno adapter for running Astro applications on the Deno runtime.
TypeScript
49
star
53

wanted_modules

Is there a missing deno module that is preventing you from building something? Let us know here.
46
star
54

cargo_gn

Cargo GN integration
Rust
40
star
55

deno-astro-template

Template repo for an Astro site, preconfigured to run with Deno and Deno Deploy
Astro
39
star
56

wasmbuild_example

Example of using wasmbuild.
JavaScript
37
star
57

deno-docs

Docusaurus site for a unified Deno docs experience
MDX
37
star
58

deno_cache_dir

Deno CLI's module cache
JavaScript
37
star
59

ga

Utilities for server side processing of Google Analytics in Deno CLI and Deploy
TypeScript
37
star
60

serde_v8

Moved to https://github.com/denoland/deno
Rust
36
star
61

chromium_build

Deno floats patches to //build here (they will be sent upstream eventually)
Python
29
star
62

import_map

An implementation of WICG Import Maps specification
Rust
29
star
63

fresh-blog-example

An example for building a blog with Fresh.
TypeScript
25
star
64

flaky_test

atttribute macro for running a flaky test multiple times
Rust
25
star
65

libffi-rs

Fork of libffi-rs which corrects autotools usage
C
22
star
66

chatspace

Real-time, collaborative GPT frontend built with Deno KV
TypeScript
22
star
67

deno_npm

npm registry client and dependency resolver used in the Deno CLI.
Rust
21
star
68

doc_components

A set of components for rendering deno_doc doc nodes
TypeScript
21
star
69

terraform-deploy-provider

Terraform provider for Deno Deploy
Go
21
star
70

deno-vue-example

An example of using Vue with Deno.
Vue
20
star
71

deploy_lume_example

An example demonstrating using static site generators on Deno Deploy
TypeScript
20
star
72

deno-kv-hackathon

Rules, details, and place to submit your project for the Deno KV hackathon.
18
star
73

rustls-tokio-stream

AsyncRead/AsyncWrite interface for rustls-on-Tokio
Rust
18
star
74

kv_api

WORK IN PROGRESS: Attach a flexible REST API to your Deno web app to manage data in Deno KV
TypeScript
18
star
75

subhosting_ide_starter

Basic starter app for a browser-based IDE using the Deno Subhosting API
JavaScript
18
star
76

chromium_buildtools

forked from chromium to use git submodules instead of gclient
Python
17
star
77

image-resizing-api

A simple image resizing API written in Deno.
TypeScript
16
star
78

benchmark_data

TypeScript
16
star
79

react18-with-deno

A starter app and tutorial with React18 and Deno.
TypeScript
16
star
80

deno-sveltekit-template

A starter template for running SvelteKit on Deno Deploy
JavaScript
15
star
81

monaco-nextjs-demo

A demo Next.js app that features an in-browser IDE built with Monaco.
JavaScript
15
star
82

automation

Automation scripts used for denoland org repos
TypeScript
15
star
83

fresh-deno-kv-oauth-demo

Fresh + Deno KV OAuth demo
TypeScript
14
star
84

terraform-provider-deno

Terraform provider for hosted Deno APIs
Go
14
star
85

fresh_template

template repository for a Fresh project
12
star
86

experimental-deno-specifiers-example

TypeScript
12
star
87

deno_lockfile

Rust
12
star
88

icu

For floating patches on top of https://chromium.googlesource.com/chromium/deps/icu.git
C++
11
star
89

deno-nuxt-template

A template repo for a Nuxt project preconfigured for Deno Deploy
TypeScript
10
star
90

notebook

TypeScript
10
star
91

deno_media_type

Media type used in Deno.
Rust
10
star
92

deno_config

Rust
9
star
93

eszip_viewer

TypeScript
8
star
94

website_feedback

For reporting issues & suggestions for deno.com and deno.land
8
star
95

fresh-auth-example

TypeScript
8
star
96

v8_valueserializer

A Rust implementation of V8's ValueSerializer and ValueDeserializer
Rust
8
star
97

ga4

A GA4 measurement protocol module for Deno.
TypeScript
7
star
98

oak_template

A template of REST API app using Oak framework
TypeScript
7
star
99

deno_semver

Semver used in Deno's CLI.
Rust
7
star
100

comparing-git-deploys-to-edge

Comparing git deployments to the edge.
TypeScript
7
star