JunoDB - A secure, consistent and highly available key-value store
What is JunoDB
JunoDB is PayPal's home-grown secure, consistent and highly available key-value store providing low, single digit millisecond, latency at any scale.
Getting Started with the JunoDB Server
Clone the repository from github
git clone https://github.com/paypal/junodb.gitSet BUILDTOP variable
export BUILDTOP=<path_to_junodb_folder>/junodb
cd $BUILDTOPContinue building JunoDB server with
Docker Build
Note : Docker build supported platforms
- Linux (Ubuntu)
- OS X (macOS)
Install Dependencies
Install Docker Engine version 20.10.0+ Check for existing docker version
docker versionInstall Docker if not installed or version is older than 20.10.0
docker/setup.sh
#If you are not added to the docker group, you will have to logout and login in the machine after running docker/setup.shIf the user is not added to docker group, you may add manually. Logout and re-login after this step.
sudo usermod -a -G docker $USERVerify that your username is added to docker group
groups
#or
cat /etc/group | grep dockerBuild JunoDB
#Login to docker hub account
docker login
# Build junodb docker images
#etcd
#clustercfg
#storageserv
#proxy
#junoclient
docker/build.sh Run JunoDB
# Setup junodb network and start junodb services
#etcd
#clustercfg
#storageserv
#proxy
#junoclient
# JunoDB proxy service listens on port
# :5080 TLS and :8080 TCP
docker/start.sh
Shutdown JunoDB services
# This will shutdown junodb services
#etcd
#clustercfg
#storageserv
#proxy
#junoclient
docker/shutdown.sh
Manually Run JunoDB services
#This can be done instead of ./start.sh to start up the docker services
cd $BUILDTOP/docker/manifest
# To run junodb services in --detach mode (recommended)
docker compose up -d
# Juno proxy service listens on port
# :5080 TLS and :8080 TCP
#To view the running containers
docker ps
# To stop junodb services
docker compose downGenerate Secrets for Dev
NOTE: secrets for TLS and Encryption can be generated for dev/testing.
sh $BUILDTOP/docker/manifest/config/secrets/gensecrets.sh
## generated secrets
# server.crt/server.pem - certificate/key for junodb proxy for TLS
# ca.crt - CA cert
# keystore.toml - sample keystore fileValidate JunoDB
Login to docker client and check connection with proxy
docker exec -it junoclient bash -c 'nc -vz proxy 5080'You can also test the junodb server by running junocli and junoload
JunoCLI
The following commands log in to the docker client and run the ./junocli command directly. The proxy ip is aliased as "proxy"
- CREATE
docker exec -it junoclient bash -c '/opt/juno/junocli create -s proxy:8080 -c config.toml -ns test_ns test_key test_value'- GET
docker exec -it junoclient bash -c '/opt/juno/junocli get -s proxy:8080 -c config.toml -ns test_ns test_key'- UPDATE
docker exec -it junoclient bash -c '/opt/juno/junocli update -s proxy:8080 -c config.toml -ns test_ns test_key test_value_updated'- DESTROY
docker exec -it junoclient bash -c '/opt/juno/junocli destroy -s proxy:8080 -c config.toml -ns test_ns test_key'More about junocli here
Junoload
The following command logs in to the docker client and runs the ./junoload command directly. Junoload can be used for benchmarking juno server. The proxy ip is aliased as "proxy"
docker exec -it junoclient bash -c '/opt/juno/junoload -s proxy:5080 -ssl -c config.toml -o 1'More about junoload here
Manual Build
Note : Manual build supported platforms
- Linux (Ubuntu 20.04)
The following sections explain the process for manually building the JunoDB server without Docker. These instructions are based on an Ubuntu 20.04.5 system
Install Dependencies
Install OpenSSL 1.0.2g+
sudo apt install opensslInstall multilog
sudo apt install daemontoolsInstall dependencies for rocksdb
sudo apt-get install build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev -yInstall Python
#install python
sudo apt-get install python3.8
#set soft link
cd /usr/bin
sudo ln -s python3.8 pythonBuild JunoDB
binary_build/build.shRun JunoDB
export JUNO_BUILD_DIR=$BUILDTOP/release-binary/code-build
script/deploy.shValidate JunoDB
#Validate if deploy was successful by checking if the proxy (junoserv), storage (junostorageserv), and etcd (junoclusterserv) processes are running
ps -eaf | grep juno
#There should be 41 processes running
#5 for junoclusterserv (3 logs, 1 etcdsvr.py, 1 etcdsvr_exe)
#20 for junostorageserv (6 logs, 1 manager, 12 workers, 1 monitor)
#16 for junoserv (6 logs, 1 manager, 8 workers, 1 monitor)Test out the server using junocli and junoload command
See instructions for junocli here
See instructions for junoload here
Run functional tests
#Assuming user is in $BUILDTOP folder
test/functest/configsetup.sh
cd test/functest
$BUILDTOP/release-binary/tool/go/bin/go test -v -config=config.tomlRun unit tests
#Assuming user is in $BUILDTOP folder
cd test/unittest
$BUILDTOP/release-binary/tool/go/bin/go test -vShutdown JunoDB Services
#Assuming user is in $BUILDTOP folder
script/deploy.sh stop