• Stars
    star
    113
  • Rank 310,115 (Top 7 %)
  • Language
    C
  • License
    BSD 3-Clause "New...
  • Created about 6 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A version of Redis that uses persistent memory
Badge Status
Current Version Current Version

What is Redis?

Redis is often referred as a data structures server. What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.

Data structures implemented into Redis have a few special properties:

  • Redis cares to store them on disk, even if they are always served and modified into the server memory. This means that Redis is fast, but that is also non-volatile.
  • Implementation of data structures stress on memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modeled using an high level programming language.
  • Redis offers a number of features that are natural to find in a database, like replication, tunable levels of durability, cluster, high availability.

Another good example is to think of Redis as a more complex version of memcached, where the operations are not just SETs and GETs, but operations to work with complex data types like Lists, Sets, ordered data structures, and so forth.

If you want to know more, this is a list of selected starting points:

What is Pmem-Redis?

Pmem-Redis is one redis version that support Intel DCPMM(Data Center Persistent Memory) based on open source redis-4.0.0. It benefits the redis's performance by taking advantage of DCPMM competitive performance and persistency.

Basically Pmem-Redis covers many aspects that related to DCPMM usage:

  • Five typical data structures optimization including: String, List, Hash, Set, Zset.
  • DCPMM copy-on-write
  • Redis LRU for DCPMM
  • Redis defragmentation support for DCPMM
  • Pointer-based redis AOF
  • Persistent ring buffer

Get code

  1. clone the code:
git clone https://github.com/pmem/pmem-redis
  1. PMDK is one submodule of this repo, so you have to init this submodule in order to get code
git submodule init
git submodule update

Building Pmem-Redis

Compile Options

Switches Value Descriptions
USE_NVM yes/no DCPMM enable Switch. W/O this option, will compile opensource redis that does not support DCPMM.
AEP_COW yes/no DCPMM Copy-On-Write Switch. W/O this option, the BG save and replication will not support
SUPPORT_PBA yes/no Pointer Based Aof support Switch. W/O this option, PBA is not support, Same AOF mechanism with open source redis.
USE_AOFGUARD yes/no Write Turbo with DCPMM option switch. W/O this option, the AOF log write to the SSD by the page cache directly.

How to compile

Prerequisite

  • Install autoconf, automake, libtool, numactl-devel and tcl on system

If you want to build the Pmem-Redis for DCPMM, please install following packages as well: libndctl-dev, libdaxctl-dev and libnuma-dev

Compile example:

If you want to build the original opensource redis-4.0.0, run command:

make

If you want to build the Pmem-Redis for DCPMM, run command:

make USE_NVM=yes

If you need enable AEP_COW, SUPPORT_PBA or USE_AOFGUARD, you need to enable USE_NVM compile option.

Fixing build problems with dependencies or cached build options

Pmem-Redis has some dependencies which are included into the deps directory. make does not automatically rebuild dependencies even if something in the source code of dependencies changes.

When you update the source code with git pull or when code inside the dependencies tree is modified in any other way, make sure to use the following command in order to really clean everything and rebuild from scratch:

make distclean

This will clean: jemalloc, lua, hiredis, linenoise, pmdk, memkind, jemallocat, aofguard.

Also if you force certain build options like 32bit target, no C compiler optimizations (for debugging purposes), and other similar build time options, those options are cached indefinitely until you issue a make distclean command.

Running Pmem-Redis

New supported server command:

server command argument describe
--nvm-maxcapacity Integer (G) Set the maximum DCPMM capacity of the redis instance
--nvm-dir Like: /mnt/pmem0 Path of DCPMM DAX file system
--nvm-threshold Integer (default 64) Threshold of value size, ifΒ  greater thanΒ  threshold, the value may be stored on the DCPMM
--pointer-based-aof yes/no Enable or disable pointer based AOF
--use-aofguard yes/no Enable or disable persistent ring buffer

Prepare nvm device for redis to run

Create namespace:

ndctl create-namespace -m fsdax -r <regionid>

Then you may see /dev/pmem<id> under /dev folder, for example, we have /dev/pmem0 in our machine.

Format the pmem device:

mkfs.ext4 /dev/pmem0

Mount as dax:

mkdir /mnt/pmem0
mount -o dax /dev/pmem0 /mnt/pmem0

Run pmem-redis:

src/redis-server --nvm-maxcapacity 1 --nvm-dir /mnt/pmem0 --nvm-threshold 64

This command starts a redis server that uses DCPMM device(/mnt/pmem0) with max capacity 1GB, and it moves the values whose length are greater than 64 into DCPMM.

Hints

  1. In our Unit Test, we default read /mnt/pmem0 as pmem device and run test, please modify your pmem device folder to /mnt/pmem0 to meet the requirement.

  2. If you enable --pointer-based-aof in redis server, than pmem-redis will generate one .ag file under /mnt/pmem0 device which link to the AOF file generated on disk. If you need to delete the AOF file, please make sure to delete the .ag file at the same time to avoid AOF load error.

Code contributions

  1. Please fork this repositry to your github account and start deveopment, please refer to redis code style
  2. Pull requests let you tell others about changes you've pushed to a repository on GitHub.
  3. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before the changes are merged into the repository.

Note: When working with pull requests, keep the following in mind:

If you're working in the shared repository model, we recommend that you use a topic branch for your pull request. While you can send pull requests from any branch or commit, with a topic branch you can push follow-up commits if you need to update your proposed changes. When pushing commits to a pull request, don't force push. Force pushing can corrupt your pull request. After initializing a pull request, you'll see a review page that shows a high-level overview of the changes between your branch (the compare branch) and the repository's base branch. You can add a summary of the proposed changes, review the changes made by commits, add labels, milestones, and assignees, and @mention individual contributors or teams. For more information, see "Creating a pull request." pull request page

Once you've created a pull request, you can push commits from your topic branch to add them to your existing pull request. These commits will appear in chronological order within your pull request and the changes will be visible in the "Files changed" tab.

Other contributors can review your proposed changes, add review comments, contribute to the pull request discussion, and even add commits to the pull request.

Enjoy!

More Repositories

1

pmdk

Persistent Memory Development Kit
C
1,336
star
2

syscall_intercept

The system call intercepting library
C
631
star
3

pmemkv

Key/Value Datastore for Persistent Memory
C++
397
star
4

ndctl

A "device memory" enabling project encompassing tools and libraries for CXL, NVDIMMs, DAX, memory tiering and other platform memory device topics.
C
262
star
5

pcj

Persistent Collections for Java
Java
221
star
6

kvdk

Key Value Development Kit
C++
201
star
7

valgrind

Enhanced Valgrind for Persistent Memory
C
107
star
8

libpmemobj-cpp

C++ bindings & containers for libpmemobj
C++
107
star
9

rpma

Remote Persistent Memory Access Library
C
101
star
10

vltrace

Tool tracing syscalls in a fast way using eBPF linux kernel feature
C
98
star
11

llpl

Low Level Persistence Library
Java
97
star
12

pmem-rocksdb

A version of RocksDB that uses persistent memory
C++
90
star
13

linux-examples

Early (now outdated) examples. Use PMDK instead.
C
59
star
14

run_qemu

A script to create bootable OS images, and run qemu with a locally built kernel.
Shell
57
star
15

pmdk-examples

PMDK examples and tutorials
C++
57
star
16

book

Persistent Memory Programming book examples
C
39
star
17

vmemcache

Buffer based LRU cache
C
35
star
18

pmemfile

Userspace implementation of file APIs using persistent memory.
C
34
star
19

pmemkv-java

Java bindings for pmemkv
Java
28
star
20

pmse

Persistent Memory Storage Engine
C++
24
star
21

vmem

Volatile Persistent Memory Allocator
C
23
star
22

pmemkv-bench

Benchmarking tools for pmemkv
C++
22
star
23

pmem.github.io

The pmem.io Website
HTML
17
star
24

pmemkv-python

Python bindings for pmemkv
Python
13
star
25

issues

Old issues repo for PMDK.
13
star
26

pmdk-tests

Extended tests for PMDK libraries and utilities
C++
10
star
27

miniasync

C
10
star
28

docs

Persistent Memory Docbook
9
star
29

pmemstream

C++
9
star
30

libpmemobj-js

JavaScript bindings for libpmemobj
C++
8
star
31

pmemkv-nodejs

NodeJS bindings for pmemkv
JavaScript
8
star
32

pmem-rocksdb-plugin

RocksDB plugin for optimized PMem support
C++
5
star
33

mpi-pmem-ext

MPI Extensions for Persistent Memory
C
4
star
34

kvm-redis

Recipe to run a memtier benchmark on a cluster of KVM-hosted Redis servers
Jinja
4
star
35

pmemkv-jni

Java Native Interface for pmemkv
C++
3
star
36

pmul

PMUL is a Java library that adds PMem programming features to Java’s foreign memory API in JDK 18
Java
2
star
37

acpi-spec-ecr

ACPI Specification ECRs
Makefile
2
star
38

dev-utils-kit

Shell
2
star
39

pmemkv-ruby

Ruby bindings for pmemkv
Ruby
2
star
40

autoflushtest

Basic data integrity test for platforms with flush-on-fail CPU caches
C
1
star
41

pmdk-convert

Conversion tool for pmdk pools
CMake
1
star
42

knowledge-base

Knowledge Base for pmem.io
SCSS
1
star