Skip to content
This repository was archived by the owner on Nov 4, 2024. It is now read-only.

Commit e3bbe95

Browse files
JiayuZzzJiayuZzz
authored
Merge volatile module, store persistent and volatile module in root dir (#349)
Cherry-pick volatile branch, store persistent and volatile module to independent directories. Signed-off-by: Wu, Jiayu <[email protected]>
1 parent d6b630b commit e3bbe95

File tree

330 files changed

+36239
-161
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

330 files changed

+36239
-161
lines changed

‎.github/workflows/ci.yml

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,14 @@ jobs:
99
steps:
1010
- uses: actions/checkout@v2
1111

12-
- name: Check Java codestyle
12+
- name: Check Persistent Java codestyle
1313
run: |
14-
cd java
14+
cd persistent/java
15+
mvn spotless:check
16+
17+
- name: Check Volatile Java codestyle
18+
run: |
19+
cd volatile/java
1520
mvn spotless:check
1621
1722
- name: Get cmake
@@ -36,8 +41,16 @@ jobs:
3641
- name: Init submodules
3742
run: git submodule update --init --recursive
3843

39-
- name: Check codestyle & Build
44+
- name: Check Persistent codestyle & Build
45+
run: |
46+
cd persistent
47+
mkdir -p build && cd build
48+
cmake .. -DCMAKE_BUILD_TYPE=Release -DCHECK_CPP_STYLE=ON -DWITH_JNI=ON
49+
make -j
50+
51+
- name: Check Volatile codestyle & Build
4052
run: |
53+
cd volatile
4154
mkdir -p build && cd build
4255
cmake .. -DCMAKE_BUILD_TYPE=Release -DCHECK_CPP_STYLE=ON -DWITH_JNI=ON
4356
make -j

‎.gitmodules

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1-
[submodule "extern/gtest"]
2-
path = extern/gtest
1+
[submodule "volatile/extern/jemalloc"]
2+
path = volatile/extern/jemalloc
3+
url = https://github.com/jemalloc/jemalloc
4+
[submodule "volatile/extern/numactl"]
5+
path = volatile/extern/numactl
6+
url = https://github.com/numactl/numactl
7+
[submodule "persistent/extern/gtest"]
8+
path = persistent/extern/gtest
9+
url = https://github.com/google/googletest.git
10+
[submodule "volatile/extern/gtest"]
11+
path = volatile/extern/gtest
312
url = https://github.com/google/googletest.git

‎pic/kvdk-qrcode.png renamed to ‎.pic/kvdk-qrcode.png

File renamed without changes.

‎pic/kvdk_logo.png renamed to ‎.pic/kvdk_logo.png

File renamed without changes.

‎README.md

Lines changed: 5 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -1,124 +1,19 @@
11
<div align="center">
2-
<p align="center"> <img src="pic/kvdk_logo.png" height="180px"><br></p>
2+
<p align="center"> <img src=".pic/kvdk_logo.png" height="180px"><br></p>
33
</div>
44

5-
`KVDK` (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for supporting DRAM, Optane persistent memory and CXL memory pool. It also demonstrates several optimization methods for high performance with tiered memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like read-modify-write, checkpoint, etc.
5+
`KVDK` (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for supporting Optane persistent memory, DRAM and CXL memory pool. It also demonstrates several optimization methods for high performance with tiered memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like read-modify-write, checkpoint, etc.
66

7-
## Features
8-
* Rich data types
9-
* string, sorted, hash, list, hash
10-
* Basic KV operations
11-
* get/put/update/delete/scan
12-
* Read-Modify-Write
13-
* Support TTL
14-
* Atomic Batch Write
15-
* Snapshot based Scan
16-
* Consistent Dump & Restore to/from storage
17-
* Consistent Checkpoint
18-
* Transaction
19-
* C/C++/Java APIs
20-
21-
# Limitations
22-
* The maximum supported key-value size is 64KB-4GB.
23-
* No approach to key-value compression.
24-
* Users can't expand the persistent memory space on the fly.
7+
**Notice: The DRAM engine and CXL memory pool are in development, you can checkout to v1.0 for a release version of PMEM based persistent engine.**
258

269
## Getting the Source
2710
```bash
2811
git clone --recurse-submodules https://github.com/pmem/kvdk.git
2912
```
3013

31-
## Building
32-
### Install dependent tools and libraries on Ubuntu 18.04
33-
```bash
34-
sudo apt install make clang-format-9 pkg-config g++ autoconf libtool asciidoctor libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev pandoc libhwloc-dev libgflags-dev libtext-diff-perl bash-completion systemd wget git curl
35-
36-
git clone https://github.com/pmem/ndctl.git
37-
cd ndctl
38-
git checkout v70.1
39-
./autogen.sh
40-
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib
41-
make
42-
sudo make install
43-
44-
git clone https://github.com/pmem/pmdk.git
45-
cd pmdk
46-
git checkout 1.11.1
47-
make
48-
sudo make install
49-
50-
wget https://github.com/Kitware/CMake/releases/download/v3.12.4/cmake-3.12.4.tar.gz
51-
tar vzxf cmake-3.12.4.tar.gz
52-
cd cmake-3.12.4
53-
./bootstrap
54-
make
55-
sudo make install
56-
57-
```
58-
59-
### Install dependent tools and libraries on CentOS 8
60-
```bash
61-
yum config-manager --add-repo /etc/yum.repos.d/CentOS-Linux-PowerTools.repo
62-
yum config-manager --set-enabled PowerTools
63-
64-
yum install -y git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkgconfig glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel gem make cmake libarchive clang-tools-extra hwloc-devel perl-Text-Diff gflags-devel curl
65-
66-
git clone https://github.com/pmem/ndctl.git
67-
cd ndctl
68-
git checkout v70.1
69-
./autogen.sh
70-
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib
71-
make
72-
sudo make install
73-
74-
git clone https://github.com/pmem/pmdk.git
75-
cd pmdk
76-
git checkout 1.11.1
77-
make
78-
sudo make install
79-
80-
wget https://github.com/Kitware/CMake/releases/download/v3.12.4/cmake-3.12.4.tar.gz
81-
tar vzxf cmake-3.12.4.tar.gz
82-
cd cmake-3.12.4
83-
./bootstrap
84-
make
85-
sudo make install
86-
```
87-
88-
### Compile KVDK
89-
```bash
90-
mkdir -p build && cd build
91-
cmake .. -DCMAKE_BUILD_TYPE=Release -DCHECK_CPP_STYLE=ON && make -j
92-
```
93-
94-
### How to test it on a system without PMEM
95-
```bash
96-
# set the correct path for pmdk library
97-
export LD_LIBRARY_PATH=/usr/local/lib64
98-
99-
# setup a tmpfs for test
100-
mkdir /mnt/pmem0
101-
mount -t tmpfs -o size=2G tmpfs /mnt/pmem0
102-
103-
# force the program work on non-pmem directory
104-
export PMEM_IS_PMEM_FORCE=1
105-
106-
cd kvdk/build/examples
107-
# Note: this requires CPU supporting AVX512
108-
./cpp_api_tutorial
109-
110-
```
111-
112-
## Benchmarks
113-
[Here](./doc/benchmark.md) are the examples of how to benchmark the performance of KVDK on your systems.
114-
115-
## Documentations
116-
117-
### User Guide
118-
119-
Please refer to [User guide](./doc/user_doc.md) for API introductions of KVDK.
14+
## User guide
12015

121-
### Architecture
16+
KVDK of different storage type are independently modules, please refer to [persistent](./persistent/README.md) and [volatile](./volatile/README.md) for building, test and docs.
12217

12318
# Support
12419
Welcome to join the wechat group or slack channel for KVDK tech discussion.

‎persistent/.gitignore

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# IDE related
2+
.idea/
3+
cmake-build-*
4+
.vscode/
5+
.vs/
6+
7+
# Prerequisites
8+
*.d
9+
10+
# Compiled Object files
11+
*.slo
12+
*.lo
13+
*.o
14+
*.obj
15+
16+
# Precompiled Headers
17+
*.gch
18+
*.pch
19+
20+
# Compiled Dynamic libraries
21+
*.so
22+
*.dylib
23+
*.dll
24+
25+
# Fortran module files
26+
*.mod
27+
*.smod
28+
29+
# Compiled Static libraries
30+
*.lai
31+
*.la
32+
*.a
33+
*.lib
34+
35+
# Executables
36+
*.exe
37+
*.out
38+
*.app
39+
40+
build/
41+
.cache/
42+
43+
scripts/result*
44+
scripts/__pycache__
45+
46+
java/out
47+
java/target
48+
java/benchmark/target
49+
java/examples/target
50+
java/test-libs
51+
java/*.log
52+
java/include/io_pmem_*.h
53+
java/src/main/resources

‎CMakeLists.txt renamed to ‎persistent/CMakeLists.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ add_cppstyle(src ${CMAKE_CURRENT_SOURCE_DIR}/engine/*.c*
156156
${CMAKE_CURRENT_SOURCE_DIR}/engine/utils/*.h*
157157
${CMAKE_CURRENT_SOURCE_DIR}/examples/tutorial/*.c*
158158
${CMAKE_CURRENT_SOURCE_DIR}/benchmark/*.c*
159-
${CMAKE_CURRENT_SOURCE_DIR}/include/kvdk/*.h*
159+
${CMAKE_CURRENT_SOURCE_DIR}/include/kvdk/persistent/*.h*
160160
${CMAKE_CURRENT_SOURCE_DIR}/tests/*.c*
161161
${CMAKE_CURRENT_SOURCE_DIR}/tests/*.h*)
162162

‎persistent/README.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
<div align="center">
2+
<p align="center"> <img src="../.pic/kvdk_logo.png" height="180px"><br></p>
3+
</div>
4+
5+
This is the persistent module of `kvdk`, based on Intel Optane Persistent Memory.
6+
7+
## Features
8+
* Rich data types
9+
* string, sorted, hash, list, hash
10+
* Basic KV operations
11+
* get/put/update/delete/scan
12+
* Read-Modify-Write
13+
* Support TTL
14+
* Atomic Batch Write
15+
* Snapshot based Scan
16+
* Consistent Dump & Restore to/from storage
17+
* Consistent Checkpoint
18+
* Transaction
19+
* C/C++/Java APIs
20+
21+
# Limitations
22+
* The maximum supported key-value size is 64KB-4GB.
23+
* No approach to key-value compression.
24+
* Users can't expand the persistent memory space on the fly.
25+
26+
## Getting the Source
27+
```bash
28+
git clone --recurse-submodules https://github.com/pmem/kvdk.git
29+
```
30+
31+
## Building
32+
### Install dependent tools and libraries on Ubuntu 18.04
33+
```bash
34+
sudo apt install make clang-format-9 pkg-config g++ autoconf libtool asciidoctor libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev pandoc libhwloc-dev libgflags-dev libtext-diff-perl bash-completion systemd wget git curl
35+
36+
git clone https://github.com/pmem/ndctl.git
37+
cd ndctl
38+
git checkout v70.1
39+
./autogen.sh
40+
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib
41+
make
42+
sudo make install
43+
44+
git clone https://github.com/pmem/pmdk.git
45+
cd pmdk
46+
git checkout 1.11.1
47+
make
48+
sudo make install
49+
50+
wget https://github.com/Kitware/CMake/releases/download/v3.12.4/cmake-3.12.4.tar.gz
51+
tar vzxf cmake-3.12.4.tar.gz
52+
cd cmake-3.12.4
53+
./bootstrap
54+
make
55+
sudo make install
56+
57+
```
58+
59+
### Install dependent tools and libraries on CentOS 8
60+
```bash
61+
yum config-manager --add-repo /etc/yum.repos.d/CentOS-Linux-PowerTools.repo
62+
yum config-manager --set-enabled PowerTools
63+
64+
yum install -y git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkgconfig glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel gem make cmake libarchive clang-tools-extra hwloc-devel perl-Text-Diff gflags-devel curl
65+
66+
git clone https://github.com/pmem/ndctl.git
67+
cd ndctl
68+
git checkout v70.1
69+
./autogen.sh
70+
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib
71+
make
72+
sudo make install
73+
74+
git clone https://github.com/pmem/pmdk.git
75+
cd pmdk
76+
git checkout 1.11.1
77+
make
78+
sudo make install
79+
80+
wget https://github.com/Kitware/CMake/releases/download/v3.12.4/cmake-3.12.4.tar.gz
81+
tar vzxf cmake-3.12.4.tar.gz
82+
cd cmake-3.12.4
83+
./bootstrap
84+
make
85+
sudo make install
86+
```
87+
88+
### Compile KVDK
89+
```bash
90+
mkdir -p build && cd build
91+
cmake .. -DCMAKE_BUILD_TYPE=Release -DCHECK_CPP_STYLE=ON && make -j
92+
```
93+
94+
### How to test it on a system without PMEM
95+
```bash
96+
# set the correct path for pmdk library
97+
export LD_LIBRARY_PATH=/usr/local/lib64
98+
99+
# setup a tmpfs for test
100+
mkdir /mnt/pmem0
101+
mount -t tmpfs -o size=2G tmpfs /mnt/pmem0
102+
103+
# force the program work on non-pmem directory
104+
export PMEM_IS_PMEM_FORCE=1
105+
106+
cd kvdk/build/examples
107+
# Note: this requires CPU supporting AVX512
108+
./cpp_api_tutorial
109+
110+
```
111+
112+
## Benchmarks
113+
[Here](./doc/benchmark.md) are the examples of how to benchmark the performance of KVDK on your systems.
114+
115+
## Documentations
116+
117+
### User Guide
118+
119+
Please refer to [User guide](./doc/user_doc.md) for API introductions of KVDK.
120+
121+
### Architecture
122+
123+
# Support
124+
Welcome to join the wechat group or slack channel for KVDK tech discussion.
125+
- [Wechat](https://github.com/pmem/kvdk/issues/143)
126+
- [Slack Channel](https://join.slack.com/t/kvdksupportcommunity/shared_invite/zt-12b66vg1c-4FGb~Ri4w8K2_msau6v86Q)

‎benchmark/bench.cpp renamed to ‎persistent/benchmark/bench.cpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@
1111
#include <thread>
1212

1313
#include "generator.hpp"
14-
#include "kvdk/engine.hpp"
15-
#include "kvdk/types.hpp"
14+
#include "kvdk/persistent/engine.hpp"
15+
#include "kvdk/persistent/types.hpp"
1616

1717
using namespace google;
1818
using namespace KVDK_NAMESPACE;

‎benchmark/generator.hpp renamed to ‎persistent/benchmark/generator.hpp

File renamed without changes.

0 commit comments

Comments
 (0)