SDFS
Symlinked Distributed File System
SDFS use Symlink method to distribute file accross node (multi data center)
Current important feature that missing in SDFS is :
- Replication (You can use Glusterfs in storage node)
- Disordered Process/Async NFS (non-queue), probably your data can lost when your node (that use <92% disk usage) is crash
- Permission Validation
- Option & Customization
- Documentation
- Web Monitoring (Usage, Health Check, etc)
- etc
Usage
SDFS Has 2 route API, /api/connect
for combine new node and /api/symlink
to tell the master what and where the file should i distribute accross available node, yes you can use FileSystem watcher, but what if you have thousand of directory ?, yeah more RAM (currently master-server only save symlink file)
Tested & Used in Ubuntu 16.04
Master Server
- create a
/data
directory - install
sudo apt-get install nfs-common
(make sure /etc/fstab are exist) - Build master-server and run as sudo
Storage Server
- create a directory to receive data from Master-Server
- install
sudo apt-get install nfs-kernel-server
(make sure /etc/exports are exist) - build storage-server and run
./storage-server connect 6.6.6.28 /path/to/receive/data storagenodeid
(6.6.6.28 is master ip address)
Example Usage
- Your application receive a file from user, and save that file in
/mnt/storage/application/image/image.jpg
- After saving that file, tell master-server to distribute
/mnt/storage/application/image/image.jpg
(use /api/symlink) - SDFS Will choose available storage-server and move the real file to storage-server then SDFS will create a symlink from storage-server to the first file path / saved file path
- You can easily use
connect
command to add new storage-server
POST master-server.ip:2219/api/symlink
# form url encoded, call this after you file has been saved
url: /path/to/saved/file.mp4
or
- use File Mover
Development Log
- 13 May 2019
Some code has not been tested (yet !)