• Stars
    star
    5
  • Rank 2,864,054 (Top 57 %)
  • Language
    Go
  • License
    MIT License
  • Created over 5 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

a Connector |> Distributed File System based on Symlinking (that fit author need)

SDFS Go Report Card

Symlinked Distributed File System

Image

SDFS use Symlink method to distribute file accross node (multi data center)

Current important feature that missing in SDFS is :

  1. Replication (You can use Glusterfs in storage node)
  2. Disordered Process/Async NFS (non-queue), probably your data can lost when your node (that use <92% disk usage) is crash
  3. Permission Validation
  4. Option & Customization
  5. Documentation
  6. Web Monitoring (Usage, Health Check, etc)
  7. etc

Usage

SDFS Has 2 route API, /api/connect for combine new node and /api/symlink to tell the master what and where the file should i distribute accross available node, yes you can use FileSystem watcher, but what if you have thousand of directory ?, yeah more RAM (currently master-server only save symlink file)

Tested & Used in Ubuntu 16.04

Master Server

  1. create a /data directory
  2. install sudo apt-get install nfs-common (make sure /etc/fstab are exist)
  3. Build master-server and run as sudo

Storage Server

  1. create a directory to receive data from Master-Server
  2. install sudo apt-get install nfs-kernel-server (make sure /etc/exports are exist)
  3. build storage-server and run ./storage-server connect 6.6.6.28 /path/to/receive/data storagenodeid (6.6.6.28 is master ip address)

Example Usage

  1. Your application receive a file from user, and save that file in /mnt/storage/application/image/image.jpg
  2. After saving that file, tell master-server to distribute /mnt/storage/application/image/image.jpg (use /api/symlink)
  3. SDFS Will choose available storage-server and move the real file to storage-server then SDFS will create a symlink from storage-server to the first file path / saved file path
  4. You can easily use connect command to add new storage-server
POST master-server.ip:2219/api/symlink
# form url encoded, call this after you file has been saved
url: /path/to/saved/file.mp4

or

  1. use File Mover

Development Log

  • 13 May 2019

Some code has not been tested (yet !)

More Repositories

1

batch-image-generator

Generate dynamic image content based on a template image and a CSV file.
Svelte
74
star
2

pc-auth-notifier

Someone tried to unlock your device
Go
60
star
3

file.io

🐈 💻 Simple File Share Service where the file is completely deleted after download.
CSS
44
star
4

docker-script

Embed your whole project files into single existing Dockerfile file with respect to .dockerignore & GZIP compression support.
Go
22
star
5

temporar

S3-Backed Anonymous Temporary File Sharing
CSS
20
star
6

file-server

Multi-source file server [💾 | 💽 | 💿] 🧐
Go
19
star
7

ficha

An super secret chat app for any body who need privacy ✉
Crystal
18
star
8

kemal-mysql-blog

Crystal, Kemal, MySQL, Blog
CSS
11
star
9

telelog

Embed to your app or tail a file, send the log to your telegram account &/ group
Go
9
star
10

sveltekit-telegram-captcha-bot

JavaScript
9
star
11

elastis

Tool for Export / Dump / Import / Copy Elastic/Open Search indexes data
Go
8
star
12

Fido

E2E Solution for Distributed Cold-Data Storage
Rust
8
star
13

golang-ktp-aws-textract

KTP Extractor based on AWS Textract
Go
7
star
14

CoBat-RAT-Malware

RAT Malware for school project, educational purpose
Java
6
star
15

gstransfer

Google Cloud Storage upload relay/gateway with destination path rewrite support & zero disk allocation ⚡
Go
6
star
16

Data-Wilayah.js

Indonesian territory data in JavaScript
Python
6
star
17

portable-http-server

Simple HTTP Server with Dir Listing, Explore & File Upload support
Go
6
star
18

elixir-sse-example

Server Sent Events example in Elixir + Phoenix
Elixir
5
star
19

install-latest-go-linux

Shell
5
star
20

GoTral

Golang Centralized Configuration management (LIBRARY)
Go
4
star
21

chuck-norris-holy-quotes

An Chuck Norris Holy Quotes written in Crystal
Crystal
4
star
22

elixir-two-factor-auth

Library for generating QR-Code, Random Secret and Verify Time Based Password
Elixir
4
star
23

node-examp-app

Node JS, SQLite, IO Exam app
CSS
4
star
24

golang-elasticsearch-crud

Go / Golang elasticsearch CRUD example
Go
4
star
25

web-mirror

Make any web accessible from your own host / domain
Go
4
star
26

cloudflare-system-internship-task-2020

🔥 Basic option, IPv4 & IPv6 support, Save output as HTML, change percentile
Go
4
star
27

wasmc.cr

A WebAssembly Compiler for Crystal. WIP : Work In Progress !
Crystal
4
star
28

phoenix-auth-example

Phoenix authentication example with ETS, Multi Session & Single Session example
Elixir
4
star
29

go-screenshot

Go package for taking screenshot from Linux+, Windows-, Mac-
Go
4
star
30

terminus-red

Terminus Red GTK 2/3 Theme
CSS
4
star
31

lazy-mongo

Insert data to mongo from text plain or file
Crystal
3
star
32

crystal-mysql-crud-example

Crystal MySQL CRUD Example
Crystal
3
star
33

go-mongo-web-boilerplate

Go net/http & mgo Web Boilerplate
Go
3
star
34

minikv

In-memory KV database with expiration (TTL) support, can be dumped and loaded (for non RT persistence).
Go
3
star
35

GoMiss

Auto `go get -u` from missing package message
Go
3
star
36

heineken-music-player

Listen to any music by given audio file or youtube url
CSS
3
star
37

Danilla

Secret Management written in Elixir based on Mnesia & ETS
CSS
3
star
38

GoTral-Server

Server for GoTral, Encrypted Golang Centralized Configuration management
Go
3
star
39

job-test

Push your test here !
JavaScript
3
star
40

grab

Collection of numerous GRAB's API in Go
Go
2
star
41

June

a Read Only Selenium for Web Scrapping
Crystal
2
star
42

md5-cpp

MD5 -> WASM
C
2
star
43

golang-code-sample

Go Code Sample plugin for vscode
2
star
44

barachy

a Static File Loader, Handler and Backuper
Crystal
2
star
45

zenly

Server Side and Portable NVR, Powered by gphoto2
2
star
46

systemd-generator

systemd service generator
2
star
47

lemmehand

Lemmehand Proxy Server, Written in Crystal
Crystal
2
star
48

.vimrc

My tiny simple .vimrc File
Vim Script
2
star
49

flutter-centrifugo-push-notification

Demo of flutter push notification using centrifugo as messaging server
Dart
2
star
50

telegram-profile-info-changer

change your telegram name and profile picture every pyrogram.Filters passed
Python
2
star
51

go-gin-recaptcha-example

Go Golang Gin reCAPTCHA Example (v2 & v3)
HTML
2
star
52

EaW-GaS-Stack

Introduction to EaW (Elixir as Web) & GaS (Golang as Service) Stack
2
star
53

clipmonkey

Learning Go Programming Language
Go
2
star
54

slicey

Spicy slice in Go, collection of slice modifier
Go
2
star
55

selenium-session-manager

Selenium session manager
Python
2
star
56

Kaskus-Thread-Remover

Python
2
star
57

s2c

SASS To CSS Converter [GUI]
Crystal
2
star
58

go-web-boilerplate

Golang Web Boilerplate with Auth, User Management, reCaptcha, Page Generator
Go
2
star
59

go-press

Simple CMS written in Go + sqlite - compatible with HUGO Template
1
star
60

GoTral-node

GoTral library for node.js
1
star
61

go-auto-deploy

Auto Deployment for Go
1
star
62

telegram-captcha-bot

1
star
63

git-push-telegram-log

1
star
64

nim-totp

Time-based One-time Password library in NIM
1
star
65

persisten-cookiejar

because @juju persisten-cookiejar doesn't works for me
1
star
66

go-module-test

1
star
67

mongodumper

Automate your mongo database backup
1
star
68

goimagemerge

Go
1
star
69

spjalla

1
star
70

WinSS

1
star
71

10K-Addons

10K is browser addons/extension that listen XHR request to collecting api value by specified routes
1
star
72

tiktod-downloader

1
star
73

youtube-channel-search

1
star
74

upwork-rss-telegram-notification

1
star
75

Shika

1
star
76

igigle

Omegle / Ome.tv written in Go
1
star
77

gin-recaptcha

Gin reCAPTCHA Middleware, Auto detect, verify and abort c.PostForm("g-recaptcha-response")
Go
1
star
78

go-image-combiner

1
star
79

goprox

Highly Intuitive Proxy Server
1
star
80

pumpkinfs

1
star
81

webdriver-protocol

Go implementation of https://www.w3.org/TR/webdriver
1
star
82

id-phone-number-parser

Indonesian phone number parser
HTML
1
star
83

TheFATWoman

TFW is screen-spyware that curently only run in Windows & Linux
1
star
84

google-image

1
star
85

keybd_converter

1
star
86

hello-world

1
star
87

pm2.go

pm2 in Go
1
star
88

json-to-anything

Convert JSON to "Anything"
1
star
89

process-manager

process-manager let you start, maintain and kill the process.
1
star
90

qupac

QUIC + msgpack rules!!, darn fast RPC based on HTTP/3 & msgpack
1
star
91

monopoly

1
star
92

GoTral-python

1
star
93

random-hitter-server

Random Hitter Server
1
star
94

fiber-recaptcha

Fiber reCAPTCHA Middleware, Auto detect, verify and abort c.FormValue("g-recaptcha-response")
Go
1
star
95

SnakesAndLadders

1
star
96

github-pm2-auto-deployer

1
star
97

gocar

1
star
98

instagram-roulette

1
star
99

gin-pure-cookie-based-auth

Golang Gin Gonic Pure Cookie Based Auth From Scratch (Video)
1
star
100

pony-http-client-example

Pony
1
star