• Stars
    star
    2,938
  • Rank 14,829 (Top 0.4 %)
  • Language
    Rust
  • License
    MIT License
  • Created over 7 years ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Mirror of https://gitlab.redox-os.org/redox-os/tfs

The TFS icon.

TFS: Next-generation file system

TFS is a modular, fast, and feature rich next-gen file system, employing modern techniques for high performance, high space efficiency, and high scalability.

TFS was created out of the need for a modern file system for Redox OS, as a replacement for ZFS, which proved to be slow to implement because of its monolithic design.

TFS is inspired by the ideas behind ZFS, but at the same time it aims to be modular and easier to implement.

TFS is not related to the file system of the same name by terminalcloud.

While many components are complete, TFS itself is not ready for use.

MIT/X11 permissive license.

GitHub Stars

Design goals

TFS is designed with the following goals in mind:

Concurrent
TFS contains very few locks and aims to be as suitable for multithreaded systems as possible. It makes use of multiple truly concurrent structures to manage the data, and scales linearly by the number of cores. This is perhaps the most important feature of TFS.
Asynchronous
TFS is asynchronous: operations can happen independently; writes and reads from the disk need not block.
Full-disk compression
TFS is the first file system to incorporate complete full-disk compression through a scheme we call RACC (random-access cluster compression). This means that every cluster is compressed only affecting performance slightly. It is estimated that you get 60-120% more usable space.
Revision history
TFS stores a revision history of every file without imposing extra overhead. This means that you can revert any file into an earlier version, backing up the system automatically and without imposed overhead from copying.
Data integrity
TFS, like ZFS, stores full checksums of the file (not just metadata), and on top of that, it is done in the parent block. That means that almost all data corruption will be detected upon read.
Copy-on-write semantics
Similarly to Btrfs and ZFS, TFS uses CoW semantics, meaning that no cluster is ever overwritten directly, but instead it is copied and written to a new cluster.
O(1) recursive copies
Like some other file systems, TFS can do recursive copies in constant time, but there is an unique addition: TFS doesn't copy even after it is mutated. How? It maintains segments of the file individually, such that only the updated segment needs copying.
Guaranteed atomicity
The system will never enter an inconsistent state (unless there is hardware failure), meaning that unexpected power-off won't ever damage the system.
Improved caching
TFS puts a lot of effort into caching the disk to speed up disk accesses. It uses machine learning to learn patterns and predict future uses to reduce the number of cache misses. TFS also compresses the in-memory cache, reducing the amount of memory needed.
Better file monitoring
CoW is very suitable for high-performance, scalable file monitoring, but unfortunately only few file systems incorporate that. TFS is one of those.
All memory safe
TFS uses only components written in Rust. As such, memory unsafety is only possible in code marked unsafe, which is checked extra carefully.
Full coverage testing
TFS aims to be full coverage with respect to testing. This gives relatively strong guarantees on correctness by instantly revealing large classes of bugs.
SSD friendly
TFS tries to avoid the write limitation in SSD by repositioning dead sectors.
Improved garbage collection
TFS uses Bloom filters for space-efficient and fast garbage collection. TFS allows the FS garbage collector to run in the background without blocking the rest of the file system.

FAQ

Why do you use SPECK as the default cipher?
SPECK is a relatively young cipher, yet it has been subject to a lot of (ineffective) cryptanalysis, so it is relatively secure. It has really good performance and a simple implementation. Portability is an important part of the TFS design, and truly portable AES implementations without side-channel attacks is harder than many think (particularly, there are issues with SubBytes in most portable implementations). SPECK does not have this issue, and can thus be securely implemented portably with minimal effort.
How similar is TFS and ZFS?
Not that similar, actually. They share many of the basic ideas, but otherwise they are essentially unconnected. But ZFS' design has shaped TFS' a lot.
Is TFS Redox-only?
No, and it was never planned to be Redox-only.
How does whole-disk compression work?
Whole-disk compression is -- to my knowledge -- exclusive to TFS. It works by collecting as many "pages" (virtual data blocks) into a "cluster" (allocation unit). By doing this, the pages can be read by simply decompressing the respective cluster.
Why is ZMicro so slow? Will it affect the performance of TFS?
The reason ZMicro is so slow is because it works on a bit level, giving excellent compression ratio on the cost of performance. This horribly slow performance is paid back by the reduced number of writes. In fact, more than 50% of the allocations with ZMicro will only write one sector, as opposed to 3. Secondly, no matter how fast your disk is, it will not get anywhere near the performance of ZMicro because disk operations are inherently slow, and when put in perspective, the performance of the compression is really unimportant.
Extendible hashing or B+ trees?
Neither. TFS uses a combination of trees and hash tables: Nested hash tables, a form of hash trees. The idea is that instead of reallocating, a new subtable is created in the bucket.

Resources on design

I've written a number of pieces on the design of TFS:

Specification

The full specification can be found in specification.tex. To render it, install texlive or another distribution with XeTeX, and run

xelatex --shell-escape specification.tex

Then open the file named specification.pdf.

More Repositories

1

redox

Mirror of https://gitlab.redox-os.org/redox-os/redox
Shell
14,647
star
2

orbtk

The Rust UI-Toolkit.
Rust
3,773
star
3

termion

Mirror of https://gitlab.redox-os.org/redox-os/termion
Rust
1,994
star
4

ion

Mirror of https://gitlab.redox-os.org/redox-os/ion
Rust
1,401
star
5

relibc

Mirror of https://gitlab.redox-os.org/redox-os/relibc
Rust
780
star
6

rusttype

Mirror of https://gitlab.redox-os.org/redox-os/rusttype
Rust
600
star
7

kernel

Mirror of https://gitlab.redox-os.org/redox-os/kernel
Rust
517
star
8

coreutils

Mirror of https://gitlab.redox-os.org/redox-os/coreutils
Rust
259
star
9

sodium

Mirror of https://gitlab.redox-os.org/redox-os/sodium
Rust
181
star
10

games

Mirror of https://gitlab.redox-os.org/redox-os/games
Rust
119
star
11

book

Mirror of https://gitlab.redox-os.org/redox-os/book
98
star
12

orbital

Mirror of https://gitlab.redox-os.org/redox-os/orbital
Rust
69
star
13

redoxfs

Mirror of https://gitlab.redox-os.org/redox-os/redoxfs
Rust
68
star
14

drivers

Mirror of https://gitlab.redox-os.org/redox-os/drivers
Rust
44
star
15

bootloader

Mirror of https://gitlab.redox-os.org/redox-os/bootloader
Rust
40
star
16

orbutils

Mirror of https://gitlab.redox-os.org/redox-os/orbutils
Rust
39
star
17

orbclient

Mirror of https://gitlab.redox-os.org/redox-os/orbclient
Rust
33
star
18

cookbook

Mirror of https://gitlab.redox-os.org/redox-os/cookbook
Shell
32
star
19

orbtk-template

A template for starting an OrbTk project
Rust
30
star
20

pkgutils

Mirror of https://gitlab.redox-os.org/redox-os/pkgutils
Rust
30
star
21

syscall

Mirror of https://gitlab.redox-os.org/redox-os/syscall
Rust
29
star
22

netstack

Mirror of https://gitlab.redox-os.org/redox-os/netstack
Rust
29
star
23

netutils

Mirror of https://gitlab.redox-os.org/redox-os/netutils
Rust
28
star
24

extrautils

Mirror of https://gitlab.redox-os.org/redox-os/extrautils
Rust
28
star
25

redox-ssh

Mirror of https://gitlab.redox-os.org/redox-os/redox-ssh
Rust
27
star
26

website

Mirror of https://gitlab.redox-os.org/redox-os/website
CSS
24
star
27

installer

Mirror of https://gitlab.redox-os.org/redox-os/installer
Rust
21
star
28

calc

Mirror of https://gitlab.redox-os.org/redox-os/calc
Rust
21
star
29

orbgame

Mirror of https://gitlab.redox-os.org/redox-os/orbgame
Rust
19
star
30

binutils

Mirror of https://gitlab.redox-os.org/redox-os/binutils
Rust
18
star
31

orbterm

Mirror of https://gitlab.redox-os.org/redox-os/orbterm
Rust
17
star
32

orbtk-book

(WIP) The OrbTk book
Rust
16
star
33

bootloader-efi

Mirror of https://gitlab.redox-os.org/redox-os/bootloader-efi
Rust
16
star
34

ransid

Mirror of https://gitlab.redox-os.org/redox-os/ransid
Rust
16
star
35

stb_truetype-rs

Mirror of https://gitlab.redox-os.org/redox-os/stb_truetype-rs
Rust
16
star
36

userutils

Mirror of https://gitlab.redox-os.org/redox-os/userutils
Rust
16
star
37

libc

Mirror of https://gitlab.redox-os.org/redox-os/libc
C
15
star
38

binutils-gdb

Mirror of https://gitlab.redox-os.org/redox-os/binutils-gdb
C
13
star
39

thrussh

Mirror of https://gitlab.redox-os.org/redox-os/thrussh
Rust
12
star
40

rfcs

Mirror of https://gitlab.redox-os.org/redox-os/rfcs
12
star
41

uefi

Mirror of https://gitlab.redox-os.org/redox-os/uefi
Rust
11
star
42

pkgar

Mirror of https://gitlab.redox-os.org/redox-os/pkgar
Rust
9
star
43

users

Mirror of https://gitlab.redox-os.org/redox-os/users
Rust
9
star
44

ipcd

Mirror of https://gitlab.redox-os.org/redox-os/ipcd
Rust
8
star
45

init

Mirror of https://gitlab.redox-os.org/redox-os/init
Rust
8
star
46

libpager

Mirror of https://gitlab.redox-os.org/redox-os/libpager
Rust
8
star
47

design

Mirror of https://gitlab.redox-os.org/redox-os/design
8
star
48

handbook

Mirror of https://gitlab.redox-os.org/redox-os/handbook
7
star
49

libextra

Mirror of https://gitlab.redox-os.org/redox-os/libextra
Rust
6
star
50

orbfont

Mirror of https://gitlab.redox-os.org/redox-os/orbfont
Rust
6
star
51

redoxer

Mirror of https://gitlab.redox-os.org/redox-os/redoxer
Rust
6
star
52

playbot

Mirror of https://gitlab.redox-os.org/redox-os/playbot
Rust
6
star
53

acid

Mirror of https://gitlab.redox-os.org/redox-os/acid
Rust
5
star
54

arg-parser

Mirror of https://gitlab.redox-os.org/redox-os/arg-parser
Rust
5
star
55

rine

Mirror of https://gitlab.redox-os.org/redox-os/rine
Rust
4
star
56

assets

Mirror of https://gitlab.redox-os.org/redox-os/assets
Shell
4
star
57

uefi_alloc

Mirror of https://gitlab.redox-os.org/redox-os/uefi_alloc
Rust
4
star
58

mesa

Mirror of https://gitlab.redox-os.org/redox-os/mesa
C
4
star
59

termios

Mirror of https://gitlab.redox-os.org/redox-os/termios
Rust
4
star
60

dash

Mirror of https://gitlab.redox-os.org/redox-os/dash
C
3
star
61

smallstring

Mirror of https://gitlab.redox-os.org/redox-os/smallstring
Rust
3
star
62

orbimage

Mirror of https://gitlab.redox-os.org/redox-os/orbimage
Rust
3
star
63

isolinux

Mirror of https://gitlab.redox-os.org/redox-os/isolinux
3
star
64

ptyd

Mirror of https://gitlab.redox-os.org/redox-os/ptyd
Rust
3
star
65

bots

Mirror of https://gitlab.redox-os.org/redox-os/bots
Rust
3
star
66

gawk

Mirror of https://gitlab.redox-os.org/redox-os/gawk
C
2
star
67

contain

Mirror of https://gitlab.redox-os.org/redox-os/contain
Rust
2
star
68

event

Mirror of https://gitlab.redox-os.org/redox-os/event
Rust
2
star
69

docgen

Mirror of https://gitlab.redox-os.org/redox-os/docgen
Rust
2
star
70

orbdata

Mirror of https://gitlab.redox-os.org/redox-os/orbdata
2
star
71

dmi

Mirror of https://gitlab.redox-os.org/redox-os/dmi
Rust
2
star
72

randd

Mirror of https://gitlab.redox-os.org/redox-os/randd
Rust
2
star
73

periodictable

Mirror of https://gitlab.redox-os.org/redox-os/periodictable
Rust
2
star
74

libc-artifacts

Mirror of https://gitlab.redox-os.org/redox-os/libc-artifacts
C
2
star
75

backgrounds

Mirror of https://gitlab.redox-os.org/redox-os/backgrounds
2
star
76

small

Mirror of https://gitlab.redox-os.org/redox-os/small
Rust
1
star
77

netdb

Mirror of https://gitlab.redox-os.org/redox-os/netdb
1
star
78

netsurf

Mirror of https://gitlab.redox-os.org/redox-os/netsurf
C
1
star
79

conc

Mirror of https://gitlab.redox-os.org/redox-os/conc
Rust
1
star
80

orbaudio

Mirror of https://gitlab.redox-os.org/redox-os/orbaudio
Rust
1
star
81

logd

Mirror of https://gitlab.redox-os.org/redox-os/logd
Rust
1
star
82

keyboard-sfx

Mirror of https://gitlab.redox-os.org/redox-os/keyboard-sfx
1
star
83

shellstorm

Mirror of https://gitlab.redox-os.org/redox-os/shellstorm
Rust
1
star