• This repository has been archived on 26/Feb/2020
  • Stars
    star
    374
  • Rank 114,346 (Top 3 %)
  • Language
    Go
  • License
    Other
  • Created almost 9 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The Swift Virtual File System

*** This project is not maintained anymore ***

The Swift Virtual File System

Release Github All Releases Build Status Go Report Card Coverage Status GoDoc

SVFS is a Virtual File System over Openstack Swift built upon fuse. It is compatible with hubiC, OVH Public Cloud Storage and basically every endpoint using a standard Openstack Swift setup. It brings a layer of abstraction over object storage, making it as accessible and convenient as a filesystem, without being intrusive on the way your data is stored.

Disclaimer

This is not an official project of the Openstack community.

Installation

Download and install the latest release packaged for your distribution.

Usage

Mount command

On Linux (requires fuse and ruby) :

mount -t svfs -o <options> <device> /mountpoint

On OSX (requires osxfuse and ruby) :

mount_svfs <device> /mountpoint -o <options>

Notes :

  • You can pick any name you want for the device parameter.
  • All available mount options are described later in this document.

Credentials can be specified in mount options, however this may be desirable to read them from an external source. The following sections desribe alternative approaches.

Reading credentials from the environment

SVFS supports reading the following set of environment variables :

  • If you are using HubiC :
 HUBIC_AUTH
 HUBIC_TOKEN
  • If you are using a vanilla Swift endpoint (like OVH PCS), after sourcing your OpenRC file :
 OS_AUTH_URL
 OS_USERNAME
 OS_PASSWORD
 OS_REGION_NAME
 OS_TENANT_NAME
  • If you already authenticated to an identity endpoint :
 OS_AUTH_TOKEN
 OS_STORAGE_URL

Reading credentials from a configuration file

All environment variables can also be set in a YAML configuration file placed at /etc/svfs.yaml.

For instance :

hubic_auth: XXXXXXXXXX..
hubic_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...

Usage with OVH products

  • Usage with OVH Public Cloud Storage is explained here.
  • Usage with hubiC is explained here.

Mount options

Keystone options

  • auth_url: keystone URL (default is https://auth.cloud.ovh.net/v2.0).
  • username: your keystone user name.
  • password: your keystone password.
  • tenant: your project name.
  • region: the region where your tenant is.
  • version: authentication version (0 means auto-discovery which is the default).
  • storage_url: the storage endpoint holding your data.
  • internal_endpoint: the storage endpoint type (default is false).
  • token: a valid token.

Options region, version, storage_url and token are guessed during authentication if not provided.

Hubic options

  • hubic_auth: hubic authorization token as returned by the hubic-application command.
  • hubic_times : use file times set by hubic synchronization clients. Option attr should also be set for this to work.
  • hubic_token : hubic refresh token as returned by the hubic-application command.

Swift options

  • container: which container should be selected while mounting the filesystem. If not set, all containers within the tenant will be available under the chosen mountpoint.
  • storage_policy: expected containers storage policy. This is used to ignore containers not matching a particular storage policy name. If empty, this setting is ignored (default).
  • segment_size: large object segments size in MB. When an object has a content larger than this setting, it will be uploaded in multiple parts of the specified size. Default is 256 MB. Segment size should not exceed 5 GB.
  • connect_timeout: connection timeout to the swift storage endpoint. Default is 15 seconds.
  • request_timeout: timeout of requests sent to the swift storage endpoint. Default is 5 minutes.

Prefetch options

  • block_size: Filesystem block size in bytes. This is only used to report correct stat() results.
  • readahead_size: Readahead size in KB. Default is 128 KB.
  • readdir: Overall concurrency factor when listing segmented objects in directories (default is 20).
  • attr: Handle base attributes.
  • xattr: Handle extended attributes.
  • transfer_mode: Enforce network transfer optimizations. The following flags / features can be combined :
  • 1 : disable explicit empty file creation.
  • 2 : disable explicit directory creation.
  • 4 : disable directory content check on removal.
  • 8 : disable file check in read only opening.

Cache options

  • cache_access: cache entry access count before refresh. Default is -1 (unlimited access).
  • cache_entries: maximum entry count in cache. Default is -1 (unlimited).
  • cache_ttl: cache entry timeout before refresh. Default is 1 minute.

Access restriction options

  • allow_other: Bypass allow_root.
  • allow_root: Restrict access to root and the user mounting the filesystem.
  • default_perm: Restrict access based on file mode (useful with allow_other).
  • uid: default files uid (defaults to current user uid).
  • gid: default files gid (defaults to current user gid).
  • mode: default files permissions (default is 0700).
  • ro: enable read-only access.

Debug options

  • debug: enable debug log.
  • stdout : stdout redirection expression (e.g. >/dev/null).
  • stderr : stderr redirection expression (e.g. >>/var/log/svfs.log).
  • profile_addr: Golang profiling information will be served at this address (ip:port) if set.
  • profile_cpu: Golang CPU profiling information will be stored to this file if set.
  • profile_ram: Golang RAM profiling information will be stored to this file if set.

Performance options

  • go_gc: set garbage collection target percentage. A garbage collection is triggered when the heap size exceeds, by this rate, the remaining heap size after the previous collection. A lower value triggers frequent GC, which means memory usage will be lower at the cost of higher CPU usage. Setting a higher value will let the heap size grow by this percent without collection, reducing GC frequency. A Garbage collection is forced if none happened for 2 minutes. Note that unused heap memory is not reclaimed after collection, it is returned to the operating system only if it appears unused for 5 minutes.

Limitations

Be aware that SVFS doesn't transform object storage to block storage.

SVFS doesn't support :

  • Opening files in other modes than O_CREAT, O_RDONLY and O_WRONLY.
  • Moving directories.
  • Renaming containers.
  • SLO (but supports DLO).
  • Per-file uid/gid/permissions (but per-mountpoint).
  • Symlink targets across containers (but within the same container).

Take a look at the docs for further discussions about SVFS approach.

FAQ

Got errors using rsync with svfs ? Can't change creation time ? Why svfs after all ?

Take a look at the FAQ.

Hacking

Make sure to use the latest version of go and follow contribution guidelines of SVFS.

License

This work is under the BSD license, see the LICENSE file for details.

More Repositories

1

cds

Enterprise-Grade Continuous Delivery & DevOps Automation Open Source Platform
Go
4,572
star
2

the-bastion

Authentication, authorization, traceability and auditability for SSH accesses.
Perl
1,514
star
3

utask

µTask is an automation engine that models and executes business processes declared in yaml. ✏️📋
Go
1,197
star
4

venom

🐍 Manage and run your integration tests with efficiency - Venom run executors (script, HTTP Request, web, imap, etc... ) and assertions
Go
1,043
star
5

debian-cis

PCI-DSS compliant Debian 10/11/12 hardening
Shell
731
star
6

celery-director

Simple and rapid framework to build workflows with Celery
Python
535
star
7

php-ovh

Lightweight PHP wrapper for OVH APIs. That's the easiest way to use OVH.com APIs in your PHP applications.
PHP
287
star
8

python-ovh

Thin wrapper around OVH's APIs. Handles all the hard work including credential creation and requests signing.
Python
270
star
9

docs

Official repository containing all docs & guides of OVH Group
HTML
224
star
10

manager

OVHcloud Control Panel
JavaScript
213
star
11

overthebox

OverTheBox - Aggregate and encrypt your multiple internet connections.
Shell
197
star
12

public-cloud-roadmap

Agile roadmap for OVHcloud Public Cloud services. Discover the features our product teams are working on, comment and influence our backlog.
187
star
13

terraform-provider-ovh

Terraform OVH provider
Go
183
star
14

ai-training-examples

Jupyter Notebook
138
star
15

metronome

Metronome is a distributed and fault-tolerant event scheduler
Go
136
star
16

go-ovh

Simple go wrapper for the OVH API
Go
127
star
17

node-ovh

Node.js wrapper for the OVH APIs
JavaScript
125
star
18

symmecrypt

Golang symmetric encryption library
Go
115
star
19

ovh-ttyrec

Enhanced (but compatible) version of the classic ttyrec
C
112
star
20

overthebox-feeds

OverTheBox - LEDE/OpenWrt feed
JavaScript
106
star
21

sv2chisel

(System)Verilog to Chisel translator
Scala
104
star
22

overthebox-openwrt

OverTheBox - OpenWrt fork
C
98
star
23

celery-dyrygent

Celery extension which allows to orchestrate 100/1000/10000 tasks combined into a complex workflow
Python
97
star
24

tat

Tat Engine - Text And Tags
Go
89
star
25

beamium

Prometheus to Warp10 metrics forwarder
Rust
84
star
26

ip-reputation-monitoring

Monitor the reputation of your IP ranges
Python
83
star
27

depc

QoS Measurement & Dependency Graph Platform
Python
80
star
28

configstore

Golang configuration management library
Go
65
star
29

erlenmeyer

Erlenmeyer is a proxy used to parse common Open Source TimeSeries DataBase query endpoints like OpenTSDB, Prometheus/PromQL, InfluxQL or Graphite. Parsed queries are translated into WarpScript to produce native Warp 10 queries.
Go
60
star
30

ovh-ui-kit

OVHcloud UI Kit - Master UI Framework
JavaScript
58
star
31

tsl

TSL is a HTTP proxy which generate WarpScript or a PromQl script based on a TSL query.
Go
56
star
32

noderig

Export OS stats as Sensision Metrics
Go
55
star
33

terraform-ovh-commons

This repo contains commons resources to interact with OVH Public Cloud using Terraform.
HTML
54
star
34

php-ovh-sms

PHP for ovh sms API
PHP
48
star
35

ovh-cli

OVH Command Line Interface
Python
42
star
36

infrastructure-roadmap

Agile roadmap for OVHcloud for Baremetal, Network and Storage IaaS services. Discover the features our product teams are working on, comment and influence our backlog.
38
star
37

the-bastion-ansible-wrapper

Using Ansible through The Bastion
Python
35
star
38

summit2016-RankingPredict

Deprecated, No more maintained - Deprecated, no longer maintained
R
34
star
39

ovh-warp10-datasource

Grafana datasource for Warp10 platform
TypeScript
33
star
40

csharp-ovh

Thin wrapper around OVH's APIs. Handles all the hard work including credential creation and requests signing
C#
28
star
41

order-cart-examples

Example code snippets demonstrating how to use new order with cart on api.ovh.com
PHP
26
star
42

ldp-tail

OVH Logs Data Platform - Tail CLI tooling
Go
25
star
43

design-system

A collection of assets, guidelines and UI components for building consistent user experiences across OVHcloud products.
SCSS
22
star
44

public-cloud-examples

Jupyter Notebook
22
star
45

haproxy-exporter

Export HAProxy stats as Sensision Metrics
Go
21
star
46

fossil

Fossil is a proxy for securing unencrypted Graphite metrics collection
Go
20
star
47

python-apispec-fromfile

APISpec plugin to import OpenAPI specifications from a file
Python
19
star
48

terraform-ovh-publiccloud-network

HCL
19
star
49

tatwebui

Tat Web UI - Text And Tags
JavaScript
19
star
50

java-ovh

Thin wrapper around OVH's APIs. Handles all the hard work including credential creation and requests signing
Java
19
star
51

pulumi-ovh

Pulumi provider for OVHcloud
Java
17
star
52

cerberus-ux

HTML
16
star
53

cerberus-core

Cerberus is a toolkit to receive, parse, process and automate abuse reports handling received by ISP or hosting providers.
Python
15
star
54

public-cloud-databases-examples

OVHcloud Public Cloud Databases Training examples
HCL
14
star
55

lxd-puppet-module

Ruby
14
star
56

osarchiver

OpenStack databases archiver
Python
14
star
57

jerem

Jerem is a golang bot that scrap JIRA project to extract Metrics. Those Metrics can then be send to a Warp 10 Backend.
Go
14
star
58

website-evidence-collector-batch

A tool to launch website-evidence-collector on several URLs or Sitemaps and generate a full report.
JavaScript
13
star
59

distronaut

Find distribution installers all across the web !
Go
13
star
60

rtm

RTM (OVH Real Time Monitoring)
Perl
13
star
61

ovh-ui-kit-bs

A bootstrap theme for the OVH managers, based on ovh-ui-kit
Less
13
star
62

prescience-client

Desktop python client for using OVH Prescience service
Jupyter Notebook
13
star
63

gulp-drupal-stack

OVH Gulp tasks for Drupal themes and modules
JavaScript
12
star
64

serving-runtime

Exposes a serialized machine learning model through a HTTP API.
Java
12
star
65

swift-ovh

This Swift package is a lightweight wrapper for OVH APIs. That's the easiest way to use OVH.com APIs in your Swift applications. Apple platforms supported: iOS, OSX, tvOS, watchOS.
Swift
11
star
66

metronome-ui

User interface for Metronome
JavaScript
11
star
67

private-cloud-roadmap

Agile roadmap for OVHcloud Hosted Private Cloud services. Discover the features our product teams are working on, comment and influence our backlog.
11
star
68

telephony-example-cti-dashboard

Shows signaling events provided by OVH France telephony services CTI
CSS
11
star
69

drucker

Drucker is a lightweight Drupal Developer Environment.
Shell
10
star
70

phishing-mitigation

Tilera-based phishing mitigation layer
JavaScript
10
star
71

lhasa

List and map micro-services-based information system and observe how they interact. Track their activities, then gamify their continuous improvement.
Go
10
star
72

python-logging-gelf

A python logging bundle to send logs using GELF
Python
9
star
73

webpaas-cli

PHP
8
star
74

overthebox-lede

OverTheBox - OpenWRT fork
C
8
star
75

terraform-provider-mimirtool

Terraform provider for Grafana Mimir
Go
8
star
76

owstats-ui

JavaScript
7
star
77

vbridge

X11 cloud desktop software
C
7
star
78

djehouty

Djehouty intends to be a set of logging formatters and handlers to easily send log entries.
Python
7
star
79

crystal-ovh

Lightweight Crystal wrapper around OVH's APIs.
Crystal
7
star
80

ovhdata-cli

Rust
7
star
81

python-warp10client

Python
7
star
82

data-processing-spark-submit

Spark CLI wrapper to run your jobs on OVHcloud Data Processing
Go
7
star
83

docs-rendering

[DEPRECATED] Official rendering engine for static generation of OVH Docs platform
HTML
7
star
84

interpretability-engine

Interpret Machine Learning black-box models deployed on Serving Engine
Python
6
star
85

yubico-piv-checker

Go
6
star
86

webhosting-ssh-bashrc

Shell
6
star
87

puppet-thebastion

Puppet module for Thebastion management.
Ruby
6
star
88

public-cloud-databases-operator

This operator allow you to automaticaly authorize your Kubernetes cluster IP on your OVHcloud cloud databases service.
Go
6
star
89

docs-developer-env

Easy to deploy developer environment, for writing/testing guides & documentations for docs.ovh.com
Shell
6
star
90

tat-contrib

Go
6
star
91

bringyourownlinux

Shell
6
star
92

terraform-ovh-publiccloud-docker-swarm

HCL
5
star
93

terraform-ovh-publiccloud-spark

HCL
5
star
94

summit2016-webhosting-example-rondcoin

PHP
5
star
95

collectd-write-warp10

Python
5
star
96

fiowebviewer

Python
5
star
97

puppet-mimir

Ruby
4
star
98

pingdom-to-graphite

A tool for copying metrics from Pingdom to Graphite.
JavaScript
4
star
99

ovh-winston-ldp

A graylog2 TCP/TLS transport for winston library
JavaScript
4
star
100

okms-sdk-go

The Golang SDK to interact with your OVHcloud KMS services.
Go
4
star