• Stars
    star
    240
  • Rank 168,229 (Top 4 %)
  • Language
    Go
  • License
    MIT License
  • Created over 3 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Go io/fs.FS filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl

GoDoc Build

This module contains a collection of Go filesystem implementations that can be discovered dynamically by URL scheme.

These filesystems implement the fs.FS interface introduced in Go 1.16. This means that currently all implementations are read-only, however this may change in the future (see golang/go#45757 for progress).

Most implementations implement the fs.ReadDirFS interface, though the httpfs filesystem does not.

Some extensions are available to help add specific functionality to certain filesystems:

  • WithContextFS - injects a context into a filesystem, for propagating cancellation in filesystems that support it.
  • WithHeaderFS - sets the http.Header for all HTTP requests used by the filesystem. This can be useful for authentication, or for requesting specific content types.
  • WithHTTPClientFS - sets the *http.Client for all HTTP requests to be made with.

Many of the filesystem packages also have their own extensions.

This module also provides ContentType, an extension to the fs.FileInfo type to help identify an appropriate MIME content type for a given file. For filesystems that support it, the HTTP Content-Type header is used for this. Otherwise, the type is guessed from the file extension.

History & Project Status

This module is in active development, and the API is still subject to breaking changes.

The filesystem pacakges should operate correctly, based on the tests, but there may be edge cases that are not covered. Please open an issue if you find one!

Most of these filesystems are based on code from gomplate, which supports all of these as datasources. This module is intended to eventually be used within gomplate.

Supported Filesystems

Here's the list of filesystems & URL schemes supported by this module:

Package Scheme(s) Description
awssmfs aws+sm AWS Secrets Manager
awssmpfs aws+smp AWS Systems Manager Parameter Store
blobfs azblob Azure Blob Storage
blobfs gs Google Cloud Storage
blobfs s3 Amazon S3
consulfs consul, consul+http, consul+https HashiCorp Consul
filefs file local filesystem
gitfs git, git+file, git+http, git+https, git+ssh local/remote git repository
httpfs http, https HTTP server
vaultfs vault, vault+http, vault+https HashiCorp Vault

See the individual package documentation for more details.

Installation

Use go get to install the latest version of go-fsimpl:

$ go get -u github.com/hairyhenderson/go-fsimpl

Usage

If you know that you want an HTTP filesystem, for example:

import (
	"net/url"

	"github.com/hairyhenderson/go-fsimpl/httpfs"
)

func main() {
	base, _ := url.Parse("https://example.com")
	fsys, _ := httpfs.New(base)

	f, _ := fsys.Open("hello.txt")
	defer f.Close()

	// now do what you like with the file...
}

If you're not sure what filesystem type you need (for example, if you're dealing with a user-provided URL), you can use a filesystem mux:

import (
	"github.com/hairyhenderson/go-fsimpl"
	"github.com/hairyhenderson/go-fsimpl/blobfs"
	"github.com/hairyhenderson/go-fsimpl/filefs"
	"github.com/hairyhenderson/go-fsimpl/gitfs"
	"github.com/hairyhenderson/go-fsimpl/httpfs"
)

func main() {
	mux := fsimpl.NewMux()
	mux.Add(filefs.FS)
	mux.Add(httpfs.FS)
	mux.Add(blobfs.FS)
	mux.Add(gitfs.FS)

	// for example, a URL that points to a subdirectory at a specific tag in a
	// given git repo, hosted on GitHub and authenticated with SSH...
	fsys, err := mux.Lookup("git+ssh://[email protected]/foo/bar.git//baz#refs/tags/v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	f, _ := fsys.Open("hello.txt")
	defer f.Close()

	// now do what you like with the file...
}

Developing

You will require git including git daemon and consul executables on your path for running the tests.

License

The MIT License

Copyright (c) 2021-2022 Dave Henderson

More Repositories

1

gomplate

A flexible commandline tool for template rendering. Supports lots of local and remote datasources.
Go
2,262
star
2

go-which

A cross-platform Go implementation of the which(1) command, usable both as a CLI and library
Go
41
star
3

caddyprom

⚠️ DEPRECATED - use the core metrics supported in Caddy v2.2.0-rc.2+
Go
28
star
4

caddy-teapot-module

✨ A Caddy (v2) module which responds with "418 I'm a teapot" to all requests.
Go
22
star
5

github-responder

A library & CLI tool that automatically sets up GitHub WebHooks and listens for events, with automatic TLS
Go
21
star
6

go-codeowners

A Go package that finds and parses GitHub CODEOWNERS files
Go
16
star
7

jiraprinter

Print issue cards from JIRA
JavaScript
12
star
8

dockerhub_ratelimit_exporter

A Prometheus-format exporter to report on DockerHub per-image rate limits
Go
10
star
9

dockerfiles

Random Dockerfiles...
Dockerfile
9
star
10

jarvis_exporter

A Prometheus exporter for the Fully Jarvis standing desk
Go
9
star
11

hitron_coda

A Go client for the Hitron CODA-4680 Cable Modem/Router (and possibly others!)
Go
8
star
12

hitron_coda_exporter

A Prometheus Exporter for the Hitron CODA-4680 Cable Modem (and probably others too!)
Go
6
star
13

teams

CLI application to manage github issues and pull requests for multiple repos
Go
5
star
14

scanbar

Look for barcodes in the webcam ¯\_(ツ)_/¯
Go
4
star
15

github-sync-labels-milestones

A port of Xiphe/github-sync-labels-milestones to Golang
Go
4
star
16

go-onerng

⚠️INCOMPLETE⚠️ A Go port of the OneRNG tools
Go
3
star
17

homebrew-tap

A homebrew tap for my stuff
Ruby
3
star
18

radiorelay

Radio Relay project for CEML
3
star
19

markdowntopdf

A markdown-to-PDF conversion tool
Dockerfile
3
star
20

passport-fellowshipone

Passport strategy for authenticating with Fellowship One using the OAuth 1.0a API.
JavaScript
3
star
21

trafficlight

🚦 Controller for a Red/Yellow/Green (+buzzer) Traffic Light on a Raspberry Pi
Go
2
star
22

hairyhenderson

2
star
23

node-fellowshipone

A Fellowship One API wrapper for Node.js
JavaScript
2
star
24

hairyhenderson.github.io

HTML
2
star
25

wnp-bridge

Go
2
star
26

nursery_alert

A Raspberry Pi-based system for displaying alert codes to a screen
JavaScript
1
star
27

oauth2_proxy

Yet another oauth2_proxy image...
Dockerfile
1
star
28

docker-shairport-sync

Dockerfile
1
star
29

arduinostuff

A bunch of random Arduino (and related) sketches and code
Arduino
1
star
30

f1foo

ARCHIVED
JavaScript
1
star
31

cassandra-lucene-docker

1
star
32

sunrise-client

🌅Client for hairyhenderson/wifi-neopixel to run daily sunrise/sunset sequences🌇
Go
1
star
33

go-revendorbot

A bot that works with github-responder to revendor a Go repo on demand
Go
1
star
34

wifi-neopixel

C
1
star
35

linuxkit-configs

some random linuxkit configs that I've been playing with
Shell
1
star
36

dotfiles

Vim Script
1
star
37

hkrelay

A HomeKit-compatible Raspberry Pi Relay controller
Go
1
star
38

henet-ipupdater

🔃 A program that periodically updates dynamic IPs
Shell
1
star
39

GarageMonitor

An Arduino sketch for monitoring the state of a garage door (or, really, the state of any switch).
Arduino
1
star