• Stars
    star
    330
  • Rank 127,657 (Top 3 %)
  • Language
    Go
  • License
    Other
  • Created about 8 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A small utility which generates Go code from any file. Useful for embedding binary data in a Go program.

bindata

This fork is maintained by Kevin Burke, and is the version trusted by Homebrew. Changes made include:

  • Atomic writes; generated file cannot be read while partially complete.

  • Better encoding of files that contain characters in the Unicode format range.

  • Generated file reports file sizes.

  • Generated code is run through go fmt.

  • SHA256 hashes are computed for all files and stored in the binary. You can use this to detect in-memory corruption and to provide easy cache-busting mechanisms.

  • Added AssetString and MustAssetString functions.

  • ByName is not public.

  • Some errors in file writes were unchecked, but are now checked.

  • File modes are stored in octal (0644) instead of nonsensical decimal (420)

This package converts any file into manageable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice.

It comes with a command line tool in the go-bindata subdirectory. This tool offers a set of command line options, used to customize the output being generated.

Installation

On Macs, you can install the binary using Homebrew:

brew install go-bindata

You can also download a binary from the releases page. Switch in your GOOS for the word "linux" below, and the latest version for the version listed below:

curl --silent --location --output /usr/local/bin/go-bindata https://github.com/kevinburke/go-bindata/releases/download/v4.0.0/go-bindata-linux-amd64
chmod 755 /usr/local/bin/go-bindata

Alternatively, if you have a working Go installation, you can build the source and install the executable into $GOPATH/bin or $GOBIN:

go install github.com/kevinburke/go-bindata/v4/...@latest
# for versions of Go < 1.11, or without module support, use:
go install github.com/kevinburke/go-bindata/...

Usage

Conversion is done on one or more sets of files. They are all embedded in a new Go source file, along with a table of contents and an Asset function, which allows quick access to the asset, based on its name.

The simplest invocation generates a bindata.go file in the current working directory. It includes all assets from the data directory.

$ go-bindata data/

To include all input sub-directories recursively, use the ellipsis postfix as defined for Go import paths. Otherwise it will only consider assets in the input directory itself.

$ go-bindata data/...

To specify the name of the output file being generated, use the -o option:

$ go-bindata -o myfile.go data/

Multiple input directories can be specified if necessary.

$ go-bindata dir1/... /path/to/dir2/... dir3

The following paragraphs detail some of the command line options which can be supplied to go-bindata. Refer to the testdata/out directory for various output examples from the assets in testdata/in. Each example uses different command line options.

To ignore files, pass in regexes using -ignore, for example:

$ go-bindata -ignore=\\.gitignore data/...

Accessing an asset

To access asset data, we use the Asset(string) ([]byte, error) function which is included in the generated output.

data, err := Asset("pub/style/foo.css")
if err != nil {
	// Asset was not found.
}

// use asset data

Debug vs Release builds

When invoking the program with the -debug flag, the generated code does not actually include the asset data. Instead, it generates function stubs which load the data from the original file on disk. The asset API remains identical between debug and release builds, so your code will not have to change.

This is useful during development when you expect the assets to change often. The host application using these assets uses the same API in both cases and will not have to care where the actual data comes from.

An example is a Go webserver with some embedded, static web content like HTML, JS and CSS files. While developing it, you do not want to rebuild the whole server and restart it every time you make a change to a bit of javascript. You just want to build and launch the server once. Then just press refresh in the browser to see those changes. Embedding the assets with the debug flag allows you to do just that. When you are finished developing and ready for deployment, just re-invoke go-bindata without the -debug flag. It will now embed the latest version of the assets.

Lower memory footprint

Using the -nomemcopy flag, will alter the way the output file is generated. It will employ a hack that allows us to read the file data directly from the compiled program's .rodata section. This ensures that when we call our generated function, we omit unnecessary memcopies.

The downside of this, is that it requires dependencies on the reflect and unsafe packages. These may be restricted on platforms like AppEngine and thus prevent you from using this mode.

Another disadvantage is that the byte slice we create, is strictly read-only. For most use-cases this is not a problem, but if you ever try to alter the returned byte slice, a runtime panic is thrown. Use this mode only on target platforms where memory constraints are an issue.

The default behavior is to use the old code generation method. This prevents the two previously mentioned issues, but will employ at least one extra memcopy and thus increase memory requirements.

For instance, consider the following two examples:

This would be the default mode, using an extra memcopy but gives a safe implementation without dependencies on reflect and unsafe:

func myfile() []byte {
    return []byte{0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a}
}

Here is the same functionality, but uses the .rodata hack. The byte slice returned from this example can not be written to without generating a runtime error.

var _myfile = "\x89\x50\x4e\x47\x0d\x0a\x1a"

func myfile() []byte {
    var empty [0]byte
    sx := (*reflect.StringHeader)(unsafe.Pointer(&_myfile))
    b := empty[:]
    bx := (*reflect.SliceHeader)(unsafe.Pointer(&b))
    bx.Data = sx.Data
    bx.Len = len(_myfile)
    bx.Cap = bx.Len
    return b
}

Optional compression

When the -nocompress flag is given, the supplied resource is not GZIP compressed before being turned into Go code. The data should still be accessed through a function call, so nothing changes in the usage of the generated file.

This feature is useful if you do not care for compression, or the supplied resource is already compressed. Doing it again would not add any value and may even increase the size of the data.

The default behavior of the program is to use compression.

Path prefix stripping

The keys used in the _bindata map, are the same as the input file name passed to go-bindata. This includes the path. In most cases, this is not desireable, as it puts potentially sensitive information in your code base. For this purpose, the tool supplies another command line flag -prefix. This accepts a portion of a path name, which should be stripped off from the map keys and function names.

For example, running without the -prefix flag, we get:

$ go-bindata /path/to/templates/

_bindata["/path/to/templates/foo.html"] = path_to_templates_foo_html

Running with the -prefix flag, we get:

$ go-bindata -prefix "/path/to/" /path/to/templates/

_bindata["templates/foo.html"] = templates_foo_html

Build tags

With the optional -tags flag, you can specify any go build tags that must be fulfilled for the output file to be included in a build. This is useful when including binary data in multiple formats, where the desired format is specified at build time with the appropriate tags.

The tags are appended to a // +build line in the beginning of the output file and must follow the build tags syntax specified by the go tool.

Related projects

go-bindata-assetfs - implements http.FileSystem interface. Allows you to serve assets with net/http.

More Repositories

1

hamms

Malformed servers to test your HTTP client
Python
1,218
star
2

doony

UI Improvements for Jenkins
JavaScript
970
star
3

nacl

Pure Go implementation of the NaCL set of API's
Go
540
star
4

ssh_config

Go parser for ssh_config files
Go
386
star
5

hulk

In-browser JSON editor
JavaScript
312
star
6

sshpass

Mirror of http://sourceforge.net/projects/sshpass/
Shell
199
star
7

hostsfile

go tool for working with /etc/hosts files
Go
131
star
8

rct

Creating cool coasters for Roller Coaster Tycoon
Go
124
star
9

tecate

Figure out when your HTML is busted
JavaScript
110
star
10

customize-twitter-1.1

Add your own custom CSS to modify the Twitter Embeddable Widget
JavaScript
91
star
11

tss

Go port of moreutils/ts
Go
43
star
12

swish

Switch SSH settings between Github profiles
Go
38
star
13

travis

Command line client for interacting with Travis CI
Go
19
star
14

handlers

Useful HTTP middlewares
Go
19
star
15

gitopen

Open one of your remote URL's in your browser
Python
17
star
16

rest

Go REST helpers
Go
14
star
17

sll

Strip long lines from output
Go
14
star
18

multi-emailer

Send personalized email messages to multiple email accounts (e.g. City Council members)
Go
13
star
19

2013

Flat UI website redesign
HTML
12
star
20

differ

Go
12
star
21

read-mongo-logs

Tail Mongo database logs
Go
11
star
22

gitlab

Command line tool for waiting for Gitlab pipelines to complete
Go
11
star
23

ansible-go

Go
9
star
24

rct-rides

working with roller coaster tycoon saved ride format
Go
9
star
25

weirdfortune

the unix fortune program, now with weird twitter
Python
8
star
26

snapchat-friends

surprise, your friend network is public
Python
8
star
27

proto-make-example

Makefile
8
star
28

gobike

Go
6
star
29

ynab-go

YNAB Go Client, including a detailed age of money calculator
Go
6
star
30

write_mailmap

Easy generate an AUTHORS file from the Git commit history
Makefile
6
star
31

metrosolver

Finding optimal Mini Metro routes
6
star
32

google-oauth-handler

HTTP middleware for handling Google authentication
Go
5
star
33

targets

I am the next coming of John Carmack
C#
5
star
34

go-random-project-generator

Random project name generator (like Github or Heroku app names)
Go
5
star
35

twilio-jsonapi

A JSON convenience wrapper for the Twilio API
Python
4
star
36

godocdoc

Start godoc and open a HTTP server to the homepage
Go
4
star
37

isec2

Go library that reports whether you are running in EC2
Go
3
star
38

envdir

Go port of djb envdir
Go
3
star
39

vault-go

Better Hashicorp Vault client
Go
3
star
40

public-comments

I write letters to local governments and post them here
Go
3
star
41

clipper

API for retrieving Clipper Card data (and parsing Clipper Transactions)
Go
3
star
42

talks

Talks I give at conferences
3
star
43

goodmorningcmc

Python
2
star
44

circle-webhook

webhook server and JSON parser for circle ci webhooks
Go
2
star
45

gerrit-heroku

Attempting to run Gerrit on Heroku
Makefile
2
star
46

slides

Presentations
HTML
2
star
47

write_config_from_env

Pull env vars into a config file
Go
2
star
48

humanbench

Human-readable benchmark output
Go
2
star
49

tt

Better Node test runner
Go
2
star
50

jenkins

Open Jenkins urls from your command line
Python
2
star
51

haa

CA Housing Accountability Act Resources
2
star
52

stubhub-tickets

checking stubhub ticket prices
Python
2
star
53

tarbz2.com

Helping you remember which tar option to use.
ApacheConf
2
star
54

goose

Maintained fork of liamstask/goose that supports ALTER TYPE, CREATE INDEX CONCURRENTLY
Go
2
star
55

buildkite

Buildkite CLI tool
Go
2
star
56

chroma-markdown

Combined Markdown + syntax highlight HTML compiler
Go
2
star
57

gostdjs

The Go standard library, implemented in Javascript
JavaScript
1
star
58

old-county-road

JavaScript
1
star
59

generic-pool-timeout

JavaScript
1
star
60

oculus-rating-data

Working with Oculus rating data
Go
1
star
61

tarsnap-old-archives

Deleting old Tarsnap archives
Go
1
star
62

rustls-postgres

1
star
63

sample-html

sample html
1
star
64

local-servers

Making your local projects more browsable
Makefile
1
star
65

recompile

Concurrent recompilation of individual files (see README)
Go
1
star
66

Twilio-Python-quickstarts

Python quickstarts for Twilio
Python
1
star
67

flagr

Rearrange flags so the options come first
Go
1
star
68

delete-phone-numbers

Delete numbers from a Twilio account
Go
1
star
69

flot.selection.js

The Flot selection plugin, with draggable left/right edges
JavaScript
1
star
70

make

Go
1
star
71

javascript-ipython

ipython notebook for javascript
Shell
1
star
72

enable_pg_logs

Enable Postgres query logging
Go
1
star
73

ipv6-etc-hosts

Go
1
star
74

telapi-python

Tel API Python Helper Library
Python
1
star
75

lucifer

Hot, hot test reloading from Javascript
JavaScript
1
star
76

waterline-self-driving-car

1
star
77

newproject

1
star
78

bigtext

Go library to display text really big using Quicksilver on a Mac
Go
1
star
79

twilio_munischedule

Ruby
1
star