• Stars
    star
    121
  • Rank 293,924 (Top 6 %)
  • Language
    Go
  • License
    BSD 3-Clause "New...
  • Created almost 8 years ago
  • Updated about 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Compile text/template / html/template to regular go code

template-compiler

Compile text/template / html/template to regular go code.

still a wip!

Install

You will need both library and binary.

go get github.com/mh-cbon/template-compiler
cd $GOPATH/src/github.com/mh-cbon/template-compiler
glide install
go install

CLI

template-compiler - 0.0.0

  -help | -h   Show this help.
  -version     Show program version.
  -keep        Keep bootstrap program compiler.
  -print       Print bootstrap program compiler.
  -var         The variable name of the configuration in your program
               default: compiledTemplates
  -wdir        The working directory where the bootstrap program is written
               default: $GOPATH/src/template-compilerxx/

Examples
  template-compiler -h
  template-compiler -version
  template-compiler -keep -var theVarName
  template-compiler -keep -var theVarName -wdir /tmp

Usage

Let s take this example package

package mypackage

import(
  "net/http"
  "html/template"
)

var tplFuncs = map[string]interface{}{
  "up": strings.ToUpper,
}

type TplData struct {
  Email string
  Name string
}

func handler(w http.ResponseWriter, r *http.Request) {
    t := template.New("").Funcs(tplFuncs).ParseFiles("tmpl/welcome.html")
    t.Execute(w, TplData{})
}

With this template

{{.Email}} {{.Name}}

To generate compiled version of your template, change it to

package mypackage

import (
  "net/http"

  "github.com/mh-cbon/template-compiler/compiled"
  "github.com/mh-cbon/template-compiler/std/text/template"
)

//go:generate template-compiler
var compiledTemplates = compiled.New(
  "gen.go",
  []compiled.TemplateConfiguration{
    compiled.TemplateConfiguration{
      HTML:          true,
      TemplatesPath: "tmpl/*.tpl",
      TemplatesData: map[string]interface{}{
        "*": TplData{},
      },
      FuncsMap:      []string{
        "somewhere/mypackage:tplFuncs",
      },
    },
    compiled.TemplateConfiguration{
      TemplateName:    "notafile",
      TemplateContent: `hello!{{define "embed"}}{{.Email}} {{.Name}}{{end}}`,
      TemplatesData: map[string]interface{}{
        "*": nil,
        "embed": TplData{},
      },
    },
  },
)

var tplFuncs = map[string]interface{}{
  "up": strings.ToUpper,
}

type TplData struct {
  Email string
  Name string
}

func handler(w http.ResponseWriter, r *http.Request) {
    compiledTemplates.MustGet("welcome.tpl").Execute(w, TplData{})
    compiledTemplates.MustGet("notafile").Execute(w, nil)
    compiledTemplates.MustGet("embed").Execute(w, TplData{})
}

Then run,

go generate

It will produce a file gen.go containing the code to declare and run the compiled templates,

package main

//golint:ignore

import (
 "io"
 "github.com/mh-cbon/template-compiler/compiled"
 "github.com/mh-cbon/template-compiler/std/text/template/parse"
 "path/to/mypackage"
)

func init () {
  compiledTemplates = compiled.NewRegistry()
  compiledTemplates.Add("welcome.tpl", fnaTplaTpl0)
}

// only demonstration purpose, not the actual real generated code for the example template.
func fnaTplaTpl0(t parse.Templater, w io.Writer, indata interface {
}) error {
  var bw bytes.Buffer
  var data aliasdata.MyTemplateData
  if d, ok := indata.(aliasdata.MyTemplateData); ok {
    data = d
  }
  if _, werr := w.Write(builtin5); werr != nil {
    return werr
  }
  var var2 []string = data.MethodItems()
  var var1 int = len(var2)
  var var0 bool = 0 != var1
  if var0 {
    if _, werr := w.Write(builtin6); werr != nil {
      return werr
    }
    var var3 []string = data.MethodItems()
    for _, iterable := range var3 {
      if _, werr := w.Write(builtin7); werr != nil {
        return werr
      }
      bw.WriteString(iterable)
      template.HTMLEscape(w, bw.Bytes())
      bw.Reset()
      if _, werr := w.Write(builtin8); werr != nil {
        return werr
      }
    }
    if _, werr := w.Write(builtin9); werr != nil {
      return werr
    }
  } else {
    if _, werr := w.Write(builtin10); werr != nil {
      return werr
    }
  }
  if _, werr := w.Write(builtin2); werr != nil {
    return werr
  }
  return nil
}
var builtin0 = []byte(" ")

// more like this

What would be the performance improvements ?

Given the templates compiled as HTML available here

  $ go test -bench=. -benchmem
  BenchmarkRenderWithCompiledTemplateA-4   	20000000	       78.9 ns/op	      48 B/op	       1 allocs/op
  BenchmarkRenderWithJitTemplateA-4        	 3000000	       668 ns/op	      96 B/op	       2 allocs/op
  BenchmarkRenderWithCompiledTemplateB-4   	20000000	       82.4 ns/op	      48 B/op	       1 allocs/op
  BenchmarkRenderWithJitTemplateB-4        	 3000000	       603 ns/op	      96 B/op	       2 allocs/op
  BenchmarkRenderWithCompiledTemplateC-4   	  500000	      2530 ns/op	     192 B/op	       6 allocs/op
  BenchmarkRenderWithJitTemplateC-4        	   50000	     38245 ns/op	    3641 B/op	      82 allocs/op
  BenchmarkRenderWithCompiledTemplateD-4   	20000000	       114 ns/op	      48 B/op	       1 allocs/op
  BenchmarkRenderWithJitTemplateD-4        	 3000000	       809 ns/op	     144 B/op	       3 allocs/op

  // the next 2 benchmarks are particularly encouraging
  // as they involve 2k html string escaping
  BenchmarkRenderWithCompiledTemplateE-4   	   10000	    103929 ns/op	     160 B/op	       2 allocs/op
  BenchmarkRenderWithJitTemplateE-4        	     300	   6207912 ns/op	  656598 B/op	   18012 allocs/op
  BenchmarkRenderWithCompiledTemplateF-4   	   10000	    111047 ns/op	     160 B/op	       2 allocs/op
  BenchmarkRenderWithJitTemplateF-4        	     200	   5836766 ns/op	  657000 B/op	   18024 allocs/op

Depending on the kind of template expect 5 to 30 times faster and much much less allocations.

Understanding

This paragraph will describe and explain the various steps from the go:generate command, to the write of the compiled go code.

  1. When go:generate is invoked, the go tool will parse and invoke your calls to template-compiler. template-compiler is invoked in the directory containing the file with the go:generate comment, go generate also declares an environment variable GOFILE. With those hints template-compiler can locate and consume the variable declared with -var parameter. We are here
  2. template-compiler will generate a bootstrap program. We are here
  3. The generation of the bootstrap program is about parsing, browsing, and re exporting an updated version of your configuration variable. It specifically looks for each compiled.TemplateConfiguration{}:
  • If the configuration is set to generate html content with the key HTML:true, it ensure that stdfunc are appropriately declared into the configuration.
  • It read and evaluates the data field Data: your.struct{}, generates a DataConfiguration{} of it, and adds it to the template configuration.
  • It checks for FuncsMap key, and export those variable targets (with the help of this package) to FuncsExport and PublicIdents keys. We are here
  1. template-compiler writes and compiles a go program into $GOPATH/src/template-compilerxxx. This program is made to compile the templates with the updated configuration. We are here
  2. bootstrap-program is now invoked. We are here
  3. bootstrap-program browses the configuration value, for each template path, it compiles it as text/template or html/template. This steps creates the standard template AST Tree. Each template tree is then transformed and simplified with the help of this package. We are here
  4. template-tree-simplifier takes in input the template tree and apply transformations:
  • It unshadows all variables declaration within the template.
  • It renames all template variables to prefix them with tpl
  • It simplifies structure such as {{"son" | split "wat"}} to {{$var0 := split "wat" "son"}}{{$var0}}
  • It produces a small type checker structure which registers variable and their type for each scope of the template. We are here
  1. bootstrap-program browses each simplified template tree, generates a go function corresponding to it. We are here
  2. bootstrap-program generates an init function to register to your configuration variable the new functions as their template name. We are here
  3. bootstrap-program writes the fully generated program.

Working with funcmap

template-compiler needs to be able to evaluate the funcmap consumed by the templates.

In that matter template-compiler can take in input a path to a variable declaring this functions.

pkgPath:variableName where pkgPath is the go pakage path such as text/template, the variable name is the name of the variable declaring the funcmap such as builtins. See this.

It can read map[string]interface{} or template.FuncMap.

It can extract exported or unexported variables.

Functions declared into the funcmap can be exported, unexported, or inlined.

Note that unexported functions needs some runtime type checking.

examples

If you like sprig, you d be able to consume those functions with the path,

github.com/Masterminds/sprig:genericMap

If you prefer gtf, you d be able to consume those functions with the path,

github.com/leekchan/gtf:GtfFuncMap

beware

it can t evaluate a function call! It must be a variable declaration into the top level context such as

package yy

var funcs = map[string]interface{}{
  "funcname": func(){},
  "funcname2": pkg.Func,
}

Working with template data

The data consumed by your template must follow few rules:

  • It must be an exported type.
  • It must not be declared into a main package.

Others warnings

As the resulting compilation is pure go code, the type system must be respected, thus unexported types may not work.

The ugly stuff

Unfortunately this package contains some ugly copy pastes :x :x :x

It duplicates both text/template and html/template.

It would be great to backport those changes into core go code to get ride of those duplications.

  1. Added a new text/template.Compiled type. Much like a text/template or an html/template, Compiled has a *parse.Tree. This tree is a bultin tree to hold only one node to execute the compiled function. Doing so allow to mix compiled and non-compiled templates. see here
  2. Added a new method text/template.GetFuncs() to get the funcs related to the template. This is usefull to the compiled template functions to get access to those unexported functions. see here
  3. Added text/template.Compiled() to attach a compiled template to a regular text/template instance. see here
  4. Added a new tree node text/template/parse.CompiledNode, which knows the function to execute for a compiled template. see here
  5. Added a new interface text/template/parse.Templater, to use in the compiled function to receive the current template executed. This instance can be one of text/template.Template, html/template.Template or text/template.Compiled. see here
  6. Added a new type CompiledTemplateFunc for the signature of a compiled template function. see here
  7. Added a new funcmap variable html/template.publicFuncMap to map all html template idents to a function. It also delcares all escapers to a public function to improve performance of compiled templates. see here
  8. Added support of CompiledNode to the state walker see here

TBD

Here are some optimizations/todos to implement later:

  • When compiling templates, funcs like _html_template_htmlescaper will translate to template.HTMLEscaper. It worth to note that many cases are probably template.HTMLEscaper(string), but template.HTMLEscaper is doing some extra job to type check this string value. An optimization is to detect those calls template.HTMLEscaper(string) and transformedform them to template.HTMLEscapeString(string)
  • Same as previous for most escaper functions of html/template
  • Detect template calls such eq(bool, bool), or neq(int, int) and transform them to an appropriate go binary test bool == bool, ect.
  • Detect templates calls such len(some) and transforms it to the builtin len function.
  • Detect prints of struct or *struct, check if they implements Stringer, or something like Byter, and make use of that to get ride of some fmt.Sprintf calls.
  • review the install procedure, i suspect it is not yet correct. Make use of glide.
  • consolidate additions to std text/template/html/template packages.
  • version releases.
  • implement cache for functions export.
  • add template.Options support (some stuff there)
  • add channel support (is it really used ? :x)
  • add a method to easily switch from compiled function to original templates without modifying the configuration, imports ect.

More Repositories

1

go-msi

Easy way to generate msi package for a Go project
Go
425
star
2

go-bin-rpm

Create binary rpm package with ease
Go
88
star
3

go-github-release

Guide to release automation
Shell
74
star
4

go-bin-deb

Create binary package for debian system
Go
41
star
5

dht

golang dht / Kademlia
Go
24
star
6

emd

Enhanced Markdown template processor
Go
14
star
7

changelog

Maintain a changelog easily
Go
8
star
8

gssc

Easily starts an HTTPS server with self-signed certificates
Go
7
star
9

gump

Bin util to bump your package using semver
Go
7
star
10

astutil

Package astutil provides useful methods to work with ast when you intend to make a generator.
Go
6
star
11

dht-store

dht fun bep44
Go
5
star
12

lister

generates typed slice
Go
5
star
13

http-file-store

HTTP server to store files
JavaScript
4
star
14

jedi

database golang generator: dbr additions
Go
4
star
15

gh-api-cli

Command line client for github api
Go
4
star
16

plumber

builds pipes to transform a data stream
Go
4
star
17

ignore-file

Parse and test an ignore file, such as .gitignore
Go
3
star
18

gigo

go generate on steroids
Go
3
star
19

webtorrent-http-api

json http api for webtorrent
JavaScript
3
star
20

the-busy-man

initialize a project
Go
3
star
21

goriller

goriller generate gorilla routers
Go
3
star
22

testndoc

generate API documentation by listening to your tests.
Go
3
star
23

go-get-started

get started with go
Go
3
star
24

extract-imports

A program to extract import directives of given go files.
Go
3
star
25

mdl-go-components

MDL for go
Go
3
star
26

youtube-dl

Python
3
star
27

state-lexer

A state Lexer similar to Rob Pike's presentation
Go
3
star
28

sudo-fs

Like fs module. To use with sudo / elevate. Works on linux, mac, windows.
JavaScript
3
star
29

rm-glob

Rm globed files
Go
2
star
30

dhtest

dht bep44 upt http
Go
2
star
31

latest

A centralized and publicly hosted sh script to install latest debian or rpm package
Go
2
star
32

netlisten

listen a net address and copy to a destination
Go
2
star
33

aze

a proxy to reduce bandwidth
Go
2
star
34

go-repo-utils

Go tool to speak with repositories
Go
2
star
35

html-parser-lexer

Parse html content
Go
2
star
36

cct

concurrently run cli commands
Go
2
star
37

report-panic

Automatically report your programs panic to their github repository
Go
2
star
38

mutexer

generator to generate mutexed version of a type
Go
2
star
39

httper

implement http interface of a type
Go
2
star
40

channeler

generate channel version of a type
Go
2
star
41

go-fmt-fail

go fmt fails when a file is formatted.
Go
2
star
42

philea

Apply commands on globbed files
Go
2
star
43

foreach

bin util to iterate over a list of separated things and call for an external command `foreach` thing found
Go
2
star
44

launchd-simple-api

Simple api for node to manage services via macosx launchd
JavaScript
2
star
45

sc-simple-api

Simple api for node to manage services via windows sc/NSSM
JavaScript
2
star
46

http-multidir

Serve multiple static directories under the same prefix
Go
2
star
47

systemd-simple-api

Simple api for node to manage services via systemd
JavaScript
2
star
48

aghfabsowecwn

A more complete support of windows elevated commands with Node
JavaScript
2
star
49

dht-hook

push dht announces to http remotes
Go
1
star
50

which-service-manager

Tells which service manager the system is running
JavaScript
1
star
51

yasudo

yet another sudo command helper
JavaScript
1
star
52

go-async

Helper for // execution
Go
1
star
53

node-rp-php

For The Fun ! Using node to reverse proxy php servers. Less than 200 lines of code !
JavaScript
1
star
54

rclone-json

Stream an rclone sync activity as a json object stream.
Go
1
star
55

template-tree-simplifier

simplify a template AST via a serie of transformations
Go
1
star
56

pkg-source-to-pkg-json

Transforms a downloadable package source string into its package.json equivalent
JavaScript
1
star
57

archive

bin to create / extract zip, tgz archives
Go
1
star
58

make-that-ssl-cert

node binary to generate an SSL certificate w/o openssl
JavaScript
1
star
59

http-clienter

generate http client of a type
Go
1
star
60

sp-nsi-sc

Parse service file used by nsi to consume windows sc
JavaScript
1
star
61

ggt

ggt's generator toolbox
Go
1
star
62

c-aghfabsowecwn

Singleton server for aghfabsowecwn
JavaScript
1
star
63

fork

fork a repository in your GOPATH
Go
1
star
64

has-systemd

tells if a system runs systemd service manager
JavaScript
1
star
65

traductions

Translating english papers to my native language (fr)
1
star
66

stringexec

cross platform go lib to execute a command string
Go
1
star
67

vagrant-pack

Tool to pack your local vagrant boxes to a tar.gz file
JavaScript
1
star
68

gen-version-file

Generate a version file
Go
1
star
69

666

Visually display success or failure of a command
Go
1
star
70

boltgrpc

gRPC interface for boltdb
Go
1
star
71

build-them-all

Bin util to build go programs to multiple targets
Go
1
star
72

service-finder

Register a concrete service, then locate it by an interface or its concrete type
Go
1
star
73

guess-path

Guess the path containing the desired packaged resources
Go
1
star
74

ghost

per directory dns delcaration for multi environment work related
1
star
75

tail-fs-readable

A readable stream to tail a file
JavaScript
1
star
76

is-consenting

Tells if consent.exe (uac) is running
JavaScript
1
star
77

which-win

Tells which windows system version is being run
JavaScript
1
star
78

vlc-invoke

Ease vlc invokation accross multiple platforms, extracted from peerflix
JavaScript
1
star
79

uri-to-stream

Transforms an uri string to a stream
JavaScript
1
star
80

sp-systemd-unit

Stream parser for systemd unit files
JavaScript
1
star
81

nssm-prebuilt

Node package for prebuilt nssm
JavaScript
1
star
82

has-sc

tells if a system runs windows sc service manager
JavaScript
1
star
83

sp-nsi-sysv

Parse service file used by nsi to consume sysv
Shell
1
star
84

bonjour-over-http

Http server to find / announce bonjour services
JavaScript
1
star
85

getent

getent to json
JavaScript
1
star
86

windows-net

Execute windows net commands and get JSON responses
JavaScript
1
star
87

upstart-simple-api

Simple api for node to manage services via upstart
JavaScript
1
star
88

monitor-power

A golang daemon to graph the computer power usage into prometheus or expvar.
Go
1
star
89

catf

just cat files
JavaScript
1
star
90

cors-proxy

A proxy to handle CORS on non-compatible CORS websites
JavaScript
1
star
91

write

binary to write a file
JavaScript
1
star
92

npm-pkg-dl

Download a package like npm and make it available into a temporary folder
JavaScript
1
star
93

dscl-users

Macos helper to help manage user and group
JavaScript
1
star
94

vagrant-box-list

List local vagrant boxes
JavaScript
1
star
95

win-uac

Enable / Disable UAC on windows
JavaScript
1
star
96

dscacheutil

dscacheutil to json
JavaScript
1
star
97

nsi

Install a node package as a service
JavaScript
1
star
98

disksinfo

get the list of partitions
Go
1
star
99

http-file-store-webapp

webapp to manage files over http
JavaScript
1
star
100

chkconfig-simple-api

Simple api to manage services via chkconfig sysvinit helper
JavaScript
1
star