miku/zek miku/zek

XML Processing
  • Stars
    star
    668
  • Rank 67,542 (Top 2 %)
  • Language
    Go
  • License
    GNU General Publi...
  • Created almost 7 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
miku/zek
github

miku

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Generate a Go struct from XML.

zek

Zek is a prototype for creating a Go struct from an XML document. The resulting struct works best for reading XML (see also #14), to create XML, you might want to use something else.

It was developed at Leipzig University Library to shorten the time to go from raw XML to a struct that allows to access XML data in Go programs.

Skip the fluff, just the code.

Given some XML, run:

$ curl -s https://raw.githubusercontent.com/miku/zek/master/fixtures/e.xml | zek -e
// Rss was generated 2018-08-30 20:24:14 by tir on sol.
type Rss struct {
    XMLName xml.Name `xml:"rss"`
    Text    string   `xml:",chardata"`
    Rdf     string   `xml:"rdf,attr"`
    Dc      string   `xml:"dc,attr"`
    Geoscan string   `xml:"geoscan,attr"`
    Media   string   `xml:"media,attr"`
    Gml     string   `xml:"gml,attr"`
    Taxo    string   `xml:"taxo,attr"`
    Georss  string   `xml:"georss,attr"`
    Content string   `xml:"content,attr"`
    Geo     string   `xml:"geo,attr"`
    Version string   `xml:"version,attr"`
    Channel struct {
        Text          string `xml:",chardata"`
        Title         string `xml:"title"`         // ESS New Releases (Display...
        Link          string `xml:"link"`          // http://tinyurl.com/ESSNew...
        Description   string `xml:"description"`   // New releases from the Ear...
        LastBuildDate string `xml:"lastBuildDate"` // Mon, 27 Nov 2017 00:06:35...
        Item          []struct {
            Text        string `xml:",chardata"`
            Title       string `xml:"title"`       // Surficial geology, Aberde...
            Link        string `xml:"link"`        // https://geoscan.nrcan.gc....
            Description string `xml:"description"` // Geological Survey of Cana...
            Guid        struct {
                Text        string `xml:",chardata"` // 304279, 306212, 306175, 3...
                IsPermaLink string `xml:"isPermaLink,attr"`
            } `xml:"guid"`
            PubDate       string   `xml:"pubDate"`      // Fri, 24 Nov 2017 00:00:00...
            Polygon       []string `xml:"polygon"`      // 64.0000 -98.0000 64.0000 ...
            Download      string   `xml:"download"`     // https://geoscan.nrcan.gc....
            License       string   `xml:"license"`      // http://data.gc.ca/eng/ope...
            Author        string   `xml:"author"`       // Geological Survey of Cana...
            Source        string   `xml:"source"`       // Geological Survey of Cana...
            SndSeries     string   `xml:"SndSeries"`    // Bedford Institute of Ocea...
            Publisher     string   `xml:"publisher"`    // Natural Resources Canada,...
            Edition       string   `xml:"edition"`      // prelim., surficial data m...
            Meeting       string   `xml:"meeting"`      // Geological Association of...
            Documenttype  string   `xml:"documenttype"` // serial, open file, serial...
            Language      string   `xml:"language"`     // English, English, English...
            Maps          string   `xml:"maps"`         // 1 map, 5 maps, Publicatio...
            Mapinfo       string   `xml:"mapinfo"`      // surficial geology, surfic...
            Medium        string   `xml:"medium"`       // on-line; digital, digital...
            Province      string   `xml:"province"`     // Nunavut, Northwest Territ...
            Nts           string   `xml:"nts"`          // 066B, 095J; 095N; 095O; 0...
            Area          string   `xml:"area"`         // Aberdeen Lake, Mackenzie ...
            Subjects      string   `xml:"subjects"`
            Program       string   `xml:"program"`       // GEM2: Geo-mapping for Ene...
            Project       string   `xml:"project"`       // Rae Province Project Mana...
            Projectnumber string   `xml:"projectnumber"` // 340521, 343202, 340557, 3...
            Abstract      string   `xml:"abstract"`      // This new surficial geolog...
            Links         string   `xml:"links"`         // Online - En ligne (PDF, 9...
            Readme        string   `xml:"readme"`        // readme | https://geoscan....
            PPIid         string   `xml:"PPIid"`         // 34532, 35096, 35438, 2563...
        } `xml:"item"`
    } `xml:"channel"`
}

Online

About

Project Status: Active – The project has reached a stable, usable state and is being actively developed.

Upsides:

  • it works fine for non-recursive structures,
  • does not need XSD or DTD,
  • it is relatively convenient to access attributes, children and text,
  • will generate a single struct, which make for a quite compact representation,
  • simple user interface,
  • comments with examples,
  • schema inference across multiple files.

Downsides:

  • experimental, early, buggy, unstable prototype,
  • no support for recursive types (similar to Russian Doll strategy, [1])
  • no type inference, everything is accessible as string (without a schema, type inference may fail if the type guess is wrong)

Bugs:

Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.

https://golang.org/pkg/encoding/xml/#pkg-note-BUG

Related projects:

And other awesome XML utilities.

Presentations:

Install

$ go install github.com/miku/zek/cmd/zek@latest

Debian and RPM packages:

It's in AUR, too.

Usage

$ zek -h
Usage of zek:
  -B    use a fixed banner string (e.g. for CI)
  -C    emit less compact struct
  -F    skip formatting
  -P string
        if set, write out struct within a package with the given name
  -S int
        read at most this many tags, approximately (0=unlimited)
  -c    emit more compact struct (noop, as this is the default since 0.1.7)
  -d    debug output
  -e    add comments with example
  -j    add JSON tags
  -m    omit empty Text fields
  -max-examples int
        limit number of examples (default 10)
  -n string
        use a different name for the top-level struct
  -o string
        if set, write to output file, not stdout
  -p    write out an example program
  -s    strict parsing and writing
  -t string
        emit struct for tag matching this name
  -u    filter out duplicated examples
  -version
        show version
  -x int
        max chars for example (default 25)

Examples:

$ cat fixtures/a.xml
<a></a>

$ zek -C < fixtures/a.xml
type A struct {
    XMLName xml.Name `xml:"a"`
    Text    string   `xml:",chardata"`
}

Debug output dumps the internal tree as JSON to stdout.

$ zek -d < fixtures/a.xml
{"name":{"Space":"","Local":"a"}}

Example program:

package main

import (
	"encoding/json"
	"encoding/xml"
	"fmt"
	"log"
	"os"
)

// A was generated 2017-12-05 17:35:21 by tir on apollo.
type A struct {
	XMLName xml.Name `xml:"a"`
	Text    string   `xml:",chardata"`
}

func main() {
	dec := xml.NewDecoder(os.Stdin)
	var doc A
	if err := dec.Decode(&doc); err != nil {
		log.Fatal(err)
	}
	b, err := json.Marshal(doc)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(b))
}

$ zek -C -p < fixtures/a.xml > sample.go && go run sample.go < fixtures/a.xml | jq . && rm sample.go
{
  "XMLName": {
    "Space": "",
    "Local": "a"
  },
  "Text": ""
}

More complex example:

$ zek < fixtures/d.xml
// Root was generated 2019-06-11 16:27:04 by tir on hayiti.
type Root struct {
        XMLName xml.Name `xml:"root"`
        Text    string   `xml:",chardata"`
        A       []struct {
                Text string `xml:",chardata"`
                B    []struct {
                        Text string `xml:",chardata"`
                        C    string `xml:"c"`
                        D    string `xml:"d"`
                } `xml:"b"`
        } `xml:"a"`
}

$ zek -p < fixtures/d.xml > sample.go && go run sample.go < fixtures/d.xml | jq . && rm sample.go
{
  "XMLName": {
    "Space": "",
    "Local": "root"
  },
  "Text": "\n\n\n\n",
  "A": [
    {
      "Text": "\n  \n  \n",
      "B": [
        {
          "Text": "\n    \n  ",
          "C": "Hi",
          "D": ""
        },
        {
          "Text": "\n    \n    \n  ",
          "C": "World",
          "D": ""
        }
      ]
    },
    {
      "Text": "\n  \n",
      "B": [
        {
          "Text": "\n    \n  ",
          "C": "Hello",
          "D": ""
        }
      ]
    },
    {
      "Text": "\n  \n",
      "B": [
        {
          "Text": "\n    \n  ",
          "C": "",
          "D": "World"
        }
      ]
    }
  ]
}

Annotate with comments:

$ zek -e < fixtures/l.xml
// Records was generated 2019-06-11 16:29:35 by tir on hayiti.
type Records struct {
        XMLName xml.Name `xml:"Records"`
        Text    string   `xml:",chardata"` // \n
        Xsi     string   `xml:"xsi,attr"`
        Record  []struct {
                Text   string `xml:",chardata"`
                Header struct {
                        Text       string `xml:",chardata"`
                        Status     string `xml:"status,attr"`
                        Identifier string `xml:"identifier"` // oai:ojs.localhost:article...
                        Datestamp  string `xml:"datestamp"`  // 2009-06-24T14:48:23Z, 200...
                        SetSpec    string `xml:"setSpec"`    // eppp:ART, eppp:ART, eppp:...
                } `xml:"header"`
                Metadata struct {
                        Text    string `xml:",chardata"`
                        Rfc1807 struct {
                                Text           string   `xml:",chardata"`
                                Xmlns          string   `xml:"xmlns,attr"`
                                Xsi            string   `xml:"xsi,attr"`
                                SchemaLocation string   `xml:"schemaLocation,attr"`
                                BibVersion     string   `xml:"bib-version"`  // v2, v2, v2...
                                ID             string   `xml:"id"`           // http://jou...
                                Entry          string   `xml:"entry"`        // 2009-06-24...
                                Organization   []string `xml:"organization"` // Proceeding...
                                Title          string   `xml:"title"`        // Introducti...
                                Type           string   `xml:"type"`
                                Author         []string `xml:"author"`       // KRAMPEN, G..
                                Copyright      string   `xml:"copyright"`    // Das Urhebe...
                                OtherAccess    string   `xml:"other_access"` // url:http:/...
                                Keyword        string   `xml:"keyword"`
                                Period         []string `xml:"period"`
                                Monitoring     string   `xml:"monitoring"`
                                Language       string   `xml:"language"` // en, en, en, e...
                                Abstract       string   `xml:"abstract"` // After a short...
                                Date           string   `xml:"date"`     // 2009-06-22 12...
                        } `xml:"rfc1807"`
                } `xml:"metadata"`
                About string `xml:"about"`
        } `xml:"Record"`
}

Only consider a nested element

$ zek -t metadata fixtures/z.xml
// Metadata was generated 2019-06-11 16:33:26 by tir on hayiti.
type Metadata struct {
        XMLName xml.Name `xml:"metadata"`
        Text    string   `xml:",chardata"`
        Dc      struct {
                Text  string `xml:",chardata"`
                Xmlns string `xml:"xmlns,attr"`
                Title struct {
                        Text  string `xml:",chardata"`
                        Xmlns string `xml:"xmlns,attr"`
                } `xml:"title"`
                Identifier struct {
                        Text  string `xml:",chardata"`
                        Xmlns string `xml:"xmlns,attr"`
                } `xml:"identifier"`
                Rights struct {
                        Text  string `xml:",chardata"`
                        Xmlns string `xml:"xmlns,attr"`
                        Lang  string `xml:"lang,attr"`
                } `xml:"rights"`
                AccessRights struct {
                        Text  string `xml:",chardata"`
                        Xmlns string `xml:"xmlns,attr"`
                } `xml:"accessRights"`
        } `xml:"dc"`
}

Inference across files

$ zek fixtures/a.xml fixtures/b.xml fixtures/c.xml
// A was generated 2017-12-05 17:40:14 by tir on apollo.
type A struct {
	XMLName xml.Name `xml:"a"`
	Text    string   `xml:",chardata"`
	B       []struct {
		Text string `xml:",chardata"`
	} `xml:"b"`
}

This is also useful, if you deal with archives containing XML files:

$ unzip -p 4082359.zip '*.xml' | zek -e

Given a directory full of zip files, you can combined find, unzip and zek:

$ for i in $(find ftp/b571 -type f -name "*zip"); do unzip -p $i '*xml'; done | zek -e

Another example (tarball with thousands of XML files, seemingly MARC):

$ tar -xOzf /tmp/20180725.125255.tar.gz | zek -e
// OAIPMH was generated 2018-09-26 15:03:29 by tir on sol.
type OAIPMH struct {
        XMLName        xml.Name `xml:"OAI-PMH"`
        Text           string   `xml:",chardata"`
        Xmlns          string   `xml:"xmlns,attr"`
        Xsi            string   `xml:"xsi,attr"`
        SchemaLocation string   `xml:"schemaLocation,attr"`
        ListRecords    struct {
                Text   string `xml:",chardata"`
                Record struct {
                        Text   string `xml:",chardata"`
                        Header struct {
                                Text       string `xml:",chardata"`
                                Identifier struct {
                                        Text string `xml:",chardata"` // aleph-pub:000000001, ...
                                } `xml:"identifier"`
                        } `xml:"header"`
                        Metadata struct {
                                Text   string `xml:",chardata"`
                                Record struct {
                                        Text           string `xml:",chardata"`
                                        Xmlns          string `xml:"xmlns,attr"`
                                        Xsi            string `xml:"xsi,attr"`
                                        SchemaLocation string `xml:"schemaLocation,attr"`
                                        Leader         struct
                                                Text string `xml:",chardata"` // 00001nM2.01200024
                                        } `xml:"leader"`
                                        Controlfield []struct {
                                                Text string `xml:",chardata"` // 00001nM2.01200024
                                                Tag  string `xml:"tag,attr"`
                                        } `xml:"controlfield"`
                                        Datafield []struct {
                                                Text     string `xml:",chardata"`
                                                Tag      string `xml:"tag,attr"`
                                                Ind1     string `xml:"ind1,attr"`
                                                Ind2     string `xml:"ind2,attr"`
                                                Subfield []struct {
                                                        Text string `xml:",chardata"` // KM0000002
                                                        Code string `xml:"code,attr"`
                                                } `xml:"subfield"`
                                        } `xml:"datafield"`
                                } `xml:"record"`
                        } `xml:"metadata"`
                } `xml:"record"`
        } `xml:"ListRecords"`
}

Generate a package

If you want in include generated file in the build process, e.g. with go generate, you may find -P and -o helpful.

$ cat fixtures/b.xml
<a><b></b></a>

Run on the command line or via go generate:

$ zek -P mypkg -o data.go < fixtures/b.xml

This would write out the following in data.go file:

// Code generated by zek; DO NOT EDIT.

package mypkg

import "encoding/xml"

// A was generated 2021-09-16 11:23:06 by tir on trieste.
type A struct {
        XMLName xml.Name `xml:"a"`
        Text    string   `xml:",chardata"`
        B       string   `xml:"b"`
}

Note that any existing file will be overwritten, without any warning.

Misc

As a side effect, zek seems to be a useful for debugging. Example:

This record is emitted from a typical OAI server (OJS, not even uncommon), yet one can quickly spot the flaw in the structure.

Over 30 different struct generated manually in the course of a few hours (around five minutes per source): https://git.io/vbTDo.

-- Current extent leader: 1532 lines struct

More Repositories

1

esbulk

Bulk indexing command line tool for elasticsearch
Go
269
star
2

metha

Command line OAI harvester and client with built-in cache,
Go
102
star
3

binpic

Create a picture from any file.
Go
89
star
4

microblob

Serve millions of JSON documents via HTTP.
Go
65
star
5

gluish

Utils around luigi.
Python
63
star
6

xmlcutty

Select elements from large XML files, fast.
Go
52
star
7

solrbulk

SOLR bulk indexing utility for the command line.
Go
41
star
8

estab

Export elasticsearch as TSV or line delimited JSON.
Go
36
star
9

haystack

Haystack and seaweedfs lightning talk.
C
25
star
10

pgrk

Command line pagerank computation with Go.
Go
20
star
11

siskin

Tasks around metadata.
Python
20
star
12

parallel

Process lines in parallel.
Go
17
star
13

exploreio

Explore IO with Golang, workshop at Golab 2017
Go
17
star
14

mlgo

Machine Learning with Go (golang) Session Material for GOLAB 2018
Makefile
17
star
15

stardust

stardust, strdist. String distance and similarity measures for the command line.
Go
16
star
16

span

Span formats.
Go
15
star
17

dwstalk

A data web service, lightning talk.
15
star
18

kat

Kat is like Preview.app for the command-line.
Go
15
star
19

filterline

Command line tool to filter file by line number.
C
12
star
20

brew-completion

bash tab completion for homebrew package manager
Shell
11
star
21

activememory

A page to test short term memory.
JavaScript
11
star
22

ntto

Small n-triples to line delimited JSON converter and prefix cutter.
Go
11
star
23

issnlister

List of valid, registered ISSN
Python
10
star
24

nntour

Neural nets intro @lpyug
Python
10
star
25

esdump

Stream documents from elasticsearch with scroll (and HTTP GET only)
Go
9
star
26

rsampling

Reservoir sampling for the command line.
Go
8
star
27

workshops

A level of indirection.
7
star
28

batchdata

Batch data processing with luigi, 90min workshop at PyCon Balkan 2018, Belgrade.
Python
7
star
29

jquery-retype

Your friendly javascript keylogger.
JavaScript
7
star
30

dcdump

Datacite API bulk access.
Go
7
star
31

goforprogrammers

Go for Programmers, Spartakiade 2021
Go
6
star
32

cignotes

Notes on Concurrency in Go
Go
6
star
33

marc21

A MARC21 library for Go.
Go
6
star
34

clinker

Dumb link checker.
Go
6
star
35

golangintro

A one day introductory Golang workshop at http://devopenspace.de 2018
Go
6
star
36

go4x4

Go materials for a set of 4x4 sessions.
HTML
6
star
37

glamline

Glamorous command line
Go
5
star
38

isbngrep

Command line ISBN sniffer and normalizer.
Go
5
star
39

oaimi

No frills OAI PMH harvesting for the command line.
Go
5
star
40

makta

Create an sqlite3 database from tabular data (2-TSV).
Go
5
star
41

cachetools

Various Python caching, pickling and memoization functions.
5
star
42

oaicrawl

OAI crawler for strange endpoints.
Go
5
star
43

wikitools

Few tools for working with wikipedia XML dumps.
Go
5
star
44

unzippa

Unzip selected members from a zipfile 150x faster than unzip.
Go
4
star
45

urlbisect

For URLs with autoincrement ids, find the highest number using binary search.
Go
4
star
46

jsoninf

JSON schema inference
Go
4
star
47

clam

A templated shell helper library.
Go
4
star
48

es-hf-2014-05-28

Experimenting with the Elasticsearch completion suggester during elasticsearch hackfest.
JavaScript
4
star
49

jpul

Jobportal Uni Leipzig
PHP
4
star
50

kollektions

kollektions
Python
3
star
51

productivego

Three reasons why go is fun to work with (even after seven years).
Makefile
3
star
52

fuzzycat

Fuzzy matching publications for fatcat (wip).
Python
3
star
53

lpug-luigi

Material from luigi presentation at LPUG meeting on 10/11/2015.
Python
3
star
54

concgo

Concurrency in Go workshop, GOLAB 2019
HTML
3
star
55

aboutgo

Materials for learning and teaching various Go topics.
Go
3
star
56

flask-gae-stub

Google App Engine Flask Stub.
Python
3
star
57

packpy

Python packaging notes for PyCon Balkan 2018.
mIRC Script
3
star
58

goexp

Go Expedition
HTML
3
star
59

hurrly

Hurry, hurrly!
Go
3
star
60

httpgetaway

HTTP GETAWAY - clients and hops, transports and timeouts.
Go
3
star
61

ottily

Ottily executes a javascript snippet on each line of an input file in parallel.
Go
3
star
62

pyflow

Advanced Python concepts and examples.
mIRC Script
3
star
63

blobproc

Webhook server that can receive raw bytes and execute commands.
Go
2
star
64

grobidclient

A Go (golang) client for GROBID.
Go
2
star
65

memcmarc

Load/Set MARC records into memcache.
Go
2
star
66

khwarizmi

Python
2
star
67

groupcover

Like uniq, but worse.
Go
2
star
68

benchtrie

Benchmarking name lookups.
Go
2
star
69

rarara

Prime buffer cache for a file via readahead from the command line (linux only).
C
2
star
70

tableau

Data and Feedback.
JavaScript
2
star
71

istools

Finc Intermediate Schema tools (linter, licensing)
Go
2
star
72

ldjtab

Extract values and line numbers from LDJ files.
Go
2
star
73

ttarc

Minimalistic TikTok trending archiver.
HTML
2
star
74

padsync

Tracking etherpads in git repositories.
Go
2
star
75

io15min

Lightning talk about the io package and its interfaces.
2
star
76

waste

A cat in a container service.
Go
2
star
77

memcldj

Load JSON blobs into memcache or memcachedb quickly.
Go
2
star
78

dvmapp

Server (prototype) for Project Die Virtuelle Mittagsfrau (defunkt)
Go
2
star
79

goenergy

Go energy lightning talk
2
star
80

creativejupyter

Creative Jupyter, PyCon Balkan 2019
2
star
81

elasticsearch-slides

JavaScript
2
star
82

gndzero

GND cache. Zeroth prototype.
Python
2
star
83

marctojson

Command line MARC to JSON converter.
Java
2
star
84

magento-tooling

Small magento analgesics.
2
star
85

goai

Go OAI.
Go
2
star
86

sundaypython

Input session for Coding da Vinci Ost 2018: Python 101 for data processing.
Jupyter Notebook
2
star
87

s2gen

Generate code for representing SOLR documents in Go from schema.xml file.
Go
2
star
88

runpad

Run code from an etherpad
Go
2
star
89

solrcount

A proxy for solr requests, that will only reveal the number of results.
Go
2
star
90

sitemapped

Export all URLs from a sitemap
Go
2
star
91

scholkit

Assorted utitlies around scholarly metadata.
Go
2
star
92

datasets

Lists of datasets and dataset lists.
1
star
93

gows

Go workshop notes.
HTML
1
star
94

zeromq-slides

JavaScript
1
star
95

libai

Assorted notes on libraries and AI
1
star
96

golang6h

Golang in six hours. Language tour and tooling.
Go
1
star
97

picourse

A WIP Raspberry Pi and Python course.
1
star
98

vcprompt

Imported from https://bitbucket.org/gward/vcprompt
C
1
star
99

marc22

An experimental fork of marc21.
Go
1
star
100

evreg

JavaScript
1
star