• Stars
    star
    3,016
  • Rank 14,318 (Top 0.3 %)
  • Language
    Go
  • License
    MIT License
  • Created about 5 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Pure Go implementation of jq

gojq

CI Status Go Report Card MIT License release pkg.go.dev

Pure Go implementation of jq

This is an implementation of jq command written in Go language. You can also embed gojq as a library to your Go products.

Usage

 $ echo '{"foo": 128}' | gojq '.foo'
128
 $ echo '{"a": {"b": 42}}' | gojq '.a.b'
42
 $ echo '{"id": "sample", "10": {"b": 42}}' | gojq '{(.id): .["10"].b}'
{
  "sample": 42
}
 $ echo '[{"id":1},{"id":2},{"id":3}]' | gojq '.[] | .id'
1
2
3
 $ echo '{"a":1,"b":2}' | gojq '.a += 1 | .b *= 2'
{
  "a": 2,
  "b": 4
}
 $ echo '{"a":1} [2] 3' | gojq '. as {$a} ?// [$a] ?// $a | $a'
1
2
3
 $ echo '{"foo": 4722366482869645213696}' | gojq .foo
4722366482869645213696  # keeps the precision of large numbers
 $ gojq -n 'def fact($n): if $n < 1 then 1 else $n * fact($n - 1) end; fact(50)'
30414093201713378043612608166064768844377641568960512000000000000 # arbitrary-precision integer calculation

Nice error messages.

 $ echo '[1,2,3]' | gojq '.foo & .bar'
gojq: invalid query: .foo & .bar
    .foo & .bar
         ^  unexpected token "&"
 $ echo '{"foo": { bar: [] } }' | gojq '.'
gojq: invalid json: <stdin>
    {"foo": { bar: [] } }
              ^  invalid character 'b' looking for beginning of object key string

Installation

Homebrew

brew install gojq

Zero Install

0install add gojq https://apps.0install.net/utils/gojq.xml

Build from source

go install github.com/itchyny/gojq/cmd/gojq@latest

Docker

docker run -i --rm itchyny/gojq
docker run -i --rm ghcr.io/itchyny/gojq

Difference to jq

  • gojq is purely implemented with Go language and is completely portable. jq depends on the C standard library so the availability of math functions depends on the library. jq also depends on the regular expression library and it makes build scripts complex.
  • gojq implements nice error messages for invalid query and JSON input. The error message of jq is sometimes difficult to tell where to fix the query.
  • gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated.
  • gojq supports arbitrary-precision integer calculation while jq does not; jq loses the precision of large integers when calculation is involved. Note that even with gojq, all mathematical functions, including floor and round, convert integers to floating-point numbers; only addition, subtraction, multiplication, modulo, and division operators (when divisible) keep the integer precision. To calculate floor division of integers without losing the precision, use def idivide($n): (. - . % $n) / $n;. To round down floating-point numbers to integers, use def ifloor: floor | tostring | tonumber;, but note that this function does not work with large floating-point numbers and also loses the precision of large integers.
  • gojq behaves differently than jq in some features, hoping that jq will fix the behaviors in the future. gojq consistently counts by characters (not by bytes) in index, rindex, and indices functions; "οΌ‘οΌ’οΌ“οΌ”οΌ•" | .[index("οΌ“"):] results in "οΌ“οΌ”οΌ•" (jq#1430, jq#1624). gojq supports string indexing; "abcde"[2] (jq#1520). gojq fixes handling files with no newline characters at the end (jq#2374). gojq consistently truncates down floating-point number indices both in indexing ([0] | .[0.5] results in 0), and slicing ([0,1,2] | .[0.5:1.5] results in [0]). gojq parses unary operators with higher precedence than variable binding ([-1 as $x | 1,$x] results in [1,-1] not [-1,-1]) (jq#3053). gojq fixes @base64d to allow binary string as the decoded string (jq#1931). gojq improves time formatting and parsing; deals with %f in strftime and strptime (jq#1409), parses timezone offsets with fromdate and fromdateiso8601 (jq#1053), supports timezone name/offset with %Z/%z in strptime (jq#929, jq#2195), and looks up correct timezone during daylight saving time on formatting with %Z (jq#1912). gojq supports nanoseconds in date and time functions.
  • gojq does not support some functions intentionally; get_jq_origin, get_prog_origin, get_search_list (unstable, not listed in jq document), input_line_number, $__loc__ (performance issue). gojq does not support some flags; --ascii-output, -a (performance issue), --seq (not used commonly), --sort-keys, -S (sorts by default because map[string]any does not keep the order), --unbuffered (unbuffered by default). gojq does not parse JSON extensions supported by jq; NaN, Infinity, and [000]. gojq normalizes floating-point numbers to fit to double-precision floating-point numbers. gojq does not support some regular expression metacharacters, backreferences, look-around assertions, and some flags (regular expression engine differences). gojq does not support BOM (encoding/json does not support this). gojq disallows using keywords for function names (def true: .; true is a confusing query), and module name prefixes in function declarations (using module prefixes like def m::f: .; is undocumented).
  • gojq supports reading from YAML input (--yaml-input) while jq does not. gojq also supports YAML output (--yaml-output). gojq supports @urid format string (jq#798, jq#2261).

Color configuration

The gojq command automatically disables coloring output when the output is not a tty. To force coloring output, specify --color-output (-C) option. When NO_COLOR environment variable is present or --monochrome-output (-M) option is specified, gojq disables coloring output.

Use GOJQ_COLORS environment variable to configure individual colors. The variable is a colon-separated list of ANSI escape sequences of null, false, true, numbers, strings, object keys, arrays, and objects. The default configuration is 90:33:33:36:32:34;1.

Usage as a library

You can use the gojq parser and interpreter from your Go products.

package main

import (
	"fmt"
	"log"

	"github.com/itchyny/gojq"
)

func main() {
	query, err := gojq.Parse(".foo | ..")
	if err != nil {
		log.Fatalln(err)
	}
	input := map[string]any{"foo": []any{1, 2, 3}}
	iter := query.Run(input) // or query.RunWithContext
	for {
		v, ok := iter.Next()
		if !ok {
			break
		}
		if err, ok := v.(error); ok {
			if err, ok := err.(*gojq.HaltError); ok && err.Value() == nil {
				break
			}
			log.Fatalln(err)
		}
		fmt.Printf("%#v\n", v)
	}
}
  • Firstly, use gojq.Parse(string) (*Query, error) to get the query from a string.
    • Use gojq.ParseError to get the error position and token of the parsing error.
  • Secondly, get the result iterator
    • using query.Run or query.RunWithContext
    • or alternatively, compile the query using gojq.Compile and then code.Run or code.RunWithContext. You can reuse the *Code against multiple inputs to avoid compilation of the same query. But for arguments of code.Run, do not give values sharing same data between multiple calls.
    • In either case, you cannot use custom type values as the query input. The type should be []any for an array and map[string]any for a map (just like decoded to an any using the encoding/json package). You can't use []int or map[string]string, for example. If you want to query your custom struct, marshal to JSON, unmarshal to any and use it as the query input.
  • Thirdly, iterate through the results using iter.Next() (any, bool). The iterator can emit an error so make sure to handle it. The method returns true with results, and false when the iterator terminates.
    • The return type is not (any, error) because the iterator may emit multiple errors. The jq and gojq commands stop the iteration on the first error, but the library user can choose to stop the iteration on errors, or to continue until it terminates.
      • In any case, it is recommended to stop the iteration on gojq.HaltError, which is emitted by halt and halt_error functions, although these functions are rarely used. The error implements gojq.ValueError, and if the error value is nil, stop the iteration without handling the error. Technically speaking, we can fix the iterator to terminate on the halting error, but it does not terminate at the moment. The halt function in jq not only stops the iteration, but also terminates the command execution, even if there are still input values. So, gojq leaves it up to the library user how to handle the halting error.
    • Note that the result iterator may emit infinite number of values; repeat(0) and range(infinite). It may stuck with no output value; def f: f; f. Use RunWithContext when you want to limit the execution time.

gojq.Compile allows to configure the following compiler options.

  • gojq.WithModuleLoader allows to load modules. By default, the module feature is disabled. If you want to load modules from the file system, use gojq.NewModuleLoader.
  • gojq.WithEnvironLoader allows to configure the environment variables referenced by env and $ENV. By default, OS environment variables are not accessible due to security reasons. You can use gojq.WithEnvironLoader(os.Environ) if you want.
  • gojq.WithVariables allows to configure the variables which can be used in the query. Pass the values of the variables to code.Run in the same order.
  • gojq.WithFunction allows to add a custom internal function. An internal function can return a single value (which can be an error) each invocation. To add a jq function (which may include a comma operator to emit multiple values, empty function, accept a filter for its argument, or call another built-in function), use LoadInitModules of the module loader.
  • gojq.WithIterFunction allows to add a custom iterator function. An iterator function returns an iterator to emit multiple values. You cannot define both iterator and non-iterator functions of the same name (with possibly different arities). You can use gojq.NewIter to convert values or an error to a gojq.Iter.
  • gojq.WithInputIter allows to use input and inputs functions. By default, these functions are disabled.

Bug Tracker

Report bug at Issues・itchyny/gojq - GitHub.

Author

itchyny (https://github.com/itchyny)

License

This software is released under the MIT License, see LICENSE.

More Repositories

1

lightline.vim

A light and configurable statusline/tabline plugin for Vim
Vim Script
6,540
star
2

calendar.vim

A calendar application for Vim
Vim Script
1,904
star
3

bed

Binary editor written in Go
Go
1,134
star
4

mmv

rename multiple files with editor
Go
724
star
5

vim-cursorword

Underlines the word under the cursor
Vim Script
581
star
6

sjsp

Simple JavaScript Profiler
Haskell
234
star
7

vim-gitbranch

Provides the branch name of the current git repository
Vim Script
199
star
8

rexdep

Roughly extract dependency relation from source code
Go
171
star
9

timefmt-go

Efficient time formatting library (strftime, strptime) for Golang
Go
165
star
10

fillin

fill-in your command and execute
Go
142
star
11

landscape.vim

A colorscheme for Vim
Vim Script
137
star
12

maze

A maze command written in Go
Go
115
star
13

base58-go

Base58 encoding/decoding package and command written in Go
Go
106
star
14

vim-haskell-indent

If the plugin does not work for some syntax, feel free to report to the issue tracker!
Vim Script
105
star
15

thumbnail.vim

A thumbnail-style buffer selector for Vim
Vim Script
101
star
16

volume-go

Cross-platform audio volume control library for Go
Go
72
star
17

rassemble-go

Go implementation of Regexp::Assemble
Go
67
star
18

vim-qfedit

Edit the quickfix/location list freely
Vim Script
65
star
19

miv

Vim plugin manager written in Haskell
Haskell
65
star
20

vim-parenmatch

An efficient alternative to the standard matchparen plugin
Vim Script
59
star
21

json2yaml

An efficient JSON to YAML converter written in Go language
Go
58
star
22

dictionary.vim

Dictionary.app interface for Vim
Vim Script
58
star
23

screensaver.vim

Screensavers for Vim
Vim Script
53
star
24

gojo

Yet another Go implementation of jo
Go
49
star
25

fastinvsqrt

Fast inverse square root in programming languages
Makefile
49
star
26

github-migrator

GitHub repository migrator
Go
46
star
27

vim-highlighturl

URL highlight everywhere
Vim Script
45
star
28

dotfiles

my dotfiles
Vim Script
37
star
29

llvm-brainfuck

Brainfuck compiler based on LLVM API
C++
35
star
30

mkrg

Mackerel graph viewer in terminal
Go
34
star
31

qhs

SQL queries on CSV and TSV files
Haskell
33
star
32

lightline-powerful

Powerful settings for lightline.vim
Vim Script
26
star
33

procout

procout peeks write(2) of another process using ptrace(2), written in Rust
Rust
26
star
34

event-go

Simple synchronous event pub-sub package for Golang
Go
20
star
35

github-better-header

Brings back a better GitHub header
HTML
20
star
36

cam

unix command cam: view images inside terminal
C
20
star
37

vim-winfix

Fix the focus and the size of windows in Vim
Vim Script
18
star
38

setup

DO NOT USE THIS
Shell
15
star
39

pihex-rs

Arbitrary place hexadecimal digits viewer of pi written in Rust
Rust
15
star
40

vim-gof

Vim Script
14
star
41

minivm

C
13
star
42

vim-external

Switch to external applications from Vim
Vim Script
11
star
43

mackerel-plugin-rs

Mackerel plugin helper library for Rust
Rust
11
star
44

jsparser

A JavaScript parser in JavaScript generated by Jison
CoffeeScript
11
star
45

ChromePlayer

A music player for local files, working on Google Chrome
JavaScript
11
star
46

screensaver.c

A clock screensaver in terminal
C
9
star
47

uptime-rs

Multi-platform uptime library for Rust
Rust
9
star
48

astgen-go

interface{} => ast.Node
Go
8
star
49

golang-simple-server-sample

A simple server sample in Go
Go
8
star
50

vim-pdf

pdf filetype plugin for Vim
Vim Script
8
star
51

shell-function-and

shell function: and
8
star
52

vim-grep

The only grep in Vim that I need...
Vim Script
8
star
53

zshhist-go

zsh histfile utility for Go
Go
7
star
54

brainfuck

brainfuck
Brainfuck
7
star
55

bin

My utility executables.
Shell
7
star
56

git-branch-name

Optimally fast branch name command for Git.
C
7
star
57

homebrew-tap

Homebrew formulae
Ruby
6
star
58

mackerel-client-rs

An API client library for Mackerel written in Rust (still in the developing stage; host APIs are not implemented yet)
Rust
6
star
59

minivm-go

Golang implementation of a stack-machine based programming language interpreter
Go
6
star
60

maze-c

unix command maze: generating a maze
C
6
star
61

vim-haskell-sort-import

Sort import statements in Haskell codes
Haskell
6
star
62

mackerel-plugin-battery

Battery plugin for Mackerel
Go
5
star
63

vim-closebuffer

Close buffers in Vim
Vim Script
5
star
64

vim-extracmd

Define extra commands.
Vim Script
5
star
65

unite-preview

A preview plugin for vimfiler, unite
Vim Script
4
star
66

mackerel-client-hs

Mackerel API client in Haskell
Haskell
4
star
67

2bf

2bf - generates a Brainfuck code
C
4
star
68

vim-term

Vim Script
3
star
69

maketen-go

Create 10 from numbers!
Go
3
star
70

fractal

fractal figures in gnuplot
3
star
71

vim-cmdline-ranges

Quickly start/edit cmdline-ranges in Vim
Vim Script
3
star
72

Filter.js

A sample of image processing with JavaScript and canvas
JavaScript
3
star
73

vim-quickrun-lightline-hooks

Vim Script
3
star
74

vim-autoft

Set filetype automatically in Vim
Vim Script
3
star
75

tie

Go
3
star
76

procalive

procalive keeps your process alive
Rust
3
star
77

zsh-auto-fillin

Automatic fillin https://github.com/itchyny/fillin
Shell
3
star
78

s3-cache-action

GitHub Action to save cache files and restore them from Amazon S3
TypeScript
3
star
79

c2bf.hs

Convert C to Brainfuck (not working, deprecated product)
Haskell
2
star
80

homebrew-rexdep

Deprecated in favor of https://github.com/itchyny/homebrew-tap
2
star
81

vim-extra-snippets

My own snippets
Vim Snippet
2
star
82

vim-cmdline-escape

Escape special characters on cmdline
Vim Script
2
star
83

setupfiles-go

Create files and directories easily for tests in Go
Go
2
star
84

bf

bf - executes a Brainfuck code
C
2
star
85

syscall-study

Go
2
star
86

vim-histexclude

Exclude by patterns from the histories.
Vim Script
2
star
87

vim-spellbad-pattern

Register regexps to SpellBad
Vim Script
2
star
88

mackerel-plugin-dice-sh

Dice plugin for Mackerel
Shell
2
star
89

unite-auto-open

starting or opening files action for unite
Vim Script
2
star
90

mtimedir

Go
2
star
91

sbt-compile-warn

Aggregate sbt compile warnings
Go
2
star
92

itchyny

itchyny's profile page
2
star
93

codeforces

Codeforces in Haskell
Haskell
2
star
94

browsershell

A shell in your browser
JavaScript
2
star
95

autolatex

A shell script to compile a LaTeX file
Shell
2
star
96

formulate

Homebrew formula managing script
Shell
2
star
97

unite-changetime

An action for unite/vimfiler, changing modified time of a file
Vim Script
1
star
98

mmpp

Mackerel metric pretty printer
Rust
1
star
99

atcoder

AtCoder in Rust
Rust
1
star
100

mackerel-plugin-dice-rs

Dice plugin example using mackerel-plugin-rs
Rust
1
star