• Stars
    star
    516
  • Rank 85,726 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created over 11 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

PAWK - A Python line processor (like AWK)

PAWK - A Python line processor (like AWK)

PAWK aims to bring the full power of Python to AWK-like line-processing.

Here are some quick examples to show some of the advantages of pawk over AWK.

The first example transforms /etc/hosts into a JSON map of host to IP:

cat /etc/hosts | pawk -B 'd={}' -E 'json.dumps(d)' '!/^#/ d[f[1]] = f[0]'

Breaking this down:

  1. -B 'd={}' is a begin statement initializing a dictionary, executed once before processing begins.
  2. -E 'json.dumps(d)' is an end statement expression, producing the JSON representation of the dictionary d.
  3. !/^#/ tells pawk to match any line not beginning with #.
  4. d[f[1]] = f[0] adds a dictionary entry where the key is the second field in the line (the first hostname) and the value is the first field (the IP address).

And another example showing how to bzip2-compress + base64-encode a file:

cat pawk.py | pawk -E 'base64.encodestring(bz2.compress(t))'

AWK example translations

Most basic AWK constructs are available. You can find more idiomatic examples below in the example section, but here are a bunch of awk commands and their equivalent pawk commands to get started with:

Print lines matching a pattern:

ls -l / | awk '/etc/'
ls -l / | pawk '/etc/'

Print lines not matching a pattern:

ls -l / | awk '!/etc/'
ls -l / | pawk '!/etc/'

Field slicing and dicing (here pawk wins because of Python's array slicing):

ls -l / | awk '/etc/ {print $5, $6, $7, $8, $9}'
ls -l / | pawk '/etc/ f[4:]'

Begin and end end actions (in this case, summing the sizes of all files):

ls -l | awk 'BEGIN {c = 0} {c += $5} END {print c}'
ls -l | pawk -B 'c = 0' -E 'c' 'c += int(f[4])'

Print files where a field matches a numeric expression (in this case where files are > 1024 bytes):

ls -l | awk '$5 > 1024'
ls -l | pawk 'int(f[4]) > 1024'

Matching a single field (any filename with "t" in it):

ls -l | awk '$NF ~/t/'
ls -l | pawk '"t" in f[-1]'

Installation

It should be as simple as:

pip install pawk

But if that doesn't work, just download the pawk.py, make it executable, and place it somewhere in your path.

Expression evaluation

PAWK evaluates a Python expression or statement against each line in stdin. The following variables are available in local context:

  • line - Current line text, including newline.
  • l - Current line text, excluding newline.
  • n - The current 1-based line number.
  • f - Fields of the line (split by the field separator -F).
  • nf - Number of fields in this line.
  • m - Tuple of match regular expression capture groups, if any.

In the context of the -E block:

  • t - The entire input text up to the current cursor position.

If the flag -H, --header is provided, each field in the first row of the input will be treated as field variable names in subsequent rows. The header is not output. For example, given the input:

count name
12 bob
34 fred

We could do:

$ pawk -H '"%s is %s" % (name, count)' < input.txt
bob is 12
fred is 34

To output a header as well, use -B:

$ pawk -H -B '"name is count"' '"%s is %s" % (name, count)' < input.txt
name is count
bob is 12
fred is 34

Module references will be automatically imported if possible. Additionally, the --import <module>[,<module>,...] flag can be used to import symbols from a set of modules into the evaluation context.

eg. --import os.path will import all symbols from os.path, such as os.path.isfile(), into the context.

Output

Line actions

The type of the evaluated expression determines how output is displayed:

  • tuple or list: the elements are converted to strings and joined with the output delimiter (-O).
  • None or False: nothing is output for that line.
  • True: the original line is output.
  • Any other value is converted to a string.

Start/end blocks

The rules are the same as for line actions with one difference. Because there is no "line" that corresponds to them, an expression returning True is ignored.

$ echo -ne 'foo\nbar' | pawk -E t
foo
bar

Command-line usage

Usage: cat input | pawk [<options>] <expr>

A Python line-processor (like awk).

See https://github.com/alecthomas/pawk for details. Based on
http://code.activestate.com/recipes/437932/.

Options:
  -h, --help            show this help message and exit
  -I <filename>, --in_place=<filename>
                        modify given input file in-place
  -i <modules>, --import=<modules>
                        comma-separated list of modules to "from x import *"
                        from
  -F <delim>            input delimiter
  -O <delim>            output delimiter
  -L <delim>            output line separator
  -B <statement>, --begin=<statement>
                        begin statement
  -E <statement>, --end=<statement>
                        end statement
  -s, --statement       DEPRECATED. retained for backward compatibility
  -H, --header          use first row as field variable names in subsequent
                        rows
  --strict              abort on exceptions

Examples

Line processing

Print the name and size of every file from stdin:

find . -type f | pawk 'f[0], os.stat(f[0]).st_size'

Note: this example also shows how pawk automatically imports referenced modules, in this case os.

Print the sum size of all files from stdin:

find . -type f | \
	pawk \
		--begin 'c=0' \
		--end c \
		'c += os.stat(f[0]).st_size'

Short-flag version:

find . -type f | pawk -B c=0 -E c 'c += os.stat(f[0]).st_size'

Whole-file processing

If you do not provide a line expression, but do provide an end statement, pawk will accumulate each line, and the entire file's text will be available in the end statement as t. This is useful for operations on entire files, like the following example of converting a file from markdown to HTML:

cat README.md | \
	pawk --end 'markdown.markdown(t)'

Short-flag version:

cat README.md | pawk -E 'markdown.markdown(t)'

More Repositories

1

chroma

A general purpose syntax highlighter in pure Go
Go
4,360
star
2

gometalinter

DEPRECATED: Use https://github.com/golangci/golangci-lint
Go
3,507
star
3

kingpin

CONTRIBUTIONS ONLY: A Go (golang) command line and flag parser
Go
3,497
star
4

participle

A parser library for Go
Go
3,479
star
5

entityx

EntityX - A fast, type-safe C++ Entity-Component system
C++
2,170
star
6

kong

Kong is a command-line parser for Go
Go
2,132
star
7

voluptuous

CONTRIBUTIONS ONLY: Voluptuous, despite the name, is a Python data validation library.
Python
1,793
star
8

go_serialization_benchmarks

Benchmarks of Go serialization methods
Go
1,560
star
9

jsonschema

Maintenance has moved to https://github.com/invopop/jsonschema
Go
750
star
10

gozmq

Go (golang) bindings for the 0mq (zmq, zeromq) C API
Go
468
star
11

log4go

Logging package similar to log4j for the Go programming language
Go
309
star
12

ondir

OnDir is a small program to automate tasks specific to certain directories
C
195
star
13

mph

Minimal Perfect Hashing for Go
Go
173
star
14

repr

Python's repr() for Go
Go
163
star
15

assert

A simple assertion library using Go generics
Go
147
star
16

units

Helpful unit multipliers and functions for Go
Go
123
star
17

importmagic

A Python library for finding unresolved symbols in Python code, and the corresponding imports
Python
120
star
18

gorx

A package and tool providing Reactive eXtensions for Go.
Go
94
star
19

devtodo2

DevTodo the Second
Go
89
star
20

template

Fork of Go's text/template adding newline elision
Go
56
star
21

hcl

Parsing, encoding and decoding of HCL to and from Go types and an AST.
Go
49
star
22

binary

General purpose binary encoder/decoder
Go
48
star
23

SublimeLinter-contrib-gometalinter

SublimeLinter plugin for gometalinter
Python
47
star
24

localcache

Local file-based atomic cache manager
Go
44
star
25

gobundle

DEPRECATED: I recommend https://github.com/GeertJohan/go.rice
Go
39
star
26

geoip

A pure Go interface to the free MaxMind GeoIP database
Go
38
star
27

unsafeslice

Unsafe zero-copy slice casts for Go
Go
37
star
28

SublimePythonImportMagic

This Sublime Text 2 plugin attempts to automatically manage Python imports.
Python
34
star
29

inject

Guice-ish dependency injection for Go.
Go
31
star
30

sequel

Sequel - A Go <-> SQL mapping package
Go
26
star
31

multiplex

This Go package multiplexes streams over a single underlying transport io.ReadWriteCloser.
Go
25
star
32

arena

A very fast arena allocator for Go
Go
22
star
33

tuplespace

A RESTful tuple space server
Go
21
star
34

langx

Language experimentation.
Go
21
star
35

mango-kong

Mango (man page generator) integration for Kong
Go
20
star
36

go-check-sumtype

A simple utility for running exhaustiveness checks on Go "sum types."
Go
20
star
37

atomic

Type-safe atomic values for Go
Go
19
star
38

go-rpcgen

Generates Go RPC server and client boilerplate for interfaces.
Go
17
star
39

SublimeFoldPythonDocstrings

Automatically folds Python docstrings longer than 1 line.
Python
16
star
40

colour

Quake-style colour formatting for Unix terminals
Go
15
star
41

protobuf

A Protobuf IDL parser for Go
Go
15
star
42

oink

Oink is a Python to Javascript translator.
Python
15
star
43

entityx_python

Python bindings for EntityX
C++
14
star
44

kong-yaml

Go
14
star
45

bit

Bit - A simple yet powerful build tool
Go
12
star
46

shreq

This utility verifies all commands used by a shell script against an allow list
Go
11
star
47

types

Useful generic types for Go
Go
11
star
48

app

Modular application framework for Go.
Go
11
star
49

kdl

Go parser for KDL
Go
10
star
50

vheap

Fast, persistent, mmapped, virtual heap.
Go
8
star
51

errors

A simple errors package for Go
Go
8
star
52

kong-hcl

Go
8
star
53

rapid

RESTful API Daemons (and Clients) for Go
Go
7
star
54

genh

genh is an opinionated tool for generating request-handler boilerplate for Go
Go
7
star
55

ReactiveDataStructures

Reactive data structures for Swift based on RxSwift
Swift
7
star
56

lunatic-go

Lunatic bindings for (Tiny)Go
Go
6
star
57

chrysalis

Chrysalis - Source to a 2D Platformer from 1994
C++
6
star
58

dotfiles

My dotfiles.
Vim Script
6
star
59

bootstrap

Go application bootstrapping
Go
6
star
60

devtodo

DevTodo (legacy)
C
6
star
61

waffle

Waffle - A Dependency-Injection-based application framework for Python
Python
5
star
62

porpoise

Porpoise - A Redis-based analytics framework
Python
5
star
63

waitgroup

Like sync.WaitGroup and ergroup.Group had a baby.
Go
5
star
64

esfmt

An opinionated, zero-configuration formatter for ES/TS/ESX/TSX
Go
5
star
65

flam

flam /flæm/ noun, verb, flammed, flam⋅ming. Informal. –noun 1. a deception or trick. 2. a falsehood; lie. –verb (used with object), verb (used without object) 3. to deceive; delude; cheat.
Python
5
star
66

cly

A Python module for adding powerful text-based consoles to your application.
Python
4
star
67

expr

Runtime evaluation of Go-like expressions
Go
4
star
68

simplenotefs

simplenotefs
Python
4
star
69

concurrency

Types and functions for managing concurrency in Go.
Go
4
star
70

replaylog

A type safe implementation of an op replay log
Go
3
star
71

Cache.swift

A flexible RAM and disk-backed cache for Swift
Swift
3
star
72

wit-go

A partial WIT parser and code generator for Go
Go
3
star
73

SublimeLinter-contrib-errcheck

SublimeLinter integration for the Go errcheck utility
Python
3
star
74

SublimeLinter-contrib-golang-cilint

DEPRECATED: Use https://github.com/cixtor/SublimeLinter-golangcilint
Python
2
star
75

aspect

Lightweight Aspect-oriented Module for Python
Python
2
star
76

gptcc

Add Conventional Commits to commit messages using ChatGPT
Shell
2
star
77

WaveGrowl.app

Wave notifications via Growl on Mac
Python
2
star
78

cut

Core Utilities - A set of core utility classes for Python.
Python
2
star
79

kong-toml

Kong configuration loader for TOML
Shell
2
star
80

rest

Go
2
star
81

prototemplate

Process Protocol Buffer definitions with text templates and JavaScript functions
Go
2
star
82

webservice

A webservice dispatcher for Go
Go
1
star
83

pathways

Pathways - An opinionated RESTful web service framework for Go
Go
1
star
84

cktphotography.com

Christine Knight Thomas Photography (website)
JavaScript
1
star
85

psmap

Persistent static maps for Go
Go
1
star