poline
poline lets you do awk-like one liners in python.
For example, the following will graph the number of connections for each hosts.
netstat -an | grep ESTABLISHED | pol "|url(_4).hostname" "counter(_)" ":x, c: Cols(17,40).f(x,'*' * c)"Example output:
216.58.193.78 *****
74.125.199.189 ****
198.252.206.25 ***
74.125.199.188 **
192.30.253.125 **
127.0.0.1 **
216.58.193.67 *
The equivalent awk version can be found on commandlinefu.
Installation
poline is compatible with Python 2.7 to Python 3.6.
You can easily install poline is via pip:
pip install poline
or
pip<version> install poline
Where <version> is the version of python you wish to use. For example, if you want to use poline with Python 3.6 installed, you should use
pip3.6 install poline
Usage
pol [-h] [-F SEPARATOR] [-s] [-q] expression1 [expression2]
positional arguments:
expression python expression
optional arguments:
-h, --help show this help message and exit
-F SEPARATOR, --separator SEPARATOR
split each line by SEPARATOR
-s, --split split each line
-q, --quiet don't implicitly print results
poline stores stdin in the variable _ (underscore) in the form of a generator of strings.
You can see what's inside _ with:
ls -la | pol "x for x in _"
'total 20'
'drwxr-xr-x 2 default root 104 Apr 2 23:34 .'
'drwxr-xr-x 10 default root 230 Apr 2 23:34 ..'
'-rw-r--r-- 1 default root 0 Mar 31 14:33 __init__.py'
'-rw-r--r-- 1 default root 1740 Apr 2 23:31 _com_collections.py'
'-rw-r--r-- 1 default root 4239 Apr 2 23:33 core.py'
'-rw-r--r-- 1 default root 750 Apr 2 23:34 fields.py'
'-rw-r--r-- 1 default root 3387 Apr 2 23:31 utilfuncs.py'If you call pol with -s option, you the _ is returned in the form of a generator of lists.
ls -la | pol -s "pformat(x,width=100) for x in _"
['total', '20']
['drwxr-xr-x', '2', 'default', 'root', '104', 'Apr', '2', '23:34', '.']
['drwxr-xr-x', '10', 'default', 'root', '230', 'Apr', '2', '23:34', '..']
['-rw-r--r--', '1', 'default', 'root', '0', 'Mar', '31', '14:33', '__init__.py']
['-rw-r--r--', '1', 'default', 'root', '1740', 'Apr', '2', '23:31', '_com_collections.py']
['-rw-r--r--', '1', 'default', 'root', '4239', 'Apr', '2', '23:33', 'core.py']
['-rw-r--r--', '1', 'default', 'root', '750', 'Apr', '2', '23:34', 'fields.py']
['-rw-r--r--', '1', 'default', 'root', '3387', 'Apr', '2', '23:31', 'utilfuncs.py']Field class
Each list item is of the class Field, which is derived from String. Field has a number of useful addition methods:
| Method | Explanation |
|---|---|
| i() | returns int value of the field |
| f() | returns int value of the field |
| h() | returns field as number of bytes in a human readable format. See bytesize() |
Chaining expressions
You can chain expressions as if you were chaining commands on bash using pipes.
netstat -an | grep ESTABLISHED | pol "|url(_4).hostname" "counter(_)" ":x, c: Cols(17,40,None).f(x, get([' '.join(l[1:]) for l in sh(['whois', x],s=T) if 'OrgName' in l[0]], 0), '*' * c)"
Expression types
Awk-like expressions
|epxression
%FS%expression
Works on the last result item by item (or line by line if the last result was stdin)
The split() is called |epxression and split(FS) is called with %FS%expression.
The list generated by the split is available in __.
Individual fields are available as _0, _1, _2 etc.
ps aux | pol "|Cols(10,None).f(_0,_10)"
USER COMMAND
default bash
default ps
default /tmp/poline/poline_venv/bin/pythonlambda-like expressions
:*args:expression
Works on the last result item by item (or line by line if the last result was stdin)
Receives the n-tuple *args.
Example
ps aux | pol "skip(_)" "|(_0,_10)" ":user,cmd:Cols(50,None).f(next(sh('id',user)),cmd)"
uid=1001(default) gid=0(root) groups=0(root) bash
uid=1001(default) gid=0(root) groups=0(root) ps
uid=1001(default) gid=0(root) groups=0(root) /tmp/poline/poline_venv/bin/python
Utility Functions
We are in the process of adding utility functions to poline. Contributions are most welcome.
barchart(x, p = False, w = 10)
Returns the value x as a barchart. If p is true, x is intepreted as a percentage, with 100% being w wide.
df -B1 | pol "|Cols(20,10,10,10,5,None,10).f(_0,_1.h(),_2.h(),_3.h(),_4, barchart(_2.i()/_1.f(),p=True) if _1.isdigit() else ' '*10,_5)"
Filesystem 1B-blocks Used Available Use% Mounted
/dev/mapper/docker-8 9.99 G 8.06 G 1.93 G 81% â–“â–“â–“â–“â–“â–“â–“â–“â–‘â–‘ /
tmpfs 31.37 G 0.00 B 31.37 G 0% â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘ /dev
tmpfs 31.37 G 0.00 B 31.37 G 0% â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘ /sys/fs/cg
/dev/sda3 211.08 G 26.49 G 173.84 G 14% â–“â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘ /etc/hosts
shm 64.00 M 0.00 B 64.00 M 0% â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘â–‘ /dev/shm
bytesize(x, u = None, s = False)
Returns the number of bytes x in a human readable string with a 'B', 'K', 'M', 'G', 'T', 'P' prefix.
If x is a string, bytesize tries to convert it to a float, or returns the string as is.
u specifies that bytes are in a different unit, for example u='K' if the size is in 1-K blocks.
s specifies that byte size should be strict.
Example:
$ pol "bytesize(972693249)"
927.63 M
counter(l, n=10)
Sorts a list by descending order, and returns a 2-tuple with element and number of appearances in l:
[(<element>, <count>), ...]
Example:
$ pol -q "pprint(counter('information'), width=20)"
[('i', 2),
('n', 2),
('o', 2),
('f', 1),
('r', 1),
('m', 1),
('a', 1),
('t', 1)]
get (l, i, d=None)
get ith element from list l if the ith element exists, or return value d
>>> get([1, 2, 3, 4],1)
2
>>> get([1, 2, 3, 4],4,0)
0sh (*args, **kwargs)
Executes shell command and arguments specified in args, returns stdout in the form of a list of lists.
Recognized kwargs: F SEPARATOR split each line by SEPARATOR s True|False split each line
Example:
The following displays the inode of each file using stat
Python>=3.6
$ ls | pol "f'{l:20.20}\t%s' % [i[3] for i in sh(['stat',l]) if 'Inode:' in i[2]][0] for l in _"
Python>=2.7
$ ls | pol "[columns(20,None).format(l,i[3]) for i in sh('stat',l,s=True) if 'Inode:' in i[2]][0] for l in _"
LICENSE 360621
Makefile 360653
pol 360606
pol.c 360637
pol.o 360599
README.md 360623
N.B. The syntax of sh has changed, but old syntax is still supported for the sake of backward compatibility.
As well, the popular shell commands cp, df, docker, du, find, git, history, ln, ls, lsof, mv, netstat, nmcli, ps, rm are all functions. For example
They behave as if the command is being passed to sh above.
pol "ls('-lah')"
total 20K
drwxr-xr-x 2 default root 104 Apr 2 23:34 .
drwxr-xr-x 10 default root 230 Apr 2 23:34 ..
-rw-r--r-- 1 default root 0 Mar 31 14:33 __init__.py
-rw-r--r-- 1 default root 1.7K Apr 2 23:31 _com_collections.py
-rw-r--r-- 1 default root 4.2K Apr 2 23:33 core.py
-rw-r--r-- 1 default root 750 Apr 2 23:34 fields.py
-rw-r--r-- 1 default root 3.4K Apr 2 23:31 utilfuncs.py
skip(iter, n=1)
Skip the first n items from the Iterator iter.
pol "ls('-lah')" "skip(_,n=3)"
-rw-r--r-- 1 default root 0 Mar 31 14:33 __init__.py
-rw-r--r-- 1 default root 1.7K Apr 2 23:31 _com_collections.py
-rw-r--r-- 1 default root 4.2K Apr 2 23:33 core.py
-rw-r--r-- 1 default root 750 Apr 2 23:34 fields.py
-rw-r--r-- 1 default root 3.4K Apr 2 23:31 utilfuncs.py
Utility classes
Cols
Cols is used to manage formatted output in the form of columns
__init__(*args, **kwargs)
The constructor generators columns by unpacking *args. For example:
Cols(10,10,20,None) generates a format string as '{10.10}\t{10.10}\t{20.20}\t{}'
f(*args, **kwargs)
Unpacks *args to pass to String.format(). If format string has not been set, sets the format string as {}\t{}
Examples
The top ten commands you use most often
history | pol "|_1" "counter(_)" ":x,c:Cols(5,None).f(x,c)"