• Stars
    star
    221
  • Rank 179,742 (Top 4 %)
  • Language
    C
  • License
    BSD 2-Clause "Sim...
  • Created over 6 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Map lines from stdin to commands

map

Map lines from stdin to commands.

Description

Map lets you process each line from stdin with a command of your choice. For example:

$ ls
LICENSE   README.md makefile  map.1   map.c
$ ls | map f 'echo $f $f'
LICENSE LICENSE
README.md README.md
makefile makefile
map.1 map.1
map.c map.c

Note that the command must be wrapped in single quotes to prevent the variable from being expanded by the shell.

Installation

Install map into /usr/local/bin with the following command:

$ make install

You can use make PREFIX=/some/other/directory install if you wish to use a different destination. If you want to remove map from your system, use make uninstall.

Motivation

There are many ways to accomplish what you can do with map, including find, xargs, awk, and shell for-loops. The approach taken by map is extremely pragmatic and allows me to express concisely what I want. Given the fact that it's designed as a filter, it can operate on any kind of list, not only lists of files.

The problem that prompted me to think about map was the following: given a list of files, I wanted to execute two commands on each. Here's how you can do it with different tools:

With map:

ls *.c | map f 'foo $f; bar $f'

With xargs:

ls *.c | xargs -I % sh -c 'foo %; bar %;'

With awk:

ls *.c | awk '{ system("foo "$0"; bar "$0) }'

With find:

find . -name \*.c -maxdepth 1 -exec foo {} \; -exec bar {} \;

With a bash for-loop:

for f in $(ls *.c)
do
  foo $f
  bar $f
done

With a csh for-loop:

foreach f (*.c)
  foo $f
  bar $f
end

Map's modest claim is that it improves on the ergonomics of existing tools. It's not only that sometimes it allows you to type less, but also the conceptual model needed to operate it is simpler.

Let's consider these tasks:

  1. Execute a command foo on each C file:
ls *.c | map f 'foo $f'
ls *.c | xargs foo
find . -name *.c -maxdepth 1 -exec foo {} \;
  1. Execute commands foo and bar on each C file:
ls *.c | map f 'foo $f; bar $f'
ls *.c | xargs -I % sh -c 'foo %; bar %;'
find . -name *.c -maxdepth 1 -exec foo {} \; -exec bar {} \;
  1. Download files from a list of URLs in a file:
cat urls | map u 'curl -O $u'
cat urls | xargs -n 1 curl -O
  1. Sleep three times for one second and say "done" after each elapsed second:
printf "1\n1\n1\n" | map t 'sleep $t && say done'
printf "1\n1\n1\n" | xargs -n 1 -I % sh -c 'sleep % && say done'
  1. Same as #4, but run the commands in parallel:
printf "1\n1\n1\n" | map t 'sleep $t && say done &'
printf "1\n1\n1\n" | xargs -n 1 -P 3 -I % sh -c 'sleep % && say done'

The last three examples are not possible with find, because it only operates on file hierarchies.

When using map, the commands don't vary much because the second argument is a template for a well known syntax. On the other hand, there's more variation in the invocations to xargs and find, which means you may need to remember those command line options if you want an idiomatic solution.

As with anything in life, familiarity helps and if you use a tool in a certain way over and over it will seem simple to operate, but we can still analyze the conceptual models and determine how much information is needed in each case. I would say the advantage of map is that it requires less knowledge.

Of course xargs and find are much larger tools, with a bigger feature set. Map has around 20 lines of code. For comparison, here's the source code of GNU xargs. No doubt xargs will offer a lot more features, but so far with map I've completely stopped using xargs and for-loops. Another way to think about map vs xargs: if map had been specified in POSIX and xargs was just released, I'm not sure I would install it unless map proved to be unfit for a given use case.

Something important to keep in mind is that map works on lines, not on files. My reasoning was that in the context of a POSIX environment, a map function can be expected to filter lines from stdin. In that regard, it is very generic because a line can represent anything.

Known issues

In the examples above I frequently use the output of ls. In my projects, filenames don't contain spaces, newlines, or other special characters because I find them inconvenient, even though they are valid. As map works on lines, filenames that contain newlines should be handled separately. For files that contain whitespace, the common solution is to wrap the variable in double quotes, so instead of $f you would use "$f".

Contributing

If you find a bug, please create an issue detailing the ways to reproduce it. If you have a suggestion, create an issue detailing the use case.

More Repositories

1

cuba

Rum based microframework for web development.
Ruby
1,438
star
2

ohm

Object-Hash Mapping for Redis
Ruby
1,390
star
3

micromachine

Minimal Finite State Machine
Ruby
522
star
4

clac

Command-line, stack-based calculator with postfix notation
C
350
star
5

mote

Minimum Operational Template
Ruby
217
star
6

nest

Generate nested namespaced keys for key-value databases.
Ruby
185
star
7

ost

Redis based queues and workers.
Ruby
166
star
8

toro

Tree oriented routing
Crystal
144
star
9

syro

Simple router for web applications
Ruby
135
star
10

cargo

Require libraries without cluttering your namespace.
Ruby
127
star
11

scrivener

Validation frontend for models.
Ruby
124
star
12

clap

Command line argument parsing
Ruby
90
star
13

ohm-crystal

Ohm for Crystal
Crystal
71
star
14

disque-rb

Disque client for Ruby
Ruby
68
star
15

gs

Gemset management
Ruby
67
star
16

totp

Time-based One-Time Passwords
Ruby
44
star
17

mailcat

Fake SMTP server that prints emails to stdout
C
38
star
18

chen

Change directory entries with your text editor
C
37
star
19

spawn

A ridiculously simple fixtures replacement for your web framework of choice.
Ruby
36
star
20

finist

Redis based Finite State Machine
Ruby
36
star
21

redisurl

Connect to Redis using a REDIS_URL and the redigo client.
Go
33
star
22

hart

Hash router
Ruby
24
star
23

stal-ruby

Set algebra solver for Redis
Ruby
22
star
24

redisent

Sentinels aware Redis client.
Ruby
22
star
25

relay

Relay commands over SSH
Ruby
21
star
26

resp

Lightweight RESP client for Lua
Lua
21
star
27

lomo

Apply a lomo filter to your pictures from the command line using ImageMagick.
Ruby
19
star
28

terco

Obstinate DNS
Ruby
16
star
29

override

The as-simple-as-possible-but-not-simpler stubbing library.
Ruby
16
star
30

ox

Skeleton for a Cuba-based JSON API.
Ruby
16
star
31

nido

Structured keys helper
Ruby
14
star
32

syro-demo

Demo application with Syro
Ruby
14
star
33

resp-crystal

Lightweight RESP client
Crystal
14
star
34

mt

Mail tester daemon.
Ruby
13
star
35

stringent

Generate a string with a target entropy
Ruby
12
star
36

rediscan

Scanner for Redis keyspace
Ruby
12
star
37

hypertext

Hypertext authoring with Ruby
Ruby
11
star
38

filmo

A single page presentation tool.
11
star
39

cuba-book

Cuba Book
11
star
40

sc

List of HTTP status codes.
Ruby
11
star
41

finist.lua

Redis based Finite State Machine
Lua
10
star
42

drawer

Ultra slim file-based cache
Ruby
10
star
43

nest-crystal

Object Oriented Keys for Redis
Crystal
9
star
44

basica

Basic authentication library.
Ruby
8
star
45

rino

Remove a file by its inode number
C
7
star
46

seg.rb

Segment matcher for paths
Ruby
7
star
47

seg

Segment matcher for paths
Crystal
7
star
48

trim

Read from stdin and remove a prefix from each line
C
7
star
49

tas

Trees as strings
Ruby
6
star
50

stal

Set algebra solver for Redis
Lua
6
star
51

pac

Package management for Lua libraries.
Shell
6
star
52

prep

Read from stdin and prepend a string to each line while preserving identation.
C
6
star
53

m4s2

Static Site Generator
HTML
5
star
54

contract

Contract helper
Ruby
5
star
55

stal-crystal

Set algebra solver for Redis
Crystal
5
star
56

loco

Lines of code counter
C
4
star
57

ook

Object Oriented Keys for Redis
Ruby
4
star
58

walk

Walk a directory tree and print the name of every regular file
C
4
star
59

rel

Ruby client for the Bandicoot database.
Ruby
4
star
60

textorama

Text slides for the terminal
Shell
4
star
61

homebrew-tools

Formulae for Homebrew
Ruby
4
star
62

rediscan.lua

Scanner for Redis keyspace in Lua
Lua
3
star
63

seg.go

Segment matcher for paths
Go
3
star
64

look

Add a vendor directory to your load path.
Ruby
3
star
65

app

Cuba application template for gn, the file generator.
Ruby
2
star
66

packer

Require different versions of the same Cargo-compatible gem
Ruby
2
star
67

tele.sh

2
star
68

ers

AWK script easy replacements in templates
Awk
2
star
69

soveran.github.io

2
star
70

filter

Workflow template for gn.
Ruby
1
star
71

remoto

Ruby
1
star
72

gem

Gem template for gn, the file generator.
Ruby
1
star
73

ohm-scripts

Lua scripts for Ohm compatible libraries
Lua
1
star
74

ostgo

Ost client.
Go
1
star
75

miga

Breadcrumb name of the working directory
C
1
star