• Stars
    star
    134
  • Rank 265,507 (Top 6 %)
  • Language
    Shell
  • License
    GNU General Publi...
  • Created about 8 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Things you can do with sed and awk

sed-awk

TLDR

$ tldr sed
  sed
  Run replacements based on regular expressions.

  - Replace the first occurrence of a string in a file, and print the result:
    sed 's/find/replace/' filename

  - Replace only on lines matching the line pattern:
    sed '/line_pattern/s/find/replace/'

  - Replace all occurrences of a string in a file, overwriting the file (i.e. in-place):
    sed -i 's/find/replace/g' filename

  - Replace all occurrences of an extended regular expression in a file:
    sed -r 's/regex/replace/g' filename

  - Apply multiple find-replace expressions to a file:
    sed -e 's/find/replace/' -e 's/find/replace/' filename
$ tldr awk

  awk
  A versatile programming language for working on files.

  - Print the fifth column in a space separated file:
    awk '{print $5}' filename

  - Print the second column of the lines containing "something" in a space separated file:
    awk '/something/ {print $2}' filename

  - Print the third column in a comma separated file:
    awk -F ',' '{print $3}' filename

  - Sum the values in the first column and print the total:
    awk '{s+=$1} END {print s}' filename

  - Sum the values in the first column and pretty-print the values and then the total:
    awk '{s+=$1; print $1} END {print "--------"; print s}' filename

Useful Commands

# List out the second column in the table.
cat text/table.txt | sed 1d | awk '{ print $2 }'

# Sum the columns in the table.
cat text/table.txt | sed 1d | awk '{ sum += $2 } END { print sum }'

# Kills all processes by name.
ps aux | grep chrome | awk '{ print $2 }' | kill
pkill chrome

# Deletes trailing whitespace.
sed 's/\s\+$//g' filename

# Deletes all blank lines from file.
sed '/^$/d' filename

# Insert 'use strict' to the top of every js file.
sed "1i 'use strict';" *.js

# Append a new line at the end of every file.
sed '1a \n' *

# Generate random numbers and then sort.
for i in {1..20}; do echo $(($RANDOM * 777 * $i)); done | sort -n

# Commatize numbers.
sed -r ':loop; s/(.*[0-9])([0-9]{3})/\1,\2/; t loop' text/numbers.txt

Tutorial

Follow the tutorials here:

# Unzip data.
unzip data.zip

# Zip data.
zip -r data.zip data/

# Preview the files.
head data/names.csv && tail data/names.csv

# Preview csv columns.
sed -n 1p data/colleges.csv | tr ',' '\n'

# Count the number of lines.
wc -l data/*

Sed Print

# Print contents of a file.
sed -n '/fox/p' text/*
sed -n '/Sysadmin/p' text/geek.txt

# Print lines starting with `3` and skipping by `2`.
sed -n '3~2p' text/geek.txt

# Print the last line.
sed -n '$p' text/geek.txt

# Prints the lines matching the between the two patterns.
sed -n '/Hardware/,/Website/p' text/geek.txt

Sed Print Line Number

# Prints the line number for all lines in the file.
sed -n '=' filename

# Prints the line number that matches the pattern.
sed -n '/Linux/=' filename

# Prints the line number in range of two patterns (inclusive).
sed -n '/Linux/,/Hardware/=' filename

# Prints the total number of lines.
sed -n '$=' filename

Sed Delete

The d command performs a deletion.

# Deletes the 3rd line from beginning of file.
sed '3d' text/geek.txt

# Delete every lines starting from 3 and skipping by 2.
sed '3~2d' text/geek.txt

# Delete lines from 3 to 5.
sed '3,5d' text/geek.txt

# Delete the last line.
sed '$d' text/geek.txt

# Delete lines matching the pattern.
sed '/Sysadmin/d' text/geek.txt

Sed Substitute

The s command performs a substitution.

# Simple substituion for the first result.
sed 's/Linux/Unix/' text/geek.txt

# Simple substituion for global instances.
sed 's/Linux/Unix/g' text/geek.txt

# Replace nth instance.
sed 's/Linux/Unix/2' text/geek.txt

# Write matched lines to output.
sed -n 's/Linux/Unix/gp' text/geek.txt > text/geek-sub.txt

# Use regex group for capturing additional patterns (up to 9).
sed 's/\(Linux\).\+/\1/g' text/geek.txt
sed -r 's/(Linux).+/\1/g' text/geek.txt

# Remove the last word.
sed -r 's/\d$//g' text/geek.txt

# Remove all letters.
sed -r 's/[a-zA-Z]//g' text/geek.txt

# Remove html tags (WIP).
sed -r 's|(</?[a-z]+>)||g' text/html.txt

# Commatize any number.
sed ':a;s/\B[0-9]\{3\}\>/,&/;ta' text/numbers.txt
sed -r ':loop; s/\B[0-9]{3}\>/,&/; t loop' text/numbers.txt

Sed Transform

The y command performs a transformation.

# Converts all lowercase chars to uppercase.
sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' text/geek.txt

# Converts all uppercase chars to lowercase.
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' text/geek.txt

# Perform a two character shift.
sed 'y/abcdefghijklmnopqrstuvwxyz/cdefghijklmnopqrstuvwxyzab/' text/geek.txt

Sed Multiple Commands

The -e flag allows for multiple commands.

sed -r -e 's/etc\.*//g' -e 's/(\s+)(\))/\2/g' text/geek.txt

More Repositories

1

dotfiles

:neckbeard: My knowledge of all things linux, servers, and bash with config files
Shell
19
star
2

github-label

🎁 Node.js CLI for creating GitHub labels
JavaScript
13
star
3

directed-graph

πŸ“¦ Node.js weighted directed graphs
JavaScript
11
star
4

inference-engine

πŸ“¦ Node.js Logical reasoning machine (WIP)
JavaScript
10
star
5

intro-to-bash

Intro to bash with real-world examples and command-line tips
8
star
6

shiny-prompt

πŸ¦„ Super wonderful bash prompt
Shell
7
star
7

obsidian-city

πŸ”° Procedural city with three.js
JavaScript
6
star
8

basketball-reference

πŸ“¦ Node.js scraper for basketball-reference.com
JavaScript
5
star
9

flip.js

πŸ“¦ Flip cards with minimal effort
JavaScript
5
star
10

forest-ecosystem

πŸ”° Forest Ecosystem simulation
JavaScript
4
star
11

gsearch

πŸ“¦ Node.js Google search suggestions
JavaScript
4
star
12

json-gallery

HTML image gallery managed with JSON
4
star
13

threejs-boilerplate

Three.js starter project with plugins
JavaScript
3
star
14

docker-cheatsheet

🐳 Docker cheatsheet and images
Python
3
star
15

tribble

πŸ”° Collection of three.js demos
JavaScript
3
star
16

language

Learning languages one phrase at a time
JavaScript
2
star
17

playground

πŸ“¦ Three.js core library
JavaScript
2
star
18

mean-phonebook

Phonebook app with angular-fullstack
JavaScript
2
star
19

nba-seasonal-wins

Data visualization of the nba standings
JavaScript
2
star
20

music

❀️ My favorite songs
JavaScript
2
star
21

fissbuzz

πŸŽ‰ FizzBuzz in CSS
CSS
2
star
22

vagrant-boxes

🎁 Collection of vagrant boxes
Python
2
star
23

elastic-beanstalk-nginx-flask

Good luck! Use OpsWorks instead!
Python
2
star
24

web-journal

:octocat: My web component journal
JavaScript
2
star
25

skully.js

πŸ“¦ Vanilla JS scroll event handlers
JavaScript
2
star
26

awesome-lunch-nyc

πŸ˜‹ Tasty and economical lunch spots in NYC
2
star
27

the-meaning-of-life

My journey to understand the meaning of life, consciousness, and free will
2
star
28

snake-game

Snake in ES6 JS and canvas with webpack
JavaScript
2
star
29

cultivate

Learn languages from your terminal
Python
2
star
30

explorer

Rediscover YouTube
JavaScript
1
star
31

calc-lang

Calculator interpreter language
JavaScript
1
star
32

scaffold-js

🎁 Node.js CLI for creating JS dotfiles
JavaScript
1
star
33

calendar

Calendar web app written in raw JS
JavaScript
1
star
34

color-sorting

Exploring different color sorting algorithms
JavaScript
1
star
35

react-cartesian

React Cartesian Product component
JavaScript
1
star
36

react-express-template

Create React App with Express template
JavaScript
1
star
37

pubtest

πŸ“¦ Browser console js unit tests
JavaScript
1
star
38

weather-flux

Crowdsourcing real-time weather updates
JavaScript
1
star
39

racket-quiz

Ruby Quiz problems with Racket
Racket
1
star
40

stylesheet

πŸ“¦ Create sylesheets with JavaScript
JavaScript
1
star