• Stars
    star
    134
  • Rank 270,967 (Top 6 %)
  • Language
    Shell
  • License
    GNU General Publi...
  • Created over 8 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Things you can do with sed and awk

sed-awk

TLDR

$ tldr sed
  sed
  Run replacements based on regular expressions.

  - Replace the first occurrence of a string in a file, and print the result:
    sed 's/find/replace/' filename

  - Replace only on lines matching the line pattern:
    sed '/line_pattern/s/find/replace/'

  - Replace all occurrences of a string in a file, overwriting the file (i.e. in-place):
    sed -i 's/find/replace/g' filename

  - Replace all occurrences of an extended regular expression in a file:
    sed -r 's/regex/replace/g' filename

  - Apply multiple find-replace expressions to a file:
    sed -e 's/find/replace/' -e 's/find/replace/' filename
$ tldr awk

  awk
  A versatile programming language for working on files.

  - Print the fifth column in a space separated file:
    awk '{print $5}' filename

  - Print the second column of the lines containing "something" in a space separated file:
    awk '/something/ {print $2}' filename

  - Print the third column in a comma separated file:
    awk -F ',' '{print $3}' filename

  - Sum the values in the first column and print the total:
    awk '{s+=$1} END {print s}' filename

  - Sum the values in the first column and pretty-print the values and then the total:
    awk '{s+=$1; print $1} END {print "--------"; print s}' filename

Useful Commands

# List out the second column in the table.
cat text/table.txt | sed 1d | awk '{ print $2 }'

# Sum the columns in the table.
cat text/table.txt | sed 1d | awk '{ sum += $2 } END { print sum }'

# Kills all processes by name.
ps aux | grep chrome | awk '{ print $2 }' | kill
pkill chrome

# Deletes trailing whitespace.
sed 's/\s\+$//g' filename

# Deletes all blank lines from file.
sed '/^$/d' filename

# Insert 'use strict' to the top of every js file.
sed "1i 'use strict';" *.js

# Append a new line at the end of every file.
sed '1a \n' *

# Generate random numbers and then sort.
for i in {1..20}; do echo $(($RANDOM * 777 * $i)); done | sort -n

# Commatize numbers.
sed -r ':loop; s/(.*[0-9])([0-9]{3})/\1,\2/; t loop' text/numbers.txt

Tutorial

Follow the tutorials here:

# Unzip data.
unzip data.zip

# Zip data.
zip -r data.zip data/

# Preview the files.
head data/names.csv && tail data/names.csv

# Preview csv columns.
sed -n 1p data/colleges.csv | tr ',' '\n'

# Count the number of lines.
wc -l data/*

Sed Print

# Print contents of a file.
sed -n '/fox/p' text/*
sed -n '/Sysadmin/p' text/geek.txt

# Print lines starting with `3` and skipping by `2`.
sed -n '3~2p' text/geek.txt

# Print the last line.
sed -n '$p' text/geek.txt

# Prints the lines matching the between the two patterns.
sed -n '/Hardware/,/Website/p' text/geek.txt

Sed Print Line Number

# Prints the line number for all lines in the file.
sed -n '=' filename

# Prints the line number that matches the pattern.
sed -n '/Linux/=' filename

# Prints the line number in range of two patterns (inclusive).
sed -n '/Linux/,/Hardware/=' filename

# Prints the total number of lines.
sed -n '$=' filename

Sed Delete

The d command performs a deletion.

# Deletes the 3rd line from beginning of file.
sed '3d' text/geek.txt

# Delete every lines starting from 3 and skipping by 2.
sed '3~2d' text/geek.txt

# Delete lines from 3 to 5.
sed '3,5d' text/geek.txt

# Delete the last line.
sed '$d' text/geek.txt

# Delete lines matching the pattern.
sed '/Sysadmin/d' text/geek.txt

Sed Substitute

The s command performs a substitution.

# Simple substituion for the first result.
sed 's/Linux/Unix/' text/geek.txt

# Simple substituion for global instances.
sed 's/Linux/Unix/g' text/geek.txt

# Replace nth instance.
sed 's/Linux/Unix/2' text/geek.txt

# Write matched lines to output.
sed -n 's/Linux/Unix/gp' text/geek.txt > text/geek-sub.txt

# Use regex group for capturing additional patterns (up to 9).
sed 's/\(Linux\).\+/\1/g' text/geek.txt
sed -r 's/(Linux).+/\1/g' text/geek.txt

# Remove the last word.
sed -r 's/\d$//g' text/geek.txt

# Remove all letters.
sed -r 's/[a-zA-Z]//g' text/geek.txt

# Remove html tags (WIP).
sed -r 's|(</?[a-z]+>)||g' text/html.txt

# Commatize any number.
sed ':a;s/\B[0-9]\{3\}\>/,&/;ta' text/numbers.txt
sed -r ':loop; s/\B[0-9]{3}\>/,&/; t loop' text/numbers.txt

Sed Transform

The y command performs a transformation.

# Converts all lowercase chars to uppercase.
sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' text/geek.txt

# Converts all uppercase chars to lowercase.
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' text/geek.txt

# Perform a two character shift.
sed 'y/abcdefghijklmnopqrstuvwxyz/cdefghijklmnopqrstuvwxyzab/' text/geek.txt

Sed Multiple Commands

The -e flag allows for multiple commands.

sed -r -e 's/etc\.*//g' -e 's/(\s+)(\))/\2/g' text/geek.txt

More Repositories

1

dotfiles

:neckbeard: My knowledge of all things linux, servers, and bash with config files
Shell
19
star
2

github-label

🎁 Node.js CLI for creating GitHub labels
JavaScript
13
star
3

directed-graph

📦 Node.js weighted directed graphs
JavaScript
11
star
4

inference-engine

📦 Node.js Logical reasoning machine (WIP)
JavaScript
10
star
5

intro-to-bash

Intro to bash with real-world examples and command-line tips
8
star
6

shiny-prompt

🦄 Super wonderful bash prompt
Shell
7
star
7

obsidian-city

🔰 Procedural city with three.js
JavaScript
6
star
8

basketball-reference

📦 Node.js scraper for basketball-reference.com
JavaScript
5
star
9

flip.js

📦 Flip cards with minimal effort
JavaScript
5
star
10

forest-ecosystem

🔰 Forest Ecosystem simulation
JavaScript
4
star
11

gsearch

📦 Node.js Google search suggestions
JavaScript
4
star
12

json-gallery

HTML image gallery managed with JSON
4
star
13

threejs-boilerplate

Three.js starter project with plugins
JavaScript
3
star
14

docker-cheatsheet

🐳 Docker cheatsheet and images
Python
3
star
15

tribble

🔰 Collection of three.js demos
JavaScript
3
star
16

language

Learning languages one phrase at a time
JavaScript
2
star
17

awesome-lunch-nyc

😋 Tasty and economical lunch spots in NYC
2
star
18

playground

📦 Three.js core library
JavaScript
2
star
19

mean-phonebook

Phonebook app with angular-fullstack
JavaScript
2
star
20

nba-seasonal-wins

Data visualization of the nba standings
JavaScript
2
star
21

music

❤️ My favorite songs
JavaScript
2
star
22

fissbuzz

🎉 FizzBuzz in CSS
CSS
2
star
23

vagrant-boxes

🎁 Collection of vagrant boxes
Python
2
star
24

elastic-beanstalk-nginx-flask

Good luck! Use OpsWorks instead!
Python
2
star
25

web-journal

:octocat: My web component journal
JavaScript
2
star
26

skully.js

📦 Vanilla JS scroll event handlers
JavaScript
2
star
27

the-meaning-of-life

My journey to understand the meaning of life, consciousness, and free will
2
star
28

snake-game

Snake in ES6 JS and canvas with webpack
JavaScript
2
star
29

cultivate

Learn languages from your terminal
Python
2
star
30

calc-lang

Calculator interpreter language
JavaScript
1
star
31

scaffold-js

🎁 Node.js CLI for creating JS dotfiles
JavaScript
1
star
32

explorer

Rediscover YouTube
JavaScript
1
star
33

calendar

Calendar web app written in raw JS
JavaScript
1
star
34

color-sorting

Exploring different color sorting algorithms
JavaScript
1
star
35

react-cartesian

React Cartesian Product component
JavaScript
1
star
36

react-express-template

Create React App with Express template
JavaScript
1
star
37

pubtest

📦 Browser console js unit tests
JavaScript
1
star
38

weather-flux

Crowdsourcing real-time weather updates
JavaScript
1
star
39

racket-quiz

Ruby Quiz problems with Racket
Racket
1
star
40

stylesheet

📦 Create sylesheets with JavaScript
JavaScript
1
star