awk One-Liners - 30 Practical Patterns for Real-World Use

awk One-Liners - 30 Practical Patterns for Real-World Use
Reading time: About 15 min Difficulty: Intermediate

What You'll Get From This Article

  • 30 copy-paste ready awk one-liners organized by use case
  • Field extraction, filtering, aggregation, and text transformation patterns
  • Real-world patterns for CSV, TSV, log files, and numeric data

Quick Summary: Three Core Uses of awk

  1. Field extraction — awk '{print $N}' to slice columns (more powerful than cut)
  2. Aggregation — Use END blocks to compute sums, averages, and max/min values
  3. Transformation — sub/gsub for regex replacement, printf for formatted output

Prerequisites

  • These examples assume GNU awk (gawk). macOS ships POSIX awk which lacks some features.
  • All patterns work both on files and as part of a pipeline via stdin.

How Does the awk Syntax Work?

The core form is awk 'pattern { action }' file. Omitting the pattern applies the action to every line; omitting the action defaults to {print}.

# Basic forms
awk '{ action }' file
awk 'pattern' file
awk 'pattern { action }' file

Essential built-in variables:

Variable Meaning
$0 Entire line
$1, $2... 1st field, 2nd field...
NF Number of fields (columns) in current line
NR Current line number (across all files)
FNR Line number within current file
FS Input field separator (default: whitespace)
OFS Output field separator (default: space) 
# Set separator with -F
awk -F: '{print $1}' /etc/passwd   # 1st field, colon-delimited
awk -F, '{print $2}' data.csv      # 2nd column from CSV

Field Manipulation — How Do You Extract and Transform Columns?

Extracting, reordering, and reformatting fields is the most common awk use case.

Pattern 1: Print specific fields

awk '{print $2}' file.txt           # 2nd field only
awk '{print $1, $3}' file.txt       # 1st and 3rd field, space-separated
awk '{print $1 "\t" $3}' file.txt   # Tab-separated output

Pattern 2: Reorder fields

# Swap 1st and 2nd fields
awk '{print $2, $1}' file.txt

Pattern 3: Last field and second-to-last

awk '{print $NF}' file.txt          # Last field
awk '{print $(NF-1)}' file.txt      # Second-to-last field

Pattern 4: Convert field separator

# Colon-delimited to comma-delimited
awk -F: '{OFS=","; $1=$1; print}' file.txt

# Space-delimited to tab-delimited
awk '{OFS="\t"; $1=$1; print}' file.txt

$1=$1 is an awk idiom that forces awk to rebuild $0 using OFS. The value of $1 does not change — the assignment just triggers reconstruction of the full line.

Pattern 5: Filter by field count

awk 'NF==5' file.txt        # Lines with exactly 5 fields
awk 'NF>=3' file.txt        # Lines with 3 or more fields

Pattern 6: Normalize inconsistent whitespace

# Collapse multiple spaces/tabs into a single OFS
awk '{$1=$1; print}' file.txt

Pattern Matching and Filtering — How Do You Extract Matching Lines?

awk lets you filter by both content and column value, which grep alone cannot do.

Pattern 7: Match lines containing a string

awk '/ERROR/' logfile            # Lines containing ERROR (same as grep)
awk '/^2026/' access.log         # Lines starting with 2026
awk '/ERROR|WARN/' logfile       # Lines containing ERROR or WARN

Pattern 8: Filter by a specific field

awk '$3 == "200"' access.log          # Lines where 3rd field is 200
awk '$2 > 100' data.txt               # Lines where 2nd field exceeds 100
awk '$1 ~ /^user/' data.txt           # 1st field starts with "user" (regex)
awk '$1 !~ /^#/' config.txt           # Skip lines where 1st field starts with #

Pattern 9: AND / OR conditions

awk '$1 == "GET" && $9 >= 500' access.log    # AND
awk '$3 == "404" || $3 == "500"' access.log  # OR

Pattern 10: Skip empty lines

awk 'NF > 0' file.txt        # Lines with at least one field (no blank lines)
awk '!/^$/' file.txt         # Same via regex

Line Numbers and Ranges — How Do You Select Specific Lines?

Pattern 11: Print a specific line or range

awk 'NR==3' file.txt             # Line 3 only
awk 'NR>=5 && NR<=10' file.txt   # Lines 5 through 10

Pattern 12: Skip the header row

awk 'NR>1' data.csv        # Skip line 1 (header)

Pattern 13: Odd and even lines

awk 'NR%2==1' file.txt     # Odd-numbered lines
awk 'NR%2==0' file.txt     # Even-numbered lines

Pattern 14: Add line numbers

awk '{print NR ": " $0}' file.txt             # Line number prefix
awk '{printf "%04d %s\n", NR, $0}' file.txt   # Zero-padded numbering

Pattern 15: Print lines between two patterns

# Print from START to END (inclusive)
awk '/START/,/END/' file.txt

# From the line after a header marker to the first blank line
awk '/^---$/,/^$/' file.txt

Aggregation and Calculation — How Do You Summarize Numeric Data?

The BEGIN and END blocks run before and after processing all lines — ideal for accumulating totals.

Pattern 16: Sum a column

awk '{sum += $1} END {print sum}' data.txt
awk -F, '{sum += $3} END {print sum}' data.csv    # 3rd column of a CSV

Pattern 17: Average

awk '{sum += $1; count++} END {print sum/count}' data.txt

Pattern 18: Max and min

awk 'NR==1 {max=$1; min=$1} {if($1>max) max=$1; if($1<min) min=$1} END {print "max:"max, "min:"min}' data.txt

Pattern 19: Count lines (conditional)

awk 'END {print NR}' file.txt                       # Same as wc -l
awk '/ERROR/ {count++} END {print count}' logfile   # Count matching lines

Pattern 20: Count occurrences per value (histogram)

# Tally by 3rd field value
awk '{count[$3]++} END {for(k in count) print k, count[k]}' access.log

# HTTP status code histogram, sorted by frequency descending
awk '{count[$3]++} END {for(k in count) print count[k], k}' access.log | sort -rn

Pattern 21: Unique values (deduplicate)

# Unique 1st field values, preserving order
awk '!seen[$1]++' file.txt

# Deduplicate entire lines, preserving order
awk '!seen[$0]++' file.txt

!seen[$1]++ works like this: if seen[$1] is 0 (first occurrence), the condition is truthy so awk prints the line; the post-increment then sets it to 1 so subsequent duplicates are suppressed. Order is preserved — unlike sort -u.

String Manipulation — How Do You Transform Text?

Pattern 22: Replace strings (sub / gsub)

# Replace first match (sub)
awk '{sub(/foo/, "bar"); print}' file.txt

# Replace all matches (gsub)
awk '{gsub(/foo/, "bar"); print}' file.txt

# Replace only in a specific field
awk '{gsub(/http:/, "https:", $1); print}' file.txt

Pattern 23: Change case

awk '{print toupper($0)}' file.txt    # Uppercase entire line
awk '{print tolower($1)}' file.txt    # Lowercase 1st field only

Pattern 24: Extract a substring

# substr(string, start, length) — 1-indexed
awk '{print substr($1, 1, 8)}' file.txt      # First 8 characters of field 1
awk '{print substr($1, 5)}' file.txt         # Field 1 from position 5 onward

Pattern 25: Filter by string length

awk 'length($1) > 10' file.txt      # 1st field longer than 10 characters
awk 'length > 80' file.txt          # Lines longer than 80 characters

Formatted Output With printf — How Do You Align Columns?

Use printf when print produces misaligned output.

Pattern 26: Align columns

# Left-aligned 20-char column, right-aligned 8-char column
awk '{printf "%-20s %8s\n", $1, $2}' file.txt

# Numeric output with 2 decimal places
awk '{printf "%.2f\n", $1}' data.txt

Pattern 27: Convert CSV to aligned table

awk -F, '{printf "%-15s %-10s %-8s\n", $1, $2, $3}' data.csv

Multiple Files and File Comparison

Pattern 28: Process multiple files, skip each header

# FNR resets per file; NR keeps counting across files
awk 'FNR>1 {print FILENAME, $0}' *.csv

Pattern 29: Lines in file1 but not in file2

# Simple diff: print lines from file2 missing in file1
awk 'NR==FNR {a[$0]=1; next} !a[$0]' file1.txt file2.txt

NR==FNR is true only while the first file is being processed. next skips the rest of the rule, so only the first file populates array a. For every line in the second file, !a[$0] filters to lines not seen in the first.

Pattern 30: Initialize variables in BEGIN, summarize in END

# Set FS/OFS before processing starts
awk 'BEGIN {FS=","; OFS="\t"} {print $1, $2, $3}' data.csv

# Print header and footer around the data
awk 'BEGIN {print "name\tscore"} {print $1, $2} END {print "---"}' data.txt

Cheatsheet — Quick-Reference Patterns

Field operations

awk '{print $1}'                      # 1st field
awk '{print $NF}'                     # Last field
awk '{print $(NF-1)}'                 # Second-to-last field
awk -F, '{print $2}'                  # 2nd column from CSV
awk -F: '{OFS=","; $1=$1; print}'     # Convert colon to comma separator

Filtering

awk 'NR>1'                            # Skip header row
awk '/ERROR/'                         # Lines containing ERROR
awk '$3 >= 500'                       # 3rd field >= 500
awk '!seen[$0]++'                     # Remove duplicate lines
awk 'NF>0'                            # Remove blank lines

Aggregation

awk '{sum+=$1} END{print sum}'        # Column sum
awk 'END{print NR}'                   # Line count (like wc -l)
awk '{c[$1]++} END{for(k in c) print k, c[k]}'   # Count per value

Next Reading