awk processes text as records (lines) made of fields (columns). That makes it perfect for log files, command output and CSV-like data: print certain columns, filter rows by value, add things up, count occurrences. A one-line awk program often replaces a short script.

awk 'pattern { action }' file

For each line, if the pattern matches (or there’s no pattern), awk runs the action.

Fields

awk splits each line on whitespace by default:

  • $1, $2, … — the fields
  • $0 — the whole line
  • NF — number of fields; $NF — the last field
  • NR — the current line number
awk '{ print $1 }' file              # first column
awk '{ print $1, $3 }' file          # columns 1 and 3, space-separated
awk '{ print $NF }' file             # last column
awk '{ print NR ": " $0 }' file      # number each line

With command output:

df -h | awk '{ print $1, $5 }'                     # filesystem and use%
ps aux | awk '{ print $2, $11 }'                   # PID and command
ls -l | awk '{ print $5, $9 }'                     # size and name

Field separator

awk -F: '{ print $1 }' /etc/passwd                 # usernames
awk -F: '$3 >= 1000 { print $1 }' /etc/passwd      # regular (non-system) users
awk -F, '{ print $2 }' data.csv                    # CSV column (simple, no quoted commas)
awk -F'\t' '{ print $1 }' file.tsv

Set the output separator with OFS:

awk -F: 'BEGIN { OFS="," } { print $1, $6 }' /etc/passwd

Filter rows

awk '$3 > 100' file                    # third column greater than 100
awk '$1 == "ERROR"' app.log
awk '/timeout/' app.log                # lines matching a regex (like grep)
awk '!/healthcheck/' access.log        # lines NOT matching
awk 'NR > 1' file.csv                  # skip the header line
awk 'NR==10, NR==20' file              # lines 10 to 20
awk 'length($0) > 200' file            # long lines

Combine conditions with && and ||:

awk '$9 >= 500 && $9 < 600' /var/log/nginx/access.log

Sum, average, count

awk '{ sum += $5 } END { print sum }' file
awk '{ sum += $5; n++ } END { print sum/n }' file
awk 'END { print NR }' file                       # line count

The END block runs once after all lines are read; BEGIN runs before the first.

Total size of files listed by ls -l, in MB:

ls -l | awk 'NR>1 { s += $5 } END { printf "%.1f MB\n", s/1024/1024 }'

Memory used by all PHP-FPM processes:

ps -C php-fpm8.3 -o rss= | awk '{ s += $1 } END { printf "%.0f MB\n", s/1024 }'

Count occurrences with arrays

awk’s associative arrays make counting easy:

awk '{ count[$1]++ } END { for (k in count) print count[k], k }' file | sort -rn | head

Analysing an Nginx/Apache access log

In the common “combined” log format, $1 is the client IP, $7 the path and $9 the status code.

LOG=/var/log/nginx/access.log

# Top 10 client IPs
awk '{ print $1 }' $LOG | sort | uniq -c | sort -rn | head

# Requests per status code
awk '{ print $9 }' $LOG | sort | uniq -c | sort -rn

# Most requested URLs returning 404
awk '$9 == 404 { print $7 }' $LOG | sort | uniq -c | sort -rn | head

# All 5xx errors
awk '$9 ~ /^5/' $LOG | tail

# Bytes sent per IP (field 10)
awk '{ bytes[$1] += $10 } END { for (ip in bytes) printf "%10.1f MB %s\n", bytes[ip]/1048576, ip }' $LOG | sort -rn | head

# Requests per hour
awk '{ split($4, t, ":"); print t[2] }' $LOG | sort | uniq -c

These one-liners quickly reveal a misbehaving bot, a broken link or a traffic spike. If you log $request_time, awk can find slow requests too.

printf for formatted output

awk '{ printf "%-20s %8.2f\n", $1, $2 }' file
df -h | awk 'NR>1 { printf "%-25s %5s used\n", $6, $5 }'

Modify fields

awk '{ $2 = toupper($2); print }' file
awk -F, 'BEGIN { OFS="," } { $3 = $3 * 1.1; print }' prices.csv   # +10% on column 3
awk '{ gsub(/http:/, "https:"); print }' urls.txt                  # replace within the line

Variables from the shell

LIMIT=90
df -h | awk -v limit="$LIMIT" 'NR>1 { gsub("%","",$5); if ($5+0 > limit) print $6 " is " $5 "% full" }'

-v passes shell variables safely — don’t paste them into the awk program string.

awk vs cut vs sed

  • cut -d: -f1 — quicker for simple column extraction with a single-character delimiter.
  • sed — substitutions and line edits — see sed examples.
  • awk — anything involving columns and logic: conditions, sums, counting, formatting.

Quick reference

Task Command
Print column 2 awk '{print $2}'
Custom separator awk -F: '{print $1}'
Last column awk '{print $NF}'
Filter by value awk '$3 > 100'
Sum a column awk '{s+=$1} END {print s}'
Count by key awk '{c[$1]++} END {for(k in c) print c[k],k}'
Skip header awk 'NR>1'
Pass a variable awk -v x="$X" '…'

Combine awk with grep and sort | uniq -c and you can answer most “what’s happening on this server?” questions in seconds.

  • Linux
  • Command Line
  • awk