sort orders lines and uniq collapses repeated lines. Together they answer questions like “which IPs hit my server most?” and “what are the most common errors?” in one line. They’re two of the most useful tools for log analysis.

sort basics

sort names.txt               # alphabetical
sort -r names.txt            # reverse
sort -n numbers.txt          # numeric (2 before 10)
sort -h sizes.txt            # human-readable sizes (1K, 5M, 2G)
sort -u names.txt            # sort and drop duplicates
sort -f names.txt            # ignore case
sort -o names.txt names.txt  # sort a file in place

Without -n, numbers sort as text: 10 comes before 9. Always use -n or -h for numbers.

Sort by a column

sort -k2 file                 # by the second field (whitespace-separated)
sort -t, -k3 -n data.csv      # CSV: comma separator, third column, numeric
sort -t: -k3 -n /etc/passwd   # users by UID
sort -k2,2 -k1,1 file         # by field 2, then field 1

-k2 means “from field 2 to the end of the line”. -k2,2 means “only field 2”, which is what you usually want when sorting on several keys.

Example: the largest directories:

du -sh /var/* 2>/dev/null | sort -rh | head

See du.

uniq: adjacent duplicates only

uniq only removes duplicates that are next to each other, so you almost always sort first:

sort file | uniq            # unique lines
sort file | uniq -c         # count each line
sort file | uniq -d         # only lines that appear more than once
sort file | uniq -u         # only lines that appear exactly once
uniq -i                     # ignore case

The top-N pattern

sort | uniq -c | sort -rn | head
  1. sort groups identical lines together.
  2. uniq -c counts each group.
  3. sort -rn puts the highest counts first.
  4. head keeps the top 10.

Log analysis examples

Top 10 client IPs in an Nginx or Apache access log:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

Most requested URLs:

awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Status code breakdown:

awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

IPs with failed SSH logins:

grep 'Failed password' /var/log/auth.log | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c | sort -rn | head

Requests per hour:

awk '{print substr($4, 2, 14)}' /var/log/nginx/access.log | sort | uniq -c

See awk and Nginx logs.

Compare lists

Lines present in both files, or only in one (both must be sorted):

comm -12 <(sort a.txt) <(sort b.txt)   # in both
comm -23 <(sort a.txt) <(sort b.txt)   # only in a
comm -13 <(sort a.txt) <(sort b.txt)   # only in b

Big files and speed

sort handles files larger than RAM by using temporary files. Options that help:

sort -S 1G --parallel=4 -T /mnt/bigdisk big.log
LC_ALL=C sort big.log      # byte order: much faster, but no locale-aware ordering

LC_ALL=C also gives predictable results across systems. Locale settings can change sort order, which surprises people who compare outputs.

Other useful flags

sort -V versions.txt       # version sort: 1.2 < 1.10
sort -R file               # random order (shuf is better)
sort -c file               # check if already sorted
sort -s -k1,1 file         # stable sort: keep original order for equal keys

Quick reference

Task Command
Numeric sort sort -n
Human sizes sort -h
Reverse sort -r
By column sort -k2,2
CSV column sort -t, -k3,3n
Unique sort -u
Count occurrences sort | uniq -c
Top 10 sort | uniq -c | sort -rn | head
Duplicates only sort | uniq -d

Related: cut, wc and grep.

  • Linux
  • Command Line
  • Logs