Getting Started with sort and uniq: Sorting Data and Removing Duplicates

Getting Started with sort and uniq: Sorting Data and Removing Duplicates
Reading time: About 10 min Difficulty: Beginner

What you'll be able to do

  • Sort lines alphabetically, numerically, and in reverse with `sort`
  • Explain why `uniq` needs `sort` in front of it
  • Build a frequency ranking with `sort | uniq -c | sort -rn`

Prerequisites (read these first)

What You'll Learn

  • How to sort lines with sort (alphabetical, numeric, reverse)
  • How to remove duplicate lines with uniq — and why it implicitly requires sort
  • How to write the classic frequency ranking pipeline sort | uniq -c | sort -rn
  • Why beginners get stuck on "uniq doesn't remove duplicates" and "numbers come out in a weird order"

Words used here (sorted out first)

  • Line: one line inside a file. Everything between one line break and the next.
  • Field: one part of a line after you split it on spaces. "Column" means the same thing. This article says "column".
  • Alphabetical order: comparing text one character at a time, from the left. "Dictionary order" and "string order" mean the same thing.
  • Numeric order: comparing the lines as numbers. The result differs from alphabetical order. That difference is the first trap.
  • Duplicate: two or more lines whose contents are exactly the same.
  • Pipe: the | symbol. It hands the output of one command straight to the next one.
  • Redirection: the > symbol. It writes output to a file instead of the screen. It does not behave like "save over the file" in an editor — section 7-3 explains why.

Quick Summary

  • Want to sort? → sort
  • Want to sort and dedupe? → sort -u
  • Want to count occurrences? → sort | uniq -c | sort -rn

Environment

  • OS: Ubuntu / typical Linux
  • GNU coreutils sort / uniq (BSD versions on macOS differ in some option details)
  • Sort order depends on your locale, the language and region setting. The output here is what you get under LC_ALL=C, which compares characters strictly by their character codes

1. What Does "Sorting Lines" Mean?

Conclusion: sort prints sorted output without touching the file.

Lina: Senpai, I often want to put logs or lists in alphabetical order. How do I do that?
Linny-senpai: That's exactly what sort is for. sort filename reads the file line by line and prints the sorted result.
Linny-senpai: The important part is that it does not rewrite anything. It only prints to the screen. So a typo cannot damage the original file.
Lina: So the original file stays untouched. That's reassuring.
Linny-senpai: Right. There are three sort orders to remember: alphabetical, numeric, and reversed. Knowing those three handles 80% of real-world cases.

Let's prepare a sample file:

$ cat fruits.txt
banana
apple
cherry
apple
banana
date

1-1. Basic: Alphabetical Order

$ sort fruits.txt
apple
apple
banana
banana
cherry
date

Key points

  • sort defaults to alphabetical (dictionary) order
  • Uppercase and lowercase are treated as different characters. Which one comes first depends on your locale
  • The original file is not modified. sort only prints the sorted result

1-2. Reverse (Descending) Order: -r

$ sort -r fruits.txt
date
cherry
banana
banana
apple
apple

-r stands for reverse.

2. The Numeric Sort Trap

Conclusion: sort compares as strings by default; add -n for numbers.

Lina: I sorted some numbers, but the order looks wrong...
Linny-senpai: Perfect example. Let's see what happens.
$ cat scores.txt
100
3
25
9
1000
$ sort scores.txt
100
1000
25
3
9
Lina: Wait — 100 comes before 25, and 3 and 9 are at the end. Is this a bug?
Linny-senpai: Not a bug. By default, sort does string comparison, one character at a time, from the left.
Linny-senpai: Looking only at the first character, 1 is smaller than 2 or 3. That is why 100 and 1000 move to the front.
Lina: Oh, so it never treated them as numbers at all. That surprised me.
Linny-senpai: To sort them as numbers, add -n. That one option fixes it.

2-1. Numeric Sort: -n

$ sort -n scores.txt
3
9
25
100
1000

-n stands for numeric.

Beginner pitfall

  • Forgetting -n when sorting sizes or counts produces the wrong order
  • Rule of thumb: if the column holds numbers, add -n

2-2. Numbers in Descending Order

$ sort -nr scores.txt
1000
100
25
9
3

-n and -r combine freely. This combination appears in nearly every ranking task.

3. Sort + Deduplicate in One Shot: sort -u

Conclusion: sort -u sorts and removes duplicates in one command.

$ sort -u fruits.txt
apple
banana
cherry
date
Lina: Oh, now apple and banana appear only once each.
Linny-senpai: Yes. -u stands for unique. It removes duplicate lines from the sorted result.
Linny-senpai: When you just want the unique values in order, this one option does it all.

In real work, "give me the unique values" is one of the most common requests. sort -u is the shortcut.

4. uniq: The Deduplication Specialist

Conclusion: uniq only drops adjacent duplicates, so run sort first.

4-1. Lina Gets Stuck: uniq Alone Removes Nothing

$ uniq fruits.txt
banana
apple
cherry
apple
banana
date
Lina: Huh, apple and banana are still duplicated. Did the command do nothing?
Linny-senpai: It did run. But uniq only removes duplicates that sit next to each other.
Lina: So the result changes depending on where the lines are?
Linny-senpai: Exactly. Here apple is on line 2 and line 4. cherry sits between them, so uniq sees two separate groups.
Lina: I see, duplicates that are far apart never get noticed. That makes sense now. So what do I do?
Linny-senpai: Run sort first. Once sort puts identical lines next to each other, uniq can collapse them properly.

Why sort has to come first

uniq reads one line at a time and compares it only with the line right before it. It never remembers the whole file.

So identical lines that sit far apart both survive. sort gathers identical lines together. That is what lets uniq do its job.

4-2. The sort | uniq Pattern

$ sort fruits.txt | uniq
apple
banana
cherry
date

Rule of thumb

  • uniq always goes after sort
  • Use uniq alone only when you already know the input is sorted
  • If "sort and dedupe" is all you want, sort -u is shorter

4-3. Counting Occurrences: uniq -c

$ sort fruits.txt | uniq -c
      2 apple
      2 banana
      1 cherry
      1 date

-c stands for count. Each line gets its occurrence count in front. This is extremely useful for aggregation.

4-4. Duplicates Only / Singletons Only

# Show only lines that appear more than once
$ sort fruits.txt | uniq -d
apple
banana
# Show only lines that appear exactly once
$ sort fruits.txt | uniq -u
cherry
date
Option Meaning Use case
-c Prepend count Aggregation
-d Duplicates only Find duplicated items
-u Singletons only Extract values seen exactly 1x
-i Case-insensitive compare Merge case variants

Both sort and uniq have a -u option, and they mean different things. sort -u collapses duplicates into one line. uniq -u keeps only the lines that appear exactly once. Do not mix them up.

5. The Real-World Workhorse: Frequency Ranking

Conclusion: sort | uniq -c | sort -rn is the frequency ranking idiom.

Lina: For access logs, I want to know which IP hits the server the most. How do I do that?
Linny-senpai: This is today's climax. The three-stage pipeline sort | uniq -c | sort -rn is the standard idiom.
Linny-senpai: It is fine to memorize this shape. We will take it apart stage by stage right after.

Sample log:

$ cat access.log
192.168.1.10
192.168.1.20
192.168.1.10
192.168.1.30
192.168.1.10
192.168.1.20

Frequency ranking:

$ sort access.log | uniq -c | sort -rn
      3 192.168.1.10
      2 192.168.1.20
      1 192.168.1.30

Pipeline breakdown

Stage Command What it does
1 sort Brings identical lines next to each other
2 uniq -c Collapses adjacent duplicates with a count
3 sort -rn Sorts by count (numeric) in descending order

Drop the first sort and stage 2 miscounts. It would treat identical lines that sit far apart as separate groups.

Stage 3 sorts again because the thing being sorted has changed. Up to stage 2 the lines were ordered by their text. Stage 3 reorders them by the count that stage 2 just added.

5-1. Top N Only

$ sort access.log | uniq -c | sort -rn | head -n 3

head -n 3 keeps the top 3 entries. Combining with head is the everyday pattern.

6. Advanced: Sort by a Specific Column with -k

Conclusion: -k2 picks the sort column; add -n for numeric fields.

For CSV or whitespace-separated data, use -k. It chooses which column to sort by.

$ cat sales.txt
apple 120
banana 80
cherry 200
date 50
# Sort by the 2nd column (numeric) in descending order
$ sort -k2 -nr sales.txt
cherry 200
apple 120
banana 80
date 50
  • -k2 selects the second column as the sort key
  • Use -n whenever that column holds numbers
  • To change the delimiter, use -t, (comma-separated) or -t:

7. Common Beginner Pitfalls

Conclusion: Duplicates linger without sort; numbers need -n.

7-1. uniq Didn't Remove the Duplicates

Cause: forgot to sort first.

# BAD: non-adjacent duplicates are not removed
$ uniq fruits.txt

# GOOD
$ sort fruits.txt | uniq
$ sort -u fruits.txt

7-2. Numbers Came Out in a Weird Order

Cause: forgot -n. sort is doing string comparison.

$ sort -n scores.txt   # Sort as numbers

7-3. The Original File Wasn't Modified

sort only prints to the screen. It never modifies the input file. To save the sorted result, redirect it yourself.

$ sort fruits.txt > fruits-sorted.txt

Never do this

# BAD: this empties the file
$ sort fruits.txt > fruits.txt

> empties the destination file before the command runs. So sort reads an empty file. This is not how "save over the file" works in a GUI editor.

To write back to the same file safely, sort into a second file and swap it in. Or use sort -o.

# GOOD: -o writes only after reading is finished
$ sort -o fruits.txt fruits.txt

The virtual terminalAn interactive program that reads the commands you type and runs them. on this site is for learning. Nothing here can damage the files on your own computer, so try it freely.

7-4. Upper/Lowercase Are Treated as Different

$ cat names.txt
Alice
bob
Alice
BOB
$ LC_ALL=C sort -u names.txt
Alice
BOB
bob

LC_ALL=C is there to make your order match this article. Without it, some locales put bob before BOB.

To ignore case, add -f (fold case).

$ LC_ALL=C sort -uf names.txt
Alice
bob

8. Mini Exercises

Conclusion: Three drills: dedupe, count, and rank in descending order.

Lina: I get the theory! I want to try it for real.
Linny-senpai: Here are three exercises. Run them in your terminal.

Exercise 1: Print the unique words from this file.

$ cat << 'EOF' > words.txt
apple
banana
apple
cherry
banana
EOF
Show hint 1 (direction)

One option does "sort" and "remove duplicates" in a single step. Look back at section 3.

Show hint 2 (command name)

The command is sort. One of its options does the sorting and the deduplication in a single step — look for it with sort --help.

Show answer 
$ sort -u words.txt
apple
banana
cherry

Exercise 2: Count how many times each word appears.

Show hint 1 (direction)

Before you can count, identical words have to sit next to each other. So this needs a two-stage pipe.

Show hint 2 (command name)

The commands are sort and uniq. uniq has an option that counts occurrences.

Show answer 
$ sort words.txt | uniq -c
      2 apple
      2 banana
      1 cherry

Exercise 3: Sort the counts in descending order and show only the top 2.

Show hint 1 (direction)

The count column holds numbers. Reorder by that number, largest first. Then take only the first two lines.

Show hint 2 (command name)

The commands are sort and head. sort has one option for treating values as numbers and another for reversing the order.

Show answer 
$ sort words.txt | uniq -c | sort -rn | head -n 2
      2 banana
      2 apple

apple and banana both appear twice. When the counts tie, the whole line decides the order. -r reverses that comparison too, so banana lands first.

9. Copy-Paste Templates

Conclusion: Keep the sort, dedupe, and ranking patterns close at hand.

Patterns to keep handy

# Sort alphabetically
sort file.txt

# Sort and deduplicate
sort -u file.txt

# Sort numerically (ascending / descending)
sort -n file.txt
sort -nr file.txt

# Count occurrences per line
sort file.txt | uniq -c

# Frequency ranking (most frequent first)
sort file.txt | uniq -c | sort -rn

# Top 10 frequency ranking
sort file.txt | uniq -c | sort -rn | head -n 10

# Sort by 2nd column, descending numeric
sort -k2 -nr file.txt

# Case-insensitive unique values
sort -uf file.txt

# Sort in place safely (avoids the > self-truncation bug)
sort -o file.txt file.txt

10. Review

Lina: Let me sum up. uniq only looks at neighbouring lines, so sort has to gather them first.
Linny-senpai: Exactly. That is one order worth remembering together with the reason.
Lina: And numbers need -n. Without it they get compared as text. I have that too.
Linny-senpai: Perfect. One last thing: never run sort file > file. It wipes the contents.

Today's 3-Line Summary

  1. uniq only drops duplicates that sit next to each other, so run sort first to gather them
  2. Add -n when sorting numbers. Without it they are compared as text
  3. The frequency ranking is the three-stage pipeline sort | uniq -c | sort -rn

Summary: What to Read Next

Share this article