Pipes and Redirection Basics: Understanding Data Flow in Linux
What You'll Learn
- The concept of "data flow" in Linux
- The difference between
|(pipe) and>>>(redirection) - How to handle stdout and stderr separately
- How to combine
grep,sort, and other tools by chaining small commands
Quick Summary
- Want to save output to a file →
>>> - Want to pass output to another command →
| - Want to handle errors separately →
2>
1. Where Does Data Come From and Go?
ls, file names appear on the screen. Where exactly are they coming from?The 3 standard streams
| Name | Short | Number | Default destination |
|---|---|---|---|
| Standard input | stdin | 0 |
Keyboard |
| Standard output | stdout | 1 |
Screen |
| Standard error | stderr | 2 |
Screen |
2. Redirection: Save Output to a File
2-1. > — Overwrite
$ ls > files.txt
> redirected the output to a file instead of the screen, so nothing prints. Check the file.$ cat files.txt
Documents Downloads files.txt Pictures
2-2. >> — Append
$ echo "line 1" > log.txt $ echo "line 2" >> log.txt $ cat log.txt
line 1 line 2
> truncates the existing file before writing. Using > on an important file will erase its contents. Always use >> when you want to append.
2-3. < — Read Input from a File (advanced)
$ wc -l < log.txt
2
wc -l log.txt?< is "feed this file in place of the keyboard." For now, mastering > and >> is enough.3. Pipe: Connect Commands
3-1. The Basics
| (vertical bar, "pipe") means "send the left command's output as the right command's input."
$ ls | wc -l
12
ls output is fed to wc -l to count files!3-2. Common Combinations
# Filter file list to only those containing ".txt" $ ls | grep .txt # Show nginx-related processes $ ps aux | grep nginx # Reverse-sort the access log and show the latest 10 lines $ cat access.log | sort -r | head -n 10
Pipes can chain as many commands as you want
$ cmd1 | cmd2 | cmd3 | ...
The trick is to give each stage a clear role: filter, sort, format.
4. Pipe vs Redirection
> and |?> vs |
cmd > file → destination is a [file] cmd | cmd2 → destination is [next command's stdin]
>: when you want to save|: when you want to process further
# Pattern A: save ls output to a file $ ls > list.txt # Pattern B: filter ls output through grep $ ls | grep ".log" # Pattern C: combine — filter, then save $ ls | grep ".log" > log-files.txt
"Chain with |, then save the result with >" is the most common real-world pattern.
5. Handling Standard Error (stderr)
5-1. Errors Come from a Different Channel
$ ls /not-exist > out.txt
ls: cannot access '/not-exist': No such file or directory
out.txt, but the error still shows on screen!> only redirects stdout. To capture errors, use the number 2.5-2. Send Errors to a Separate File
$ ls /not-exist 2> error.log
(nothing on screen)
$ cat error.log
ls: cannot access '/not-exist': No such file or directory
5-3. Merge stdout and stderr Into One File
$ command > all.log 2>&1
What 2>&1 means
"Send stderr (2) to the same place as stdout (1)." This pattern is everywhere in log collection — memorize the form.
Order matters. > all.log 2>&1 is correct; 2>&1 > all.log does not capture stderr in all.log (stderr ends up wherever stdout was pointing before the redirect).
6. Common Beginner Pitfalls
6-1. Using > Wiped My File
$ cat important.txt > important.txt # BAD: file becomes empty
> truncates the destination file before the command runs. So cat reads an already-empty file.
Never redirect a command's output back into the same file it reads.
6-2. Do I Need Spaces Around |?
Either works. Adding spaces is the convention for readability.
$ ls|grep txt # works, but hard to read $ ls | grep txt # recommended
6-3. I Want to See Intermediate Pipe Output
$ ls | tee list.txt | wc -l
tee records what's flowing through into a file while passing it on. Useful for debugging or keeping a snapshot of intermediate results.
7. Mini Exercises
Exercise 1: Save the file list of your home directory into home-files.txt.
Show hint
Use > to redirect output to a file.
Show answer
$ ls ~ > home-files.txt
Exercise 2: Count how many entries are under /etc and print the count to the screen (use a pipe).
Show hint
Pass ls output to wc -l.
Show answer
$ ls /etc | wc -l
Exercise 3: From /etc, extract only files ending in .conf and save them to conf-list.txt.
Show hint
Combine ls → grep → >.
Show answer
$ ls /etc | grep "\.conf$" > conf-list.txt
The $ in grep "\.conf$" is a regex anchor meaning "end of line", so only filenames ending with .conf survive.
8. Copy-Paste Templates
Patterns to keep handy
# Save output to a file (overwrite) command > out.txt # Append output to a file command >> out.txt # Filter output command | grep keyword # Sort and take the first 10 lines command | sort | head -n 10 # Capture both stdout and stderr in one file command > all.log 2>&1 # Capture only errors command 2> error.log # Save intermediate output AND keep the pipe flowing command | tee progress.txt | next-command