How to Use head, tail, and Pipes - Linux File Operations Advanced
After mastering basic file operations, let's learn advanced techniques. In this advanced edition, we'll explain practical file analysis and operation techniques using head, tail, file, stat, pipes, and redirection.
head/tail - Log Analysis and File Analysis Techniques
head and tail are the most frequently used file analysis commands in professional work. They're indispensable especially for server log analysis and debugging.
head - Smart Display of File Beginning
Basic Usage
$ head access.log
Displays first 10 lines by default. Perfect for understanding file structure.
Flexible Display with Line Number Specification
$ head -n 5 error.log # First 5 lines $ head -5 error.log # Short form $ head -n 100 config.txt # First 100 lines
Batch Check of Multiple Files
$ head -n 3 *.log
==> access.log <== 192.168.1.10 - - [11/Jan/2025:10:00:01] "GET /" 192.168.1.11 - - [11/Jan/2025:10:00:02] "GET /api" 192.168.1.12 - - [11/Jan/2025:10:00:03] "POST /login" ==> error.log <== [2025-01-11 10:00:01] ERROR: Database connection failed [2025-01-11 10:00:05] WARN: Slow query detected [2025-01-11 10:00:10] ERROR: Authentication failed
Check contents of multiple log files at once.
tail - Latest Information and Real-time Monitoring
Check Latest Errors
$ tail error.log
Shows last 10 lines by default. Quickly identify latest errors and events.
Real-time Log Monitoring (Most Important)
$ tail -f /var/log/app.log
Most frequently used feature in professional work. Monitor logs in real-time, automatically displaying new lines as they're added. Exit method: Ctrl+C
Display from Specific Position
$ tail -n +50 large_file.txt # From line 50 to end $ tail -n 20 access.log # Last 20 lines
Advanced Log Analysis Techniques in Professional Work
Identify and Track Error Occurrence Time
# Check context just before error occurrence $ grep -n "ERROR" app.log | tail -1 # Get line number of latest error 47:ERROR: Connection timeout # Detailed check of surroundings $ head -n 50 app.log | tail -n 10 # Display lines 41-50
Log Rotation Compatible Monitoring
# Continue monitoring even when log file rotates $ tail -F /var/log/app.log # Capital F handles file recreation
Simultaneous Monitoring of Multiple Logs
# Monitor multiple log files simultaneously $ tail -f /var/log/app.log /var/log/error.log # More advanced method: multitail command $ multitail /var/log/app.log /var/log/error.log /var/log/access.log
Professional head/tail Usage Techniques
Performance Issue Tracking
# Check latest trends in access log $ tail -f access.log | grep "slow\|timeout\|error"
Real-time Monitoring During Deployment
# In separate terminal during deployment $ tail -f /var/log/deploy.log | tee deploy_$(date +%Y%m%d).log
Appropriate Selection Based on File Size
# cat for small files, head/tail for large files $ wc -l logfile.txt # Check line count $ [[ $(wc -l < file.txt) -gt 50 ]] && head file.txt || cat file.txt
file/stat - Detailed File Information Investigation
Commands for investigating file identity and detailed information. Very useful for security checks and debugging.
file - File Type Identification
Basic File Diagnosis
$ file mysterious_file mysterious_file: UTF-8 Unicode text, with CRLF line terminators
Identifies file type, encoding, and line endings.
Batch Diagnosis of Multiple Files
$ file * config.txt: ASCII text data.bin: data image.jpg: JPEG image data, JFIF standard script.sh: Bourne-Again shell script, ASCII text executable archive.tar.gz: gzip compressed data
Security Check Utilization
# Detect extension spoofing $ file suspicious_file.txt suspicious_file.txt: PE32 executable (console) Intel 80386, for MS Windows # .txt but actually a Windows executable!
If extension and actual file type differ, it's possibly a malicious file.
stat - Detailed File Information
Complete File Information Display
$ stat important_file.txt File: important_file.txt Size: 1024 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 1234567 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/username) Gid: ( 1000/usergroup) Access: 2025-01-11 10:30:45.123456789 +0900 Modify: 2025-01-11 10:25:30.987654321 +0900 Change: 2025-01-11 10:25:30.987654321 +0900 Birth: 2025-01-11 10:20:15.555666777 +0900
Detailed information on file size, permissions, timestamps.
Display with Custom Format
# Display only size and modification time $ stat --format="%n: %s bytes, modified %y" *.txt config.txt: 2048 bytes, modified 2025-01-11 09:15:22.123456789 +0900 log.txt: 51200 bytes, modified 2025-01-11 10:45:33.987654321 +0900
Practical Work Scenarios
File Change Tracking Investigation
# Security incident response $ stat --format="%n - Last modified: %y, Last accessed: %x" /etc/passwd /etc/passwd - Last modified: 2025-01-11 08:30:15.123456789 +0900, Last accessed: 2025-01-11 10:45:22.987654321 +0900 # Check for unauthorized access
Disk Space Diagnosis
# Detailed info on large files $ stat --format="%n: %s bytes (%S blocks)" large_files/* database.db: 1073741824 bytes (262144 blocks) backup.tar.gz: 536870912 bytes (131072 blocks)
Symbolic Link Diagnosis
$ file suspicious_link suspicious_link: symbolic link to /tmp/malicious_file $ stat suspicious_link File: suspicious_link -> /tmp/malicious_file Size: 18 Blocks: 0 IO Block: 4096 symbolic link
Identify link target and verify safety.
Professional file/stat Command Utilization
File Type Filtering in Batch Processing
# Extract only executable files
for f in *; do
[[ $(file "$f") == *"executable"* ]] && echo "$f"
doneBatch Sorting by File Size
# File list sorted by size stat --format="%s %n" * | sort -n | tail -10
Pipes and Redirection
Combine multiple commands to achieve powerful processing.
Pipe (|)
Pass the output of one command as input to another command.
$ ls -la | grep ".txt"
Display only .txt files.
Redirection
| Symbol | Description | Example |
|---|---|---|
> |
Overwrite output to file | ls > list.txt |
>> |
Append output to file | echo "text" >> file.txt |
< |
Input from file | sort < data.txt |
2> |
Redirect error output | command 2> error.log |
&> |
Both stdout and stderr to same file | command &> all.log |
Practical Combination Techniques
Introduce command combination examples commonly used in actual work.
Example 1: Extract and Count Errors from Log File
$ grep "ERROR" app.log | cut -d' ' -f3 | sort | uniq -c | sort -rn
Display occurrence count by error type in descending order.
Example 2: Find Top 10 Largest Files
$ find . -type f -exec ls -lh {} \; | sort -k5 -rh | head -10Top 10 largest files under current directory.
Example 3: Count Files with Specific Extension
$ find . -name "*.txt" | wc -l
Display total number of .txt files.
Example 4: Top 5 Memory Usage Processes
$ ps aux | sort -k4 -rn | head -5
Display 5 processes with highest memory usage.
Example 5: Create File Backups
$ find . -name "*.conf" -exec cp {} {}.backup \;Create backups of all .conf files.
Practical Exercises: Daily Work Scenarios
Learn practical use of file operation commands through actual work scenarios.
Scenario 1: Server Maintenance Work
Task: Log File Rotation and Archiving
# 1. Check current log file size $ ls -lah /var/log/app.log # 2. Check latest errors $ tail -20 /var/log/app.log | grep "ERROR" # 3. Create safe archive $ cp /var/log/app.log /var/log/archive/app.log.$(date +%Y%m%d) # 4. Clear log (execute while service stopped, requires root) $ sudo sh -c '> /var/log/app.log' # Empty the file
Scenario 2: Application Deployment
Task: Safe Deployment of New Version
# 1. Backup current version $ cp -rp /opt/myapp /opt/myapp.backup.$(date +%Y%m%d_%H%M%S) # 2. Extract and verify new version $ tar -tf new_version.tar.gz | head -10 # Check contents $ tar -xzf new_version.tar.gz -C /tmp/ # Temporary extraction # 3. Merge configuration files $ cp /opt/myapp/config.ini /tmp/myapp/config.ini $ diff /opt/myapp/config.ini /tmp/myapp/config.ini # Check diff # 4. Safe overwrite to production $ mv /opt/myapp /opt/myapp.old $ mv /tmp/myapp /opt/myapp
Scenario 3: Security Incident Response
Task: Investigate Unauthorized Access and Preserve Evidence
# 1. Emergency backup of access logs $ cp /var/log/access.log /home/incident/access.log.$(date +%Y%m%d_%H%M%S) # 2. Identify unauthorized access time $ grep "suspicious_pattern" /var/log/access.log | head -1 $ grep "suspicious_pattern" /var/log/access.log | tail -1 # 3. Extract logs from relevant time period $ awk '/2025-01-11 10:30:/,/2025-01-11 11:00:/' /var/log/access.log > incident_logs.txt # 4. Collect related file information $ stat /var/log/access.log > file_metadata.txt $ file /var/log/access.log >> file_metadata.txt
Scenario 4: Disk Space Cleanup
Task: Safe Deletion of Unnecessary Files
# 1. Check disk usage
$ df -h
$ du -sh /var/log/* | sort -hr | head -10
# 2. Identify old log files
$ find /var/log -name "*.log" -mtime +30 -type f
# 3. Safe deletion sequence
$ find /var/log -name "*.log" -mtime +30 -type f -exec ls -la {} \; # Confirm
$ find /var/log -name "*.log" -mtime +30 -type f -ok rm {} \; # Delete with confirmation
# 4. Post-deletion confirmation
$ df -h # Check if space increasedPractice Problems: Try in Safe Environment
Basics: File Operation Fundamentals
- Create a file named test.txt and write "Initial data"
- Copy test.txt as test_backup.txt
- Rename test.txt to test_renamed.txt
- Display contents of test_backup.txt
- Safely delete unnecessary files
Intermediate: Work Simulation
- Create "project" directory and create config.ini, app.py, README.md inside
- Backup entire project directory with timestamp
- Modify config.ini contents and check differences with original file
- Execute rollback procedure if problems occur
Summary: Towards Safe File Operation Mastery
Important points learned in this article:
- head/tail: Essential techniques for log analysis and real-time monitoring
- file/stat: Strong allies for security checks and debugging
- Pipes & Redirection: Powerful command combinations
- Practical Exercises: Comprehensive utilization in work scenarios
In professional work, operate with "safety first" and never skip backups and verification. Always remember that a single mistake can have irreversible consequences.