Linux File Operations Advanced
After mastering basic file operations, let's learn advanced techniques. In this advanced edition, we'll explain practical file analysis and operation techniques using head, tail, file, stat, pipes, and redirection.
Table of Contents
1. head/tail - Log Analysis and File Analysis Techniques
head and tail are the most frequently used file analysis commands in professional work. They're indispensable especially for server log analysis and debugging.
head - Smart Display of File Beginning
π Basic Usage
$ head access.log
Displays first 10 lines by default. Perfect for understanding file structure
π Flexible Display with Line Number Specification
$ head -n 5 error.log # First 5 lines $ head -5 error.log # Short form $ head -n 100 config.txt # First 100 lines
π Batch Check of Multiple Files
$ head -n 3 *.log
==> access.log <== 192.168.1.10 - - [11/Jan/2025:10:00:01] "GET /" 192.168.1.11 - - [11/Jan/2025:10:00:02] "GET /api" 192.168.1.12 - - [11/Jan/2025:10:00:03] "POST /login" ==> error.log <== [2025-01-11 10:00:01] ERROR: Database connection failed [2025-01-11 10:00:05] WARN: Slow query detected [2025-01-11 10:00:10] ERROR: Authentication failed
Check contents of multiple log files at once
tail - Latest Information and Real-time Monitoring
π°οΈ Check Latest Errors
$ tail error.log
Shows last 10 lines by default. Quickly identify latest errors and events
π₯ Real-time Log Monitoring (Most Important)
$ tail -f /var/log/app.log
Most frequently used feature in professional work. Monitor logs in real-time, automatically displaying new lines as they're added
Exit method: Ctrl + C
π Display from Specific Position
$ tail -n +50 large_file.txt # From line 50 to end $ tail -n 20 access.log # Last 20 lines
Advanced Log Analysis Techniques in Professional Work
π Identify and Track Error Occurrence Time
# Check context just before error occurrence $ grep -n "ERROR" app.log | tail -1 # Get line number of latest error 47:ERROR: Connection timeout # Detailed check of surroundings $ head -n 50 app.log | tail -n 10 # Display lines 41-50
π Log Rotation Compatible Monitoring
# Continue monitoring even when log file rotates $ tail -F /var/log/app.log # Capital F handles file recreation
π₯ Simultaneous Monitoring of Multiple Logs
# Monitor multiple log files simultaneously $ tail -f /var/log/app.log /var/log/error.log # More advanced method: multitail command $ multitail /var/log/app.log /var/log/error.log /var/log/access.log
π‘ Professional head/tail Usage Techniques
Performance Issue Tracking
# Check latest trends in access log $ tail -f access.log | grep "slow\|timeout\|error"
Real-time Monitoring During Deployment
# In separate terminal during deployment $ tail -f /var/log/deploy.log | tee deploy_$(date +%Y%m%d).log
Appropriate Selection Based on File Size
# cat for small files, head/tail for large files $ wc -l logfile.txt # Check line count $ [[ $(wc -l < file.txt) -gt 50 ]] && head file.txt || cat file.txt
2. file/stat - Detailed File Information Investigation
Commands for investigating file identity and detailed information. Very useful for security checks and debugging.
file - File Type Identification
π Basic File Diagnosis
$ file mysterious_file
mysterious_file: UTF-8 Unicode text, with CRLF line terminators
Identifies file type, encoding, and line endings
π Batch Diagnosis of Multiple Files
$ file *
config.txt: ASCII text data.bin: data image.jpg: JPEG image data, JFIF standard script.sh: Bourne-Again shell script, ASCII text executable archive.tar.gz: gzip compressed data
Batch identification of file types in directory
π‘οΈ Security Check Utilization
# Detect extension spoofing $ file suspicious_file.txt suspicious_file.txt: PE32 executable (console) Intel 80386, for MS Windows # .txt but actually a Windows executable!
If extension and actual file type differ, possibly malicious file
stat - Detailed File Information
π Complete File Information Display
$ stat important_file.txt
File: important_file.txt Size: 1024 Blocks: 8 IO Block: 4096 regular file Device: 801h/2049d Inode: 1234567 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1000/username) Gid: ( 1000/usergroup) Access: 2025-01-11 10:30:45.123456789 +0900 Modify: 2025-01-11 10:25:30.987654321 +0900 Change: 2025-01-11 10:25:30.987654321 +0900 Birth: 2025-01-11 10:20:15.555666777 +0900
Detailed information on file size, permissions, timestamps
π°οΈ Display with Custom Format
# Display only size and modification time $ stat --format="%n: %s bytes, modified %y" *.txt
config.txt: 2048 bytes, modified 2025-01-11 09:15:22.123456789 +0900 log.txt: 51200 bytes, modified 2025-01-11 10:45:33.987654321 +0900
Practical Work Scenarios
π΅οΈ File Change Tracking Investigation
# Security incident response $ stat --format="%n - Last modified: %y, Last accessed: %x" /etc/passwd /etc/passwd - Last modified: 2025-01-11 08:30:15.123456789 +0900, Last accessed: 2025-01-11 10:45:22.987654321 +0900 # Check for unauthorized access
πΎ Disk Space Diagnosis
# Detailed info on large files $ stat --format="%n: %s bytes (%S blocks)" large_files/* database.db: 1073741824 bytes (262144 blocks) backup.tar.gz: 536870912 bytes (131072 blocks)
π Symbolic Link Diagnosis
$ file suspicious_link suspicious_link: symbolic link to /tmp/malicious_file $ stat suspicious_link File: suspicious_link -> /tmp/malicious_file Size: 18 Blocks: 0 IO Block: 4096 symbolic link
Identify link target and verify safety
π‘ Professional file/stat Command Utilization
File Type Filtering in Batch Processing
# Extract only executable files
for f in *; do
[[ $(file "$f") == *"executable"* ]] && echo "$f"
done
Batch Sorting by File Size
# File list sorted by size stat --format="%s %n" * | sort -n | tail -10
3. Pipes and Redirection
Combine multiple commands to achieve powerful processing.
Pipe (|)
Pass the output of one command as input to another command.
Basic Pipe
$ ls -la | grep ".txt"
Display only .txt files
Redirection
| Symbol | Description | Example |
|---|---|---|
> |
Overwrite output to file | ls > list.txt |
>> |
Append output to file | echo "text" >> file.txt |
< |
Input from file | sort < data.txt |
2> |
Redirect error output | command 2> error.log |
&> |
Both stdout and stderr to same file | command &> all.log |
4. Practical Combination Techniques
Introduce command combination examples commonly used in actual work.
Example 1: Extract and Count Errors from Log File
$ grep "ERROR" app.log | cut -d' ' -f3 | sort | uniq -c | sort -rn
Display occurrence count by error type in descending order
Example 2: Find Top 10 Largest Files
$ find . -type f -exec ls -lh {} \; | sort -k5 -rh | head -10
Top 10 largest files under current directory
Example 3: Count Files with Specific Extension
$ find . -name "*.txt" | wc -l
Display total number of .txt files
Example 4: Top 5 Memory Usage Processes
$ ps aux | sort -k4 -rn | head -5
Display 5 processes with highest memory usage
Example 5: Create File Backups
$ find . -name "*.conf" -exec cp {} {}.backup \;
Create backups of all .conf files
5. Practical Exercises: Daily Work Scenarios
Learn practical use of file operation commands through actual work scenarios.
π» Scenario 1: Server Maintenance Work
Task: Log File Rotation and Archiving
# 1. Check current log file size $ ls -lah /var/log/app.log # 2. Check latest errors $ tail -20 /var/log/app.log | grep "ERROR" # 3. Create safe archive $ cp /var/log/app.log /var/log/archive/app.log.$(date +%Y%m%d) # 4. Clear log (execute while service stopped) $ > /var/log/app.log # Empty the file
π Scenario 2: Application Deployment
Task: Safe Deployment of New Version
# 1. Backup current version $ cp -rp /opt/myapp /opt/myapp.backup.$(date +%Y%m%d_%H%M%S) # 2. Extract and verify new version $ tar -tf new_version.tar.gz | head -10 # Check contents $ tar -xzf new_version.tar.gz -C /tmp/ # Temporary extraction # 3. Merge configuration files $ cp /opt/myapp/config.ini /tmp/myapp/config.ini $ diff /opt/myapp/config.ini /tmp/myapp/config.ini # Check diff # 4. Safe overwrite to production $ mv /opt/myapp /opt/myapp.old $ mv /tmp/myapp /opt/myapp
π Scenario 3: Security Incident Response
Task: Investigate Unauthorized Access and Preserve Evidence
# 1. Emergency backup of access logs $ cp /var/log/access.log /home/incident/access.log.$(date +%Y%m%d_%H%M%S) # 2. Identify unauthorized access time $ grep "suspicious_pattern" /var/log/access.log | head -1 $ grep "suspicious_pattern" /var/log/access.log | tail -1 # 3. Extract logs from relevant time period $ awk '/2025-01-11 10:30:/,/2025-01-11 11:00:/' /var/log/access.log > incident_logs.txt # 4. Collect related file information $ stat /var/log/access.log > file_metadata.txt $ file /var/log/access.log >> file_metadata.txt
π Scenario 4: Disk Space Cleanup
Task: Safe Deletion of Unnecessary Files
# 1. Check disk usage
$ df -h
$ du -sh /var/log/* | sort -hr | head -10
# 2. Identify old log files
$ find /var/log -name "*.log" -mtime +30 -type f
# 3. Safe deletion sequence
$ find /var/log -name "*.log" -mtime +30 -type f -exec ls -la {} \; # Confirm
$ find /var/log -name "*.log" -mtime +30 -type f -ok rm {} \; # Delete with confirmation
# 4. Post-deletion confirmation
$ df -h # Check if space increased
π― Practice Problems: Try in Safe Environment!
Basics: File Operation Fundamentals
- Create a file named test.txt and write "Initial data"
- Copy test.txt as test_backup.txt
- Rename test.txt to test_renamed.txt
- Display contents of test_backup.txt
- Safely delete unnecessary files
Intermediate: Work Simulation
- Create "project" directory and create config.ini, app.py, README.md inside
- Backup entire project directory with timestamp
- Modify config.ini contents and check differences with original file
- Execute rollback procedure if problems occur
Summary: Towards Safe File Operation Mastery
Important points learned in this article:
- head/tail: Essential techniques for log analysis and real-time monitoring
- file/stat: Strong allies for security checks and debugging
- Pipes & Redirection: Powerful command combinations
- Practical Exercises: Comprehensive utilization in work scenarios
Most Important: In professional work, operate with "safety first" and never skip backups and verification.
Always remember that a single mistake can have irreversible consequences.
π File Operations Master Series Completed!
Great work! Through the Basics and Advanced editions, you've become a Linux file operations expert.
Use your acquired knowledge to practice efficient file management and analysis.
π Complete File Operations Series
- Basics Edition - Basic operations and safety measures for cp, mv, rm, cat
- Advanced Edition (This Article) - head, tail, file, stat, pipes, practical exercises
This site is a participant in the Amazon Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.co.jp. Product prices are not affected.
π Recommended Books for File Operations & Command Utilization
We've carefully selected practice-focused learning books to efficiently master advanced file operation techniques and command utilization. Learn practical usage of head, tail, grep, find, pipes, and more.
π Detailed Shell Scripting (詳解γ·γ§γ«γΉγ―γͺγγ)
Target Level: Intermediate to Advanced
Detailed explanation of advanced file operation techniques and practical utilization of pipes and redirection. Practical book for mastering powerful text processing techniques combining filter commands like grep, find, and awk.
π Introduction to UNIX Shell Programming (ε ₯ιUNIXγ·γ§γ«γγγ°γ©γγ³γ°)
Target Level: Beginner to Intermediate
Systematic learning from file operation command basics to practical utilization in shell scripts. Detailed explanation of practical patterns combining text processing tools like grep, sed, and awk.
π Advanced File Command Utilization (ι«εΊ¦γͺγγ‘γ€γ«γ³γγ³γζ΄»η¨θ‘)
Target Level: Intermediate to Advanced
Practical learning of advanced commands like find, sort, uniq, diff. Specialized book for mastering advanced techniques useful in professional work such as large-scale file processing, log analysis, and data conversion.
π Bash Command Line Introduction (Bashγ³γγ³γγ©γ€γ³ε ₯ι)
Target Level: Beginner to Intermediate
Practical learning from basics to advanced Bash command line operations and file operations. Detailed explanation of efficient usage of pipes, redirection, wildcards, etc. with abundant practical examples.