Shell Scripting Practical: Advanced Techniques and Real Examples
After mastering shell scripting basics, learn practical advanced techniques. From functions and arrays to file operations, error handling, and real-world examples, master professional script creation techniques.
Table of Contents
1. Functions
Defining and Calling Functions
Basic Function
#!/bin/bash
# Function definition
greet() {
echo "Hello, $1!"
}
# Function calls
greet "Taro"
greet "Hanako"
Function with Return Value
#!/bin/bash
# Function to calculate square
square() {
local num=$1
local result=$((num * num))
echo $result
}
# Usage example
number=5
result=$(square $number)
echo "Square of $number is $result"
Multi-argument Function
#!/bin/bash
# Function to display file info
file_info() {
local filepath=$1
if [ -f "$filepath" ]; then
echo "File: $filepath"
echo "Size: $(wc -c < "$filepath") bytes"
echo "Lines: $(wc -l < "$filepath") lines"
echo "Last modified: $(stat -c %y "$filepath")"
else
echo "File $filepath does not exist"
return 1
fi
}
# Usage example
file_info "/etc/passwd"
file_info "nonexistent.txt"
Advanced Functions
Local and Global Variables
#!/bin/bash
global_var="Global"
demo_scope() {
local local_var="Local"
global_var="Modified Global"
echo "Inside function: local_var = $local_var"
echo "Inside function: global_var = $global_var"
}
echo "Before call: global_var = $global_var"
demo_scope
echo "After call: global_var = $global_var"
echo "Outside function: local_var = $local_var" # Empty
Recursive Function
#!/bin/bash
# Recursive factorial function
factorial() {
local n=$1
if [ $n -le 1 ]; then
echo 1
else
local prev=$(factorial $((n - 1)))
echo $((n * prev))
fi
}
# Usage example
for i in {1..5}; do
result=$(factorial $i)
echo "$i! = $result"
done
Function with Error Handling
#!/bin/bash
# Safe directory creation function
safe_mkdir() {
local dir_path=$1
if [ -z "$dir_path" ]; then
echo "Error: Directory path not specified" >&2
return 1
fi
if [ -d "$dir_path" ]; then
echo "Directory $dir_path already exists"
return 0
fi
if mkdir -p "$dir_path" 2>/dev/null; then
echo "Created directory $dir_path"
return 0
else
echo "Error: Failed to create directory $dir_path" >&2
return 1
fi
}
# Usage example
safe_mkdir "/tmp/test_dir"
safe_mkdir "/root/forbidden" # Permission error example
2. Arrays
Basic Array Operations
Creating Arrays and Accessing Elements
#!/bin/bash
# Define arrays
fruits=("apple" "banana" "orange" "grape")
numbers=(1 2 3 4 5)
# Access elements
echo "First fruit: ${fruits[0]}"
echo "Third number: ${numbers[2]}"
# Display all elements
echo "All fruits: ${fruits[@]}"
echo "All numbers: ${numbers[*]}"
# Array length
echo "Number of fruits: ${#fruits[@]}"
Adding and Removing Array Elements
#!/bin/bash
# Initial array
colors=("red" "green" "blue")
echo "Initial array: ${colors[@]}"
# Add elements
colors+=("yellow")
colors[${#colors[@]}]="purple"
echo "After adding: ${colors[@]}"
# Remove element (using unset)
unset colors[1] # Remove "green"
echo "After removal: ${colors[@]}"
# Show indices
echo "Indices: ${!colors[@]}"
Associative Arrays (Bash 4.0+)
#!/bin/bash
# Declare associative array
declare -A person
# Set values
person["name"]="Taro Tanaka"
person["age"]="30"
person["city"]="Tokyo"
# Get values
echo "Name: ${person["name"]}"
echo "Age: ${person["age"]}"
echo "City: ${person["city"]}"
# Get all keys
echo "All keys: ${!person[@]}"
# Get all values
echo "All values: ${person[@]}"
Practical Array Usage
#!/bin/bash
# Log file pattern array
log_patterns=("/var/log/*.log" "/tmp/*.log" "$HOME/*.log")
echo "Log file search results:"
for pattern in "${log_patterns[@]}"; do
echo "Pattern: $pattern"
# Store files in array
files=($pattern)
if [ ${#files[@]} -gt 0 ] && [ -f "${files[0]}" ]; then
for file in "${files[@]}"; do
if [ -f "$file" ]; then
size=$(wc -c < "$file")
echo " - $file ($size bytes)"
fi
done
else
echo " - No matching files"
fi
echo
done
3. File Operations
Reading and Writing Files
Various File Reading Methods
#!/bin/bash
filename="data.txt"
# Method 1: Using while read
echo "=== while read method ==="
while IFS= read -r line || [ -n "$line" ]; do
echo "Line: $line"
done < "$filename"
echo
echo "=== cat + pipe method ==="
cat "$filename" | while read -r line; do
echo "Process: $line"
done
echo
echo "=== Read into array ==="
mapfile -t lines < "$filename"
for i in "${!lines[@]}"; do
echo "Line $((i+1)): ${lines[i]}"
done
Writing Files
#!/bin/bash output_file="output.txt" # Create new (overwrite) echo "New file content" > "$output_file" # Append echo "Additional line 1" >> "$output_file" echo "Additional line 2" >> "$output_file" # Write multiple lines cat << EOF >> "$output_file" Multiple lines of text Line 2 Line 3 EOF # Verify file contents echo "File contents:" cat "$output_file"
Processing CSV Files
#!/bin/bash
# CSV file processing example
csv_file="employees.csv"
# Create CSV sample
cat << EOF > "$csv_file"
Name,Age,Department
Tanaka,30,Development
Sato,25,Sales
Suzuki,35,Management
EOF
echo "CSV processing results:"
# Skip header and process
tail -n +2 "$csv_file" | while IFS=',' read -r name age dept; do
echo "Employee: $name (${age} years old) - $dept"
done
Advanced File Operations Example
#!/bin/bash
# File sync script example
source_dir="/path/to/source"
backup_dir="/path/to/backup"
# Backup function
backup_files() {
local src="$1"
local dest="$2"
if [ ! -d "$src" ]; then
echo "Error: Source directory does not exist: $src"
return 1
fi
# Create backup directory
mkdir -p "$dest"
# Sync files
rsync -av --delete "$src/" "$dest/"
echo "Backup complete: $src -> $dest"
}
# Execute backup with logging
log_file="/tmp/backup.log"
{
echo "Backup started: $(date)"
backup_files "$source_dir" "$backup_dir"
echo "Backup ended: $(date)"
} >> "$log_file" 2>&1
4. Error Handling
Error Handling Best Practices
Strict Error Handling with set Options
#!/bin/bash # Strict error handling settings set -euo pipefail # set -e: Exit immediately on command failure # set -u: Error on undefined variable usage # set -o pipefail: Error if any pipeline command fails # Set error trap trap 'echo "Error occurred: line number $LINENO" >&2' ERR echo "Strict error handling demo" # Normal command echo "Normal processing 1" # Intentional error (comment out to run) # false # This command will fail echo "This line won't execute (stopped by above error)"
Manual Error Checking
#!/bin/bash
# File processing function (with error checking)
process_file() {
local filename="$1"
# File existence check
if [ ! -f "$filename" ]; then
echo "Error: File '$filename' not found" >&2
return 1
fi
# Read permission check
if [ ! -r "$filename" ]; then
echo "Error: No read permission for file '$filename'" >&2
return 1
fi
# Process file
local line_count
if ! line_count=$(wc -l < "$filename" 2>/dev/null); then
echo "Error: Failed to count lines in file '$filename'" >&2
return 1
fi
echo "File '$filename' line count: $line_count"
return 0
}
# Usage example
if process_file "test.txt"; then
echo "Processing completed successfully"
else
echo "Processing failed"
exit 1
fi
Error Handling with Logging
#!/bin/bash
LOG_FILE="/tmp/script.log"
# Log function
log() {
local level="$1"
shift
local message="$*"
local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
echo "[$timestamp] [$level] $message" | tee -a "$LOG_FILE"
}
# Error handling function
handle_error() {
local exit_code=$?
local line_no=$1
log "ERROR" "Script exited with error (exit code: $exit_code, line: $line_no)"
exit $exit_code
}
# Set error trap
trap 'handle_error $LINENO' ERR
# Main processing
log "INFO" "Script started"
# Dangerous operation example
dangerous_operation() {
log "INFO" "Executing dangerous operation..."
# Some processing
sleep 1
# Intentional error (50% chance)
if [ $((RANDOM % 2)) -eq 0 ]; then
log "ERROR" "Operation failed"
return 1
fi
log "INFO" "Operation succeeded"
return 0
}
if dangerous_operation; then
log "INFO" "All processing completed"
else
log "WARN" "Some processing failed, but continuing"
fi
log "INFO" "Script ended"
5. Practical Script Examples
Example 1: System Backup Script
#!/bin/bash
# System backup script
set -euo pipefail
# Configuration
BACKUP_ROOT="/backup"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=7
DATE=$(date +%Y%m%d_%H%M%S)
# Backup targets
BACKUP_DIRS=(
"/etc"
"/home"
"/var/www"
"/usr/local/bin"
)
# Log function
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}
# Cleanup function
cleanup() {
log "Starting old backup deletion"
find "$BACKUP_ROOT" -type f -name "backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete
log "Old backup deletion complete"
}
# Main backup function
main_backup() {
local backup_file="$BACKUP_ROOT/backup_$DATE.tar.gz"
log "Backup started: $backup_file"
# Create backup directory
mkdir -p "$BACKUP_ROOT"
# Create backup with tar
if tar -czf "$backup_file" "${BACKUP_DIRS[@]}" 2>/dev/null; then
local size=$(du -h "$backup_file" | cut -f1)
log "Backup successful: $backup_file (size: $size)"
else
log "Error: Backup creation failed"
exit 1
fi
}
# Permission check
if [ "$(id -u)" -ne 0 ]; then
echo "This script requires root privileges"
exit 1
fi
# Main execution
log "System backup process started"
main_backup
cleanup
log "System backup process completed"
Example 2: Log Monitoring Script
#!/bin/bash
# Log monitoring script
set -euo pipefail
# Configuration
LOG_FILE="/var/log/syslog"
ALERT_PATTERNS=("ERROR" "CRITICAL" "FAILED")
CHECK_INTERVAL=10
LAST_CHECK_FILE="/tmp/log_monitor_last_check"
# Alert sending function
send_alert() {
local message="$1"
local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
# Implement email or Slack notification here
echo "[$timestamp] ALERT: $message" >> "/var/log/alerts.log"
# Simple notification (system log)
logger "LOG_MONITOR_ALERT: $message"
}
# Log monitoring main process
monitor_logs() {
local last_position=0
# Load last check position
if [ -f "$LAST_CHECK_FILE" ]; then
last_position=$(cat "$LAST_CHECK_FILE")
fi
# Get file size
local current_size=$(wc -c < "$LOG_FILE")
# Process only if new log entries exist
if [ "$current_size" -gt "$last_position" ]; then
# Extract new portion only
local new_lines=$(tail -c +$((last_position + 1)) "$LOG_FILE")
# Pattern matching
for pattern in "${ALERT_PATTERNS[@]}"; do
if echo "$new_lines" | grep -q "$pattern"; then
local matches=$(echo "$new_lines" | grep "$pattern")
send_alert "Detected pattern '$pattern' in $LOG_FILE: $matches"
fi
done
# Save current position
echo "$current_size" > "$LAST_CHECK_FILE"
fi
}
# Main loop
echo "Log monitoring started: $LOG_FILE"
echo "Monitoring patterns: ${ALERT_PATTERNS[*]}"
while true; do
monitor_logs
sleep "$CHECK_INTERVAL"
done
Example 3: File Organization Script
#!/bin/bash
# File organization script
set -euo pipefail
# Configuration
SOURCE_DIR="$HOME/Downloads"
ORGANIZE_ROOT="$HOME/Organized"
LOG_FILE="/tmp/file_organizer.log"
# File type to destination mapping
declare -A FILE_TYPES=(
["pdf"]="Documents/PDF"
["doc,docx"]="Documents/Word"
["xls,xlsx"]="Documents/Excel"
["ppt,pptx"]="Documents/PowerPoint"
["txt"]="Documents/Text"
["jpg,jpeg,png,gif"]="Images"
["mp4,avi,mov"]="Videos"
["mp3,wav,flac"]="Audio"
["zip,rar,7z,tar,gz"]="Archives"
)
# Log function
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}
# File type detection function
get_file_type() {
local filename="$1"
local extension="${filename##*.}"
extension=$(echo "$extension" | tr '[:upper:]' '[:lower:]')
for types in "${!FILE_TYPES[@]}"; do
if [[ ",$types," == *",$extension,"* ]]; then
echo "${FILE_TYPES[$types]}"
return 0
fi
done
echo "Others"
}
# File move function
move_file() {
local source_file="$1"
local file_type="$2"
local dest_dir="$ORGANIZE_ROOT/$file_type"
local filename=$(basename "$source_file")
# Create destination directory
mkdir -p "$dest_dir"
# Handle duplicate filenames
local dest_file="$dest_dir/$filename"
local counter=1
while [ -f "$dest_file" ]; do
local name="${filename%.*}"
local ext="${filename##*.}"
dest_file="$dest_dir/${name}_${counter}.${ext}"
((counter++))
done
# Move file
if mv "$source_file" "$dest_file"; then
log "Moved: $filename -> $file_type/"
else
log "Error: Failed to move $filename"
fi
}
# Main processing
organize_files() {
log "File organization started: $SOURCE_DIR"
if [ ! -d "$SOURCE_DIR" ]; then
log "Error: Source directory does not exist: $SOURCE_DIR"
exit 1
fi
local file_count=0
# Process files
while IFS= read -r -d '' file; do
if [ -f "$file" ]; then
local file_type=$(get_file_type "$(basename "$file")")
move_file "$file" "$file_type"
((file_count++))
fi
done < <(find "$SOURCE_DIR" -maxdepth 1 -type f -print0)
log "File organization complete: Processed $file_count files"
}
# Show statistics
show_statistics() {
log "=== Organization Results Statistics ==="
for dir in "$ORGANIZE_ROOT"/*; do
if [ -d "$dir" ]; then
local count=$(find "$dir" -type f | wc -l)
local dirname=$(basename "$dir")
log "$dirname: $count files"
fi
done
}
# Main execution
organize_files
show_statistics
β οΈ Common Practical Mistakes and Pitfalls
Advanced problems commonly encountered in practical script creation, with professional solutions explained.
π« Mistake 1: Incorrect Array Usage
β Problematic Example
# Treating array as string
files=("file1.txt" "file 2.txt" "file3.txt")
for file in $files; do # Split by spaces
echo $file
done
# Index confusion
echo ${files[1, 2]} # Syntax error
Array not expanded properly, issues with filenames containing spaces.
β Correct Usage
# Proper array expansion
files=("file1.txt" "file 2.txt" "file3.txt")
for file in "${files[@]}"; do # Expand properly as array
echo "$file"
done
# Multiple index specification
echo "${files[1]}" "${files[2]}" # Specify individually
Use "${array[@]}" to safely expand entire array.
π« Mistake 2: Insufficient Understanding of Function Scope
β Unexpected Behavior
#!/bin/bash
counter=0
increment() {
counter=$((counter + 1)) # Modifying global variable
local result=$counter # Local variable
echo $result
}
increment
echo "Global counter: $counter" # Unintended modification
Global variable gets modified unintentionally.
β Proper Scope Management
#!/bin/bash
global_counter=0
increment() {
local local_counter=$1 # Accept argument
local_counter=$((local_counter + 1))
echo $local_counter # Return result (echo)
}
# Use function return value
result=$(increment $global_counter)
global_counter=$result # Explicitly update
Use local appropriately and control function's scope of influence.
π« Mistake 3: Pipeline Variable Modification Trap
β Unexpected Results
#!/bin/bash
count=0
# Executed in pipeline subshell
cat file.txt | while read line; do
count=$((count + 1))
done
echo "Lines: $count" # Remains 0 (not changed)
Pipeline right side executes in subshell, variable changes not reflected in parent.
β Correct Methods
#!/bin/bash
count=0
# Use redirection
while read line; do
count=$((count + 1))
done < file.txt
echo "Lines: $count" # Correctly counted
# Or use command substitution
count=$(wc -l < file.txt)
echo "Lines: $count"
Avoid the problem with redirection or command substitution.
π« Mistake 4: Inadequate Error Handling
β Insufficient Error Handling
#!/bin/bash # File processing (no error checking) cp source.txt backup.txt rm source.txt # Delete even if copy failed # Ignore external command results curl -o data.json http://api.example.com/data process_data data.json # Continue processing even if download failed
Errors can cascade into serious problems.
β Robust Error Handling
#!/bin/bash
set -euo pipefail
# File processing (proper error checking)
if cp source.txt backup.txt; then
echo "Backup successful"
rm source.txt
else
echo "Error: Backup failed" >&2
exit 1
fi
# Verify external command results
if curl -f -o data.json http://api.example.com/data; then
process_data data.json
else
echo "Error: Failed to retrieve data" >&2
exit 1
fi
Check errors at each step and handle appropriately.
π« Mistake 5: Implementation Without Security Consideration
β Security Issues
#!/bin/bash # Execute user input directly (dangerous) read -p "Enter command: " user_command eval $user_command # Predictable temp filename temp_file="/tmp/script_data.txt" echo "sensitive data" > $temp_file # Password on command line mysql -u user -p'password123' -e "SELECT * FROM users"
Risks of arbitrary command execution and password leakage.
β Secure Implementation
#!/bin/bash
# Input validation
read -p "Enter filename: " filename
if [[ "$filename" =~ ^[a-zA-Z0-9._-]+$ ]]; then
echo "Processing: $filename"
else
echo "Error: Invalid filename" >&2
exit 1
fi
# Safe temporary file creation
temp_file=$(mktemp)
trap 'rm -f "$temp_file"' EXIT
echo "sensitive data" > "$temp_file"
# Read password from config file
if [ -f ~/.mysql_config ]; then
source ~/.mysql_config
mysql -u "$DB_USER" -p"$DB_PASS" -e "SELECT * FROM users"
fi
Ensure security with input validation, safe temp files, and config file usage.
π« Mistake 6: Implementation Without Performance Consideration
β Inefficient Processing
#!/bin/bash
# Process many files individually (slow)
for file in /path/to/large/directory/*; do
wc -l "$file" # Start process for each file
done
# Repeatedly execute same command
for i in {1..1000}; do
current_time=$(date +%s) # Execute date command each time
echo "Processing $i at $current_time"
done
Massive process creation makes processing extremely slow.
β Efficient Implementation
#!/bin/bash
# Optimize with batch processing
find /path/to/large/directory -name "*" -type f -exec wc -l {} +
# Get once and reuse
start_time=$(date +%s)
for i in {1..1000}; do
current_time=$((start_time + i))
echo "Processing $i at $current_time"
done
# Speed up with parallel processing
export -f process_file
find /path -name "*.txt" | xargs -P 4 -I {} bash -c 'process_file "$@"' _ {}
Improve performance with batch processing, value reuse, and parallel processing.
π― Best Practices for Advanced Users
π Standardized Logging
# Unified log function
log() {
local level=$1; shift
local message="$*"
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $message" >&2
}
log "INFO" "Processing started"
log "ERROR" "File not found: $filename"
log "DEBUG" "Variable value: var=$var"
π Separate Configuration Management
# config.sh
DB_HOST="localhost"
DB_PORT="3306"
MAX_RETRIES=3
LOG_LEVEL="INFO"
# main.sh
#!/bin/bash
CONFIG_FILE="${1:-./config.sh}"
if [ -f "$CONFIG_FILE" ]; then
source "$CONFIG_FILE"
else
echo "Config file not found: $CONFIG_FILE" >&2
exit 1
fi
π§ͺ Testable Design
# Testable function design
process_data() {
local input_file="$1"
local output_file="$2"
# Input validation
[ -f "$input_file" ] || return 1
# Processing
awk '{print NR ": " $0}' "$input_file" > "$output_file"
# Result validation
[ -f "$output_file" ] && [ -s "$output_file" ]
}
# Test script
test_process_data() {
local test_input=$(mktemp)
local test_output=$(mktemp)
echo -e "line1\nline2" > "$test_input"
if process_data "$test_input" "$test_output"; then
echo "β
Test passed"
else
echo "β Test failed"
fi
rm -f "$test_input" "$test_output"
}
Shell Scripting Best Practices
π Coding Standards
- Use meaningful variable names
- Functions focus on single responsibility
- Clarify intent with comments
- Consistent indentation (2 spaces recommended)
- Break long lines appropriately
π Security
- Always validate input values
- Create temp files in safe locations
- Set permissions to minimum required
- Don't hardcode sensitive information
- Validate external command results
π Debugging
- Enable debug mode with
set -x - Implement appropriate logging
- Record state on errors
- Develop and test incrementally
- Use static analysis tools like ShellCheck
Summary
Mastering practical shell scripting techniques enables efficient and reliable automation.
Key Points
- Functions improve code reusability and maintainability
- Arrays streamline complex data processing
- Proper error handling creates robust scripts
- Apply practical examples to real work
π Complete Shell Scripting Series
- Basics - Variables, conditionals, loops, functions
- Practical (This Article) - Real-world scenarios, error handling, optimization
This site is a participant in the Amazon Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.co.jp. Product prices are not affected.
π Recommended Books for Practical Shell Scripting & Automation
Carefully selected practical-focused books to efficiently master shell scripting techniques and automation know-how. Learn deep knowledge of functions, arrays, error handling, and utility script creation.
π Detailed Explanation of Shell Scripting (Japanese)
Target Level: Intermediate~Advanced
Detailed explanation of advanced shell script techniques and practical know-how. A specialized book for systematically mastering practical techniques that are immediately useful in real work: functions, arrays, error handling, pipe utilization, etc.
π Introduction to UNIX Shell Programming (Japanese)
Target Level: Beginner~Intermediate
Systematic learning from shell scripting basics to practical applications. A practical guide for progressively mastering techniques useful in real work: functions, arrays, file operations, automation script creation, etc.
π Shell Script Basic Reference (Japanese)
Target Level: Intermediate
Complete coverage of shell script syntax and practical patterns in reference format. Detailed explanations with abundant practical examples of frequently used techniques in real work: functions, arrays, control structures, error handling, etc.
π Bash Command Line Introduction (Japanese)
Target Level: Beginner~Intermediate
Practical learning from Bash command line operations to script creation fundamentals through applications. Detailed explanations with abundant practical examples of efficient usage: variables, functions, control structures, pipes, etc.