Shell Scripting Practical: Advanced Techniques and Real Examples

After mastering shell scripting basics, learn practical advanced techniques. From functions and arrays to file operations, error handling, and real-world examples, master professional script creation techniques.

Table of Contents

  1. Functions
  2. Arrays
  3. File Operations
  4. Error Handling
  5. Practical Script Examples

1. Functions

Defining and Calling Functions

Basic Function

#!/bin/bash

# Function definition
greet() {
    echo "Hello, $1!"
}

# Function calls
greet "Taro"
greet "Hanako"

Function with Return Value

#!/bin/bash

# Function to calculate square
square() {
    local num=$1
    local result=$((num * num))
    echo $result
}

# Usage example
number=5
result=$(square $number)
echo "Square of $number is $result"

Multi-argument Function

#!/bin/bash

# Function to display file info
file_info() {
    local filepath=$1

    if [ -f "$filepath" ]; then
        echo "File: $filepath"
        echo "Size: $(wc -c < "$filepath") bytes"
        echo "Lines: $(wc -l < "$filepath") lines"
        echo "Last modified: $(stat -c %y "$filepath")"
    else
        echo "File $filepath does not exist"
        return 1
    fi
}

# Usage example
file_info "/etc/passwd"
file_info "nonexistent.txt"

Advanced Functions

Local and Global Variables

#!/bin/bash

global_var="Global"

demo_scope() {
    local local_var="Local"
    global_var="Modified Global"

    echo "Inside function: local_var = $local_var"
    echo "Inside function: global_var = $global_var"
}

echo "Before call: global_var = $global_var"
demo_scope
echo "After call: global_var = $global_var"
echo "Outside function: local_var = $local_var"  # Empty

Recursive Function

#!/bin/bash

# Recursive factorial function
factorial() {
    local n=$1

    if [ $n -le 1 ]; then
        echo 1
    else
        local prev=$(factorial $((n - 1)))
        echo $((n * prev))
    fi
}

# Usage example
for i in {1..5}; do
    result=$(factorial $i)
    echo "$i! = $result"
done

Function with Error Handling

#!/bin/bash

# Safe directory creation function
safe_mkdir() {
    local dir_path=$1

    if [ -z "$dir_path" ]; then
        echo "Error: Directory path not specified" >&2
        return 1
    fi

    if [ -d "$dir_path" ]; then
        echo "Directory $dir_path already exists"
        return 0
    fi

    if mkdir -p "$dir_path" 2>/dev/null; then
        echo "Created directory $dir_path"
        return 0
    else
        echo "Error: Failed to create directory $dir_path" >&2
        return 1
    fi
}

# Usage example
safe_mkdir "/tmp/test_dir"
safe_mkdir "/root/forbidden"  # Permission error example

2. Arrays

Basic Array Operations

Creating Arrays and Accessing Elements

#!/bin/bash

# Define arrays
fruits=("apple" "banana" "orange" "grape")
numbers=(1 2 3 4 5)

# Access elements
echo "First fruit: ${fruits[0]}"
echo "Third number: ${numbers[2]}"

# Display all elements
echo "All fruits: ${fruits[@]}"
echo "All numbers: ${numbers[*]}"

# Array length
echo "Number of fruits: ${#fruits[@]}"

Adding and Removing Array Elements

#!/bin/bash

# Initial array
colors=("red" "green" "blue")
echo "Initial array: ${colors[@]}"

# Add elements
colors+=("yellow")
colors[${#colors[@]}]="purple"
echo "After adding: ${colors[@]}"

# Remove element (using unset)
unset colors[1]  # Remove "green"
echo "After removal: ${colors[@]}"

# Show indices
echo "Indices: ${!colors[@]}"

Associative Arrays (Bash 4.0+)

#!/bin/bash

# Declare associative array
declare -A person

# Set values
person["name"]="Taro Tanaka"
person["age"]="30"
person["city"]="Tokyo"

# Get values
echo "Name: ${person["name"]}"
echo "Age: ${person["age"]}"
echo "City: ${person["city"]}"

# Get all keys
echo "All keys: ${!person[@]}"

# Get all values
echo "All values: ${person[@]}"

Practical Array Usage

#!/bin/bash

# Log file pattern array
log_patterns=("/var/log/*.log" "/tmp/*.log" "$HOME/*.log")

echo "Log file search results:"
for pattern in "${log_patterns[@]}"; do
    echo "Pattern: $pattern"

    # Store files in array
    files=($pattern)

    if [ ${#files[@]} -gt 0 ] && [ -f "${files[0]}" ]; then
        for file in "${files[@]}"; do
            if [ -f "$file" ]; then
                size=$(wc -c < "$file")
                echo "  - $file ($size bytes)"
            fi
        done
    else
        echo "  - No matching files"
    fi
    echo
done

3. File Operations

Reading and Writing Files

Various File Reading Methods

#!/bin/bash

filename="data.txt"

# Method 1: Using while read
echo "=== while read method ==="
while IFS= read -r line || [ -n "$line" ]; do
    echo "Line: $line"
done < "$filename"

echo
echo "=== cat + pipe method ==="
cat "$filename" | while read -r line; do
    echo "Process: $line"
done

echo
echo "=== Read into array ==="
mapfile -t lines < "$filename"
for i in "${!lines[@]}"; do
    echo "Line $((i+1)): ${lines[i]}"
done

Writing Files

#!/bin/bash

output_file="output.txt"

# Create new (overwrite)
echo "New file content" > "$output_file"

# Append
echo "Additional line 1" >> "$output_file"
echo "Additional line 2" >> "$output_file"

# Write multiple lines
cat << EOF >> "$output_file"
Multiple lines of text
Line 2
Line 3
EOF

# Verify file contents
echo "File contents:"
cat "$output_file"

Processing CSV Files

#!/bin/bash

# CSV file processing example
csv_file="employees.csv"

# Create CSV sample
cat << EOF > "$csv_file"
Name,Age,Department
Tanaka,30,Development
Sato,25,Sales
Suzuki,35,Management
EOF

echo "CSV processing results:"
# Skip header and process
tail -n +2 "$csv_file" | while IFS=',' read -r name age dept; do
    echo "Employee: $name (${age} years old) - $dept"
done

Advanced File Operations Example

#!/bin/bash

# File sync script example
source_dir="/path/to/source"
backup_dir="/path/to/backup"

# Backup function
backup_files() {
    local src="$1"
    local dest="$2"

    if [ ! -d "$src" ]; then
        echo "Error: Source directory does not exist: $src"
        return 1
    fi

    # Create backup directory
    mkdir -p "$dest"

    # Sync files
    rsync -av --delete "$src/" "$dest/"

    echo "Backup complete: $src -> $dest"
}

# Execute backup with logging
log_file="/tmp/backup.log"
{
    echo "Backup started: $(date)"
    backup_files "$source_dir" "$backup_dir"
    echo "Backup ended: $(date)"
} >> "$log_file" 2>&1

4. Error Handling

Error Handling Best Practices

Strict Error Handling with set Options

#!/bin/bash

# Strict error handling settings
set -euo pipefail

# set -e: Exit immediately on command failure
# set -u: Error on undefined variable usage
# set -o pipefail: Error if any pipeline command fails

# Set error trap
trap 'echo "Error occurred: line number $LINENO" >&2' ERR

echo "Strict error handling demo"

# Normal command
echo "Normal processing 1"

# Intentional error (comment out to run)
# false  # This command will fail

echo "This line won't execute (stopped by above error)"

Manual Error Checking

#!/bin/bash

# File processing function (with error checking)
process_file() {
    local filename="$1"

    # File existence check
    if [ ! -f "$filename" ]; then
        echo "Error: File '$filename' not found" >&2
        return 1
    fi

    # Read permission check
    if [ ! -r "$filename" ]; then
        echo "Error: No read permission for file '$filename'" >&2
        return 1
    fi

    # Process file
    local line_count
    if ! line_count=$(wc -l < "$filename" 2>/dev/null); then
        echo "Error: Failed to count lines in file '$filename'" >&2
        return 1
    fi

    echo "File '$filename' line count: $line_count"
    return 0
}

# Usage example
if process_file "test.txt"; then
    echo "Processing completed successfully"
else
    echo "Processing failed"
    exit 1
fi

Error Handling with Logging

#!/bin/bash

LOG_FILE="/tmp/script.log"

# Log function
log() {
    local level="$1"
    shift
    local message="$*"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    echo "[$timestamp] [$level] $message" | tee -a "$LOG_FILE"
}

# Error handling function
handle_error() {
    local exit_code=$?
    local line_no=$1

    log "ERROR" "Script exited with error (exit code: $exit_code, line: $line_no)"
    exit $exit_code
}

# Set error trap
trap 'handle_error $LINENO' ERR

# Main processing
log "INFO" "Script started"

# Dangerous operation example
dangerous_operation() {
    log "INFO" "Executing dangerous operation..."

    # Some processing
    sleep 1

    # Intentional error (50% chance)
    if [ $((RANDOM % 2)) -eq 0 ]; then
        log "ERROR" "Operation failed"
        return 1
    fi

    log "INFO" "Operation succeeded"
    return 0
}

if dangerous_operation; then
    log "INFO" "All processing completed"
else
    log "WARN" "Some processing failed, but continuing"
fi

log "INFO" "Script ended"

5. Practical Script Examples

Example 1: System Backup Script

#!/bin/bash

# System backup script
set -euo pipefail

# Configuration
BACKUP_ROOT="/backup"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=7
DATE=$(date +%Y%m%d_%H%M%S)

# Backup targets
BACKUP_DIRS=(
    "/etc"
    "/home"
    "/var/www"
    "/usr/local/bin"
)

# Log function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

# Cleanup function
cleanup() {
    log "Starting old backup deletion"
    find "$BACKUP_ROOT" -type f -name "backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete
    log "Old backup deletion complete"
}

# Main backup function
main_backup() {
    local backup_file="$BACKUP_ROOT/backup_$DATE.tar.gz"

    log "Backup started: $backup_file"

    # Create backup directory
    mkdir -p "$BACKUP_ROOT"

    # Create backup with tar
    if tar -czf "$backup_file" "${BACKUP_DIRS[@]}" 2>/dev/null; then
        local size=$(du -h "$backup_file" | cut -f1)
        log "Backup successful: $backup_file (size: $size)"
    else
        log "Error: Backup creation failed"
        exit 1
    fi
}

# Permission check
if [ "$(id -u)" -ne 0 ]; then
    echo "This script requires root privileges"
    exit 1
fi

# Main execution
log "System backup process started"
main_backup
cleanup
log "System backup process completed"

Example 2: Log Monitoring Script

#!/bin/bash

# Log monitoring script
set -euo pipefail

# Configuration
LOG_FILE="/var/log/syslog"
ALERT_PATTERNS=("ERROR" "CRITICAL" "FAILED")
CHECK_INTERVAL=10
LAST_CHECK_FILE="/tmp/log_monitor_last_check"

# Alert sending function
send_alert() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    # Implement email or Slack notification here
    echo "[$timestamp] ALERT: $message" >> "/var/log/alerts.log"

    # Simple notification (system log)
    logger "LOG_MONITOR_ALERT: $message"
}

# Log monitoring main process
monitor_logs() {
    local last_position=0

    # Load last check position
    if [ -f "$LAST_CHECK_FILE" ]; then
        last_position=$(cat "$LAST_CHECK_FILE")
    fi

    # Get file size
    local current_size=$(wc -c < "$LOG_FILE")

    # Process only if new log entries exist
    if [ "$current_size" -gt "$last_position" ]; then
        # Extract new portion only
        local new_lines=$(tail -c +$((last_position + 1)) "$LOG_FILE")

        # Pattern matching
        for pattern in "${ALERT_PATTERNS[@]}"; do
            if echo "$new_lines" | grep -q "$pattern"; then
                local matches=$(echo "$new_lines" | grep "$pattern")
                send_alert "Detected pattern '$pattern' in $LOG_FILE: $matches"
            fi
        done

        # Save current position
        echo "$current_size" > "$LAST_CHECK_FILE"
    fi
}

# Main loop
echo "Log monitoring started: $LOG_FILE"
echo "Monitoring patterns: ${ALERT_PATTERNS[*]}"

while true; do
    monitor_logs
    sleep "$CHECK_INTERVAL"
done

Example 3: File Organization Script

#!/bin/bash

# File organization script
set -euo pipefail

# Configuration
SOURCE_DIR="$HOME/Downloads"
ORGANIZE_ROOT="$HOME/Organized"
LOG_FILE="/tmp/file_organizer.log"

# File type to destination mapping
declare -A FILE_TYPES=(
    ["pdf"]="Documents/PDF"
    ["doc,docx"]="Documents/Word"
    ["xls,xlsx"]="Documents/Excel"
    ["ppt,pptx"]="Documents/PowerPoint"
    ["txt"]="Documents/Text"
    ["jpg,jpeg,png,gif"]="Images"
    ["mp4,avi,mov"]="Videos"
    ["mp3,wav,flac"]="Audio"
    ["zip,rar,7z,tar,gz"]="Archives"
)

# Log function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

# File type detection function
get_file_type() {
    local filename="$1"
    local extension="${filename##*.}"
    extension=$(echo "$extension" | tr '[:upper:]' '[:lower:]')

    for types in "${!FILE_TYPES[@]}"; do
        if [[ ",$types," == *",$extension,"* ]]; then
            echo "${FILE_TYPES[$types]}"
            return 0
        fi
    done

    echo "Others"
}

# File move function
move_file() {
    local source_file="$1"
    local file_type="$2"
    local dest_dir="$ORGANIZE_ROOT/$file_type"
    local filename=$(basename "$source_file")

    # Create destination directory
    mkdir -p "$dest_dir"

    # Handle duplicate filenames
    local dest_file="$dest_dir/$filename"
    local counter=1

    while [ -f "$dest_file" ]; do
        local name="${filename%.*}"
        local ext="${filename##*.}"
        dest_file="$dest_dir/${name}_${counter}.${ext}"
        ((counter++))
    done

    # Move file
    if mv "$source_file" "$dest_file"; then
        log "Moved: $filename -> $file_type/"
    else
        log "Error: Failed to move $filename"
    fi
}

# Main processing
organize_files() {
    log "File organization started: $SOURCE_DIR"

    if [ ! -d "$SOURCE_DIR" ]; then
        log "Error: Source directory does not exist: $SOURCE_DIR"
        exit 1
    fi

    local file_count=0

    # Process files
    while IFS= read -r -d '' file; do
        if [ -f "$file" ]; then
            local file_type=$(get_file_type "$(basename "$file")")
            move_file "$file" "$file_type"
            ((file_count++))
        fi
    done < <(find "$SOURCE_DIR" -maxdepth 1 -type f -print0)

    log "File organization complete: Processed $file_count files"
}

# Show statistics
show_statistics() {
    log "=== Organization Results Statistics ==="

    for dir in "$ORGANIZE_ROOT"/*; do
        if [ -d "$dir" ]; then
            local count=$(find "$dir" -type f | wc -l)
            local dirname=$(basename "$dir")
            log "$dirname: $count files"
        fi
    done
}

# Main execution
organize_files
show_statistics

⚠️ Common Practical Mistakes and Pitfalls

Advanced problems commonly encountered in practical script creation, with professional solutions explained.

🚫 Mistake 1: Incorrect Array Usage

❌ Problematic Example

# Treating array as string
files=("file1.txt" "file 2.txt" "file3.txt")
for file in $files; do    # Split by spaces
    echo $file
done

# Index confusion
echo ${files[1, 2]}      # Syntax error

Array not expanded properly, issues with filenames containing spaces.

βœ… Correct Usage

# Proper array expansion
files=("file1.txt" "file 2.txt" "file3.txt")
for file in "${files[@]}"; do    # Expand properly as array
    echo "$file"
done

# Multiple index specification
echo "${files[1]}" "${files[2]}"  # Specify individually

Use "${array[@]}" to safely expand entire array.

🚫 Mistake 2: Insufficient Understanding of Function Scope

❌ Unexpected Behavior

#!/bin/bash
counter=0

increment() {
    counter=$((counter + 1))    # Modifying global variable
    local result=$counter       # Local variable
    echo $result
}

increment
echo "Global counter: $counter"  # Unintended modification

Global variable gets modified unintentionally.

βœ… Proper Scope Management

#!/bin/bash
global_counter=0

increment() {
    local local_counter=$1      # Accept argument
    local_counter=$((local_counter + 1))
    echo $local_counter         # Return result (echo)
}

# Use function return value
result=$(increment $global_counter)
global_counter=$result          # Explicitly update

Use local appropriately and control function's scope of influence.

🚫 Mistake 3: Pipeline Variable Modification Trap

❌ Unexpected Results

#!/bin/bash
count=0

# Executed in pipeline subshell
cat file.txt | while read line; do
    count=$((count + 1))
done

echo "Lines: $count"    # Remains 0 (not changed)

Pipeline right side executes in subshell, variable changes not reflected in parent.

βœ… Correct Methods

#!/bin/bash
count=0

# Use redirection
while read line; do
    count=$((count + 1))
done < file.txt

echo "Lines: $count"    # Correctly counted

# Or use command substitution
count=$(wc -l < file.txt)
echo "Lines: $count"

Avoid the problem with redirection or command substitution.

🚫 Mistake 4: Inadequate Error Handling

❌ Insufficient Error Handling

#!/bin/bash

# File processing (no error checking)
cp source.txt backup.txt
rm source.txt               # Delete even if copy failed

# Ignore external command results
curl -o data.json http://api.example.com/data
process_data data.json      # Continue processing even if download failed

Errors can cascade into serious problems.

βœ… Robust Error Handling

#!/bin/bash
set -euo pipefail

# File processing (proper error checking)
if cp source.txt backup.txt; then
    echo "Backup successful"
    rm source.txt
else
    echo "Error: Backup failed" >&2
    exit 1
fi

# Verify external command results
if curl -f -o data.json http://api.example.com/data; then
    process_data data.json
else
    echo "Error: Failed to retrieve data" >&2
    exit 1
fi

Check errors at each step and handle appropriately.

🚫 Mistake 5: Implementation Without Security Consideration

❌ Security Issues

#!/bin/bash

# Execute user input directly (dangerous)
read -p "Enter command: " user_command
eval $user_command

# Predictable temp filename
temp_file="/tmp/script_data.txt"
echo "sensitive data" > $temp_file

# Password on command line
mysql -u user -p'password123' -e "SELECT * FROM users"

Risks of arbitrary command execution and password leakage.

βœ… Secure Implementation

#!/bin/bash

# Input validation
read -p "Enter filename: " filename
if [[ "$filename" =~ ^[a-zA-Z0-9._-]+$ ]]; then
    echo "Processing: $filename"
else
    echo "Error: Invalid filename" >&2
    exit 1
fi

# Safe temporary file creation
temp_file=$(mktemp)
trap 'rm -f "$temp_file"' EXIT
echo "sensitive data" > "$temp_file"

# Read password from config file
if [ -f ~/.mysql_config ]; then
    source ~/.mysql_config
    mysql -u "$DB_USER" -p"$DB_PASS" -e "SELECT * FROM users"
fi

Ensure security with input validation, safe temp files, and config file usage.

🚫 Mistake 6: Implementation Without Performance Consideration

❌ Inefficient Processing

#!/bin/bash

# Process many files individually (slow)
for file in /path/to/large/directory/*; do
    wc -l "$file"           # Start process for each file
done

# Repeatedly execute same command
for i in {1..1000}; do
    current_time=$(date +%s)    # Execute date command each time
    echo "Processing $i at $current_time"
done

Massive process creation makes processing extremely slow.

βœ… Efficient Implementation

#!/bin/bash

# Optimize with batch processing
find /path/to/large/directory -name "*" -type f -exec wc -l {} +

# Get once and reuse
start_time=$(date +%s)
for i in {1..1000}; do
    current_time=$((start_time + i))
    echo "Processing $i at $current_time"
done

# Speed up with parallel processing
export -f process_file
find /path -name "*.txt" | xargs -P 4 -I {} bash -c 'process_file "$@"' _ {}

Improve performance with batch processing, value reuse, and parallel processing.

🎯 Best Practices for Advanced Users

πŸ“‹ Standardized Logging

# Unified log function
log() {
    local level=$1; shift
    local message="$*"
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $message" >&2
}

log "INFO" "Processing started"
log "ERROR" "File not found: $filename"
log "DEBUG" "Variable value: var=$var"

πŸ”„ Separate Configuration Management

# config.sh
DB_HOST="localhost"
DB_PORT="3306"
MAX_RETRIES=3
LOG_LEVEL="INFO"

# main.sh
#!/bin/bash
CONFIG_FILE="${1:-./config.sh}"
if [ -f "$CONFIG_FILE" ]; then
    source "$CONFIG_FILE"
else
    echo "Config file not found: $CONFIG_FILE" >&2
    exit 1
fi

πŸ§ͺ Testable Design

# Testable function design
process_data() {
    local input_file="$1"
    local output_file="$2"

    # Input validation
    [ -f "$input_file" ] || return 1

    # Processing
    awk '{print NR ": " $0}' "$input_file" > "$output_file"

    # Result validation
    [ -f "$output_file" ] && [ -s "$output_file" ]
}

# Test script
test_process_data() {
    local test_input=$(mktemp)
    local test_output=$(mktemp)

    echo -e "line1\nline2" > "$test_input"

    if process_data "$test_input" "$test_output"; then
        echo "βœ… Test passed"
    else
        echo "❌ Test failed"
    fi

    rm -f "$test_input" "$test_output"
}

Shell Scripting Best Practices

πŸ“ Coding Standards

  • Use meaningful variable names
  • Functions focus on single responsibility
  • Clarify intent with comments
  • Consistent indentation (2 spaces recommended)
  • Break long lines appropriately

πŸ”’ Security

  • Always validate input values
  • Create temp files in safe locations
  • Set permissions to minimum required
  • Don't hardcode sensitive information
  • Validate external command results

πŸ› Debugging

  • Enable debug mode with set -x
  • Implement appropriate logging
  • Record state on errors
  • Develop and test incrementally
  • Use static analysis tools like ShellCheck

Summary

Mastering practical shell scripting techniques enables efficient and reliable automation.

Key Points

  • Functions improve code reusability and maintainability
  • Arrays streamline complex data processing
  • Proper error handling creates robust scripts
  • Apply practical examples to real work

Next Steps

πŸ“š Further Learning

  • Advanced Shell Features - Process substitution, coprocesses
  • Language Integration - Combining with Python, awk, sed
  • CI/CD Automation - Using with GitHub Actions, Jenkins
  • System Administration - Integration with cron, systemd

πŸ“‹ Complete Shell Scripting Series

  1. Basics - Variables, conditionals, loops, functions
  2. Practical (This Article) - Real-world scenarios, error handling, optimization
πŸ“’ About Affiliate Links

This site is a participant in the Amazon Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.co.jp. Product prices are not affected.