Shell Scripting Practical - Functions, Arrays, and File Operation Techniques

2025-01-11 Reading time: About 20 min Difficulty: Intermediate~Advanced

Once you've mastered shell scripting basics, it's time to build practical advanced skills. This guide covers functions, arrays, file operations, error handling, and real-world automation examples to help you write professional-grade Bash scripts.

Functions

Defining and Calling Functions

Basic Function

#!/bin/bash

greet() {
    echo "Hello, $1!"
}

greet "Alice"
greet "Bob"

Function with Return Value

#!/bin/bash

square() {
    local num=$1
    local result=$((num * num))
    echo $result
}

number=5
result=$(square $number)
echo "Square of $number is $result"

Function with Multiple Arguments

#!/bin/bash

file_info() {
    local filepath=$1

    if [ -f "$filepath" ]; then
        echo "File: $filepath"
        echo "Size: $(wc -c < "$filepath") bytes"
        echo "Lines: $(wc -l < "$filepath") lines"
        echo "Last modified: $(stat -c %y "$filepath")"
    else
        echo "File $filepath does not exist"
        return 1
    fi
}

file_info "/etc/passwd"
file_info "nonexistent.txt"

Advanced Functions

Local vs Global Variables

#!/bin/bash

global_var="global"

demo_scope() {
    local local_var="local"
    global_var="modified global"

    echo "Inside function: local_var = $local_var"
    echo "Inside function: global_var = $global_var"
}

echo "Before call: global_var = $global_var"
demo_scope
echo "After call: global_var = $global_var"
echo "Outside function: local_var = $local_var"  # empty

Recursive Function

#!/bin/bash

factorial() {
    local n=$1

    if [ $n -le 1 ]; then
        echo 1
    else
        local prev=$(factorial $((n - 1)))
        echo $((n * prev))
    fi
}

for i in {1..5}; do
    result=$(factorial $i)
    echo "$i! = $result"
done

Function with Error Handling

#!/bin/bash

safe_mkdir() {
    local dir_path=$1

    if [ -z "$dir_path" ]; then
        echo "Error: no directory path specified" >&2
        return 1
    fi

    if [ -d "$dir_path" ]; then
        echo "Directory $dir_path already exists"
        return 0
    fi

    if mkdir -p "$dir_path" 2>/dev/null; then
        echo "Created directory $dir_path"
        return 0
    else
        echo "Error: failed to create directory $dir_path" >&2
        return 1
    fi
}

safe_mkdir "/tmp/test_dir"
safe_mkdir "/root/forbidden"  # permission error example

Arrays

Basic Array Operations

Creating Arrays and Accessing Elements

#!/bin/bash

fruits=("apple" "banana" "orange" "grape")
numbers=(1 2 3 4 5)

echo "First fruit: ${fruits[0]}"
echo "Third number: ${numbers[2]}"
echo "All fruits: ${fruits[@]}"
echo "Number of fruits: ${#fruits[@]}"

Adding and Removing Elements

#!/bin/bash

colors=("red" "green" "blue")
echo "Initial array: ${colors[@]}"

colors+=("yellow")
echo "After add: ${colors[@]}"

unset colors[1]  # remove "green"
echo "After delete: ${colors[@]}"

echo "Indices: ${!colors[@]}"

Associative Arrays (Bash 4.0+)

#!/bin/bash

declare -A person

person["name"]="John Smith"
person["age"]="30"
person["city"]="New York"

echo "Name: ${person["name"]}"
echo "Age: ${person["age"]}"
echo "All keys: ${!person[@]}"

Practical Array Usage

#!/bin/bash

log_patterns=("/var/log/*.log" "/tmp/*.log" "$HOME/*.log")

for pattern in "${log_patterns[@]}"; do
    echo "Pattern: $pattern"
    files=($pattern)

    if [ ${#files[@]} -gt 0 ] && [ -f "${files[0]}" ]; then
        for file in "${files[@]}"; do
            if [ -f "$file" ]; then
                size=$(wc -c < "$file")
                echo "  - $file ($size bytes)"
            fi
        done
    else
        echo "  - No matching files"
    fi
done

File Operations

Reading and Writing Files

Various File Reading Methods

#!/bin/bash

filename="data.txt"

# Method 1: using while read
while IFS= read -r line || [ -n "$line" ]; do
    echo "Line: $line"
done < "$filename"

# Method 2: read into array
mapfile -t lines < "$filename"
for i in "${!lines[@]}"; do
    echo "Line $((i+1)): ${lines[i]}"
done

Writing to Files

#!/bin/bash

output_file="output.txt"

echo "New file content" > "$output_file"
echo "Additional line 1" >> "$output_file"

cat << EOF >> "$output_file"
Multi-line text
Line 2
Line 3
EOF

cat "$output_file"

Processing CSV Files

#!/bin/bash

csv_file="employees.csv"

cat << EOF > "$csv_file"
Name,Age,Department
Alice,30,Engineering
Bob,25,Sales
Carol,35,Management
EOF

tail -n +2 "$csv_file" | while IFS=',' read -r name age dept; do
    echo "Employee: $name (age $age) - $dept"
done

Advanced File Operation Example

#!/bin/bash

source_dir="/path/to/source"
backup_dir="/path/to/backup"

backup_files() {
    local src="$1"
    local dest="$2"

    if [ ! -d "$src" ]; then
        echo "Error: source directory does not exist: $src"
        return 1
    fi

    mkdir -p "$dest"
    rsync -av --delete "$src/" "$dest/"
    echo "Backup complete: $src -> $dest"
}

log_file="/tmp/backup.log"
{
    echo "Backup started: $(date)"
    backup_files "$source_dir" "$backup_dir"
    echo "Backup finished: $(date)"
} >> "$log_file" 2>&1

Error Handling

Error Handling Best Practices

Strict Error Handling with set Options

#!/bin/bash

set -euo pipefail
# set -e: exit immediately when a command fails
# set -u: error on undefined variables
# set -o pipefail: error if any command in a pipeline fails

trap 'echo "Error occurred: line $LINENO" >&2' ERR

echo "Normal processing 1"

Manual Error Checking

#!/bin/bash

process_file() {
    local filename="$1"

    if [ ! -f "$filename" ]; then
        echo "Error: file '$filename' not found" >&2
        return 1
    fi

    if [ ! -r "$filename" ]; then
        echo "Error: no read permission for '$filename'" >&2
        return 1
    fi

    local line_count
    if ! line_count=$(wc -l < "$filename" 2>/dev/null); then
        echo "Error: failed to count lines" >&2
        return 1
    fi

    echo "Line count for '$filename': $line_count"
    return 0
}

if process_file "test.txt"; then
    echo "Processing completed successfully"
else
    echo "Processing failed"
    exit 1
fi

Error Handling with Logging

#!/bin/bash

LOG_FILE="/tmp/script.log"

log() {
    local level="$1"
    shift
    local message="$*"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] [$level] $message" | tee -a "$LOG_FILE"
}

handle_error() {
    local exit_code=$?
    local line_no=$1
    log "ERROR" "Script exited with error (exit code: $exit_code, line: $line_no)"
    exit $exit_code
}

trap 'handle_error $LINENO' ERR

log "INFO" "Script started"
log "INFO" "Script finished"

Real-World Script Examples

Example 1: System Backup Script

#!/bin/bash

set -euo pipefail

BACKUP_ROOT="/backup"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=7
DATE=$(date +%Y%m%d_%H%M%S)

BACKUP_DIRS=(
    "/etc"
    "/home"
    "/var/www"
    "/usr/local/bin"
)

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

cleanup() {
    log "Removing old backups"
    find "$BACKUP_ROOT" -type f -name "backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete
    log "Old backups removed"
}

main_backup() {
    local backup_file="$BACKUP_ROOT/backup_$DATE.tar.gz"
    log "Starting backup: $backup_file"
    mkdir -p "$BACKUP_ROOT"

    if tar -czf "$backup_file" "${BACKUP_DIRS[@]}" 2>/dev/null; then
        local size=$(du -h "$backup_file" | cut -f1)
        log "Backup successful: $backup_file (size: $size)"
    else
        log "Error: backup creation failed"
        exit 1
    fi
}

if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root"
    exit 1
fi

log "System backup started"
main_backup
cleanup
log "System backup completed"

Example 2: Log Monitoring Script

#!/bin/bash

set -euo pipefail

LOG_FILE="/var/log/syslog"
ALERT_PATTERNS=("ERROR" "CRITICAL" "FAILED")
CHECK_INTERVAL=10
LAST_CHECK_FILE="$HOME/.log_monitor_last_check"  # home dir is safer than predictable /tmp path

send_alert() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] ALERT: $message" >> "/var/log/alerts.log"
    logger "LOG_MONITOR_ALERT: $message"
}

monitor_logs() {
    local last_position=0

    if [ -f "$LAST_CHECK_FILE" ]; then
        last_position=$(cat "$LAST_CHECK_FILE")
    fi

    local current_size=$(wc -c < "$LOG_FILE")

    if [ "$current_size" -gt "$last_position" ]; then
        local new_lines=$(tail -c +$((last_position + 1)) "$LOG_FILE")

        for pattern in "${ALERT_PATTERNS[@]}"; do
            if echo "$new_lines" | grep -q "$pattern"; then
                local matches=$(echo "$new_lines" | grep "$pattern")
                send_alert "Detected pattern '$pattern': $matches"
            fi
        done

        echo "$current_size" > "$LAST_CHECK_FILE"
    fi
}

echo "Starting log monitor: $LOG_FILE"
while true; do
    monitor_logs
    sleep "$CHECK_INTERVAL"
done

Example 3: File Organizer Script

#!/bin/bash

set -euo pipefail

SOURCE_DIR="$HOME/Downloads"
ORGANIZE_ROOT="$HOME/Organized"
LOG_FILE="/tmp/file_organizer.log"

declare -A FILE_TYPES=(
    ["pdf"]="Documents/PDF"
    ["doc,docx"]="Documents/Word"
    ["jpg,jpeg,png,gif"]="Images"
    ["mp4,avi,mov"]="Videos"
    ["zip,rar,7z,tar,gz"]="Archives"
)

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

get_file_type() {
    local filename="$1"
    local extension="${filename##*.}"
    extension=$(echo "$extension" | tr '[:upper:]' '[:lower:]')

    for types in "${!FILE_TYPES[@]}"; do
        if [[ ",$types," == *",$extension,"* ]]; then
            echo "${FILE_TYPES[$types]}"
            return 0
        fi
    done

    echo "Others"
}

organize_files() {
    log "Starting file organization: $SOURCE_DIR"

    if [ ! -d "$SOURCE_DIR" ]; then
        log "Error: source directory does not exist: $SOURCE_DIR"
        exit 1
    fi

    local file_count=0

    while IFS= read -r -d '' file; do
        if [ -f "$file" ]; then
            local file_type=$(get_file_type "$(basename "$file")")
            local dest_dir="$ORGANIZE_ROOT/$file_type"
            mkdir -p "$dest_dir"
            mv "$file" "$dest_dir/"
            log "Moved: $(basename "$file") -> $file_type/"
            file_count=$((file_count + 1))
        fi
    done < <(find "$SOURCE_DIR" -maxdepth 1 -type f -print0)

    log "Organization complete: processed $file_count files"
}

organize_files

Common Mistakes and Pitfalls

Mistake 1: Incorrect Array Usage

NG (split by spaces)

files=("file1.txt" "file 2.txt" "file3.txt")
for file in $files; do    # split by spaces
    echo $file
done

OK (correct expansion)

files=("file1.txt" "file 2.txt" "file3.txt")
for file in "${files[@]}"; do    # expand as array
    echo "$file"
done

Use "${array[@]}" to safely expand the full array.

Mistake 2: Misunderstanding Function Scope

NG (global variable unintentionally modified)

counter=0

increment() {
    counter=$((counter + 1))    # modifies global variable
    local result=$counter
    echo $result
}

increment
echo "Global counter: $counter"  # unintended change

OK (proper scope management)

global_counter=0

increment() {
    local local_counter=$1
    local_counter=$((local_counter + 1))
    echo $local_counter
}

result=$(increment $global_counter)
global_counter=$result

Mistake 3: Pipeline Variable Modification Trap

NG (variable changes not reflected)

count=0

cat file.txt | while IFS= read -r line; do
    count=$((count + 1))
done

echo "Lines: $count"    # stays 0 (runs in subshell)

OK (use redirection)

count=0

while IFS= read -r line; do
    count=$((count + 1))
done < file.txt

echo "Lines: $count"    # counted correctly

# or
count=$(wc -l < file.txt)

Mistake 4: Missing Error Handling

NG (errors cascade)

cp source.txt backup.txt
rm source.txt               # deletes even if copy failed

curl -o data.json http://api.example.com/data
process_data data.json      # continues even if download failed

OK (robust error handling)

set -euo pipefail

if cp source.txt backup.txt; then
    echo "Backup successful"
    rm source.txt
else
    echo "Error: backup failed" >&2
    exit 1
fi

Mistake 5: Security-Unaware Implementation

NG (security issues)

read -p "Enter command: " user_command
eval $user_command                    # arbitrary command execution (dangerous)

temp_file="/tmp/script_data.txt"      # predictable filename
echo "sensitive data" > $temp_file

mysql -u user -p'password123' -e "SELECT * FROM users"  # password in logs

OK (secure implementation)

# validate input
read -p "Enter filename: " filename
if [[ "$filename" =~ ^[a-zA-Z0-9._-]+$ ]]; then
    echo "Processing: $filename"
else
    echo "Error: invalid filename" >&2
    exit 1
fi

# safe temp file creation
temp_file=$(mktemp)
trap 'rm -f "$temp_file"' EXIT

# read password from config file (must protect with: chmod 600 ~/.mysql_config)
if [ -f ~/.mysql_config ]; then
    source ~/.mysql_config
    mysql -u "$DB_USER" -p"$DB_PASS" -e "SELECT * FROM users"
fi

Mistake 6: Performance-Unaware Implementation

NG (inefficient processing)

for file in /path/to/large/directory/*; do
    wc -l "$file"           # spawns a new process per file
done

for i in {1..1000}; do
    current_time=$(date +%s)    # runs date command every iteration
    echo "Processing $i at $current_time"
done

OK (efficient implementation)

# batch processing for efficiency
find /path/to/large/directory -name "*" -type f -exec wc -l {} +

# fetch once, reuse
start_time=$(date +%s)
for i in {1..1000}; do
    current_time=$((start_time + i))
    echo "Processing $i at $current_time"
done

Best Practices

Coding Standards

Use meaningful variable names
Keep functions focused on a single responsibility
Comment to clarify intent, not mechanics
Use consistent indentation (2 spaces recommended)

Security

Always validate user input
Use mktemp for safe temp file creation
Set minimal required permissions
Never hardcode sensitive information

Debugging

Enable debug mode with set -x
Implement appropriate logging
Develop and test incrementally
Use static analysis tools like ShellCheck

Summary

Mastering practical shell scripting techniques enables efficient and reliable automation.

Functions improve code reusability and maintainability
Arrays streamline complex data processing
Proper error handling creates robust scripts
Real-world examples provide patterns you can adapt for production

Next Reading

Shell Scripting Basics — Variables, conditionals, and loops fundamentals
Process Management (Practical) — Job control, pkill, nice
File Operations (Advanced)