Shell Scripting Practical - Functions, Arrays, and File Operation Techniques

Shell Scripting Practical - Functions, Arrays, and File Operation Techniques
Reading time: About 20 min Difficulty: Intermediate+

What you'll be able to do

  • Split work into functions with local variables and arrays you can reuse
  • Write scripts that stop on failure using `set -euo pipefail` and `trap`
  • Run backup, log-monitoring, and file-organizing scripts without destroying data

Prerequisites (read these first)

Once you've mastered shell scripting basics, it's time to build practical advanced skills. This guide covers functions, arrays, file operations, and error handling. It closes with three scripts you can put straight to work.

Terms used in this article (defined up front)

  • Function: a named block of work, so you do not write the same steps twice
  • Local variable: a variable that exists only inside a function. Declare it with local. Without local it becomes a global variable shared by the whole script
  • Array: several values held in one variable, retrieved by index as in ${arr[0]}
  • Associative array: an array keyed by strings instead of numbers. Also called a hash or dictionary. Available from Bash 4.0
  • Exit code: the number a command returns when it finishes. 0 means success, anything else means failure. Also called exit status
  • Strict mode: shorthand for set -euo pipefail, the settings that stop a script instead of letting a failure slip past
  • trap: the mechanism for registering work to run when something happens, such as an error or the script exiting
  • Subshell: a separate shell process branched off the parent. Variables changed inside it do not carry back out

What You'll Learn

  • How to define functions with local variables, return values, and error handling
  • How to create and manipulate indexed and associative arrays
  • Techniques for reading files line-by-line, writing output, and processing CSV data
  • How set -euo pipefail and trap build robust error handling into scripts
  • Three real-world automation scripts: system backup, log monitoring, and file organizer

Functions

Conclusion: In functions: local for scope, echo to return values, return 1 to signal errors.

Defining and Calling Functions

Basic Function

$1 is the first argument passed to the function; $2 is the second.

#!/bin/bash

greet() {
    echo "Hello, $1!"
}

greet "Alice"
greet "Bob"
Hello, Alice!
Hello, Bob!

Function with Return Value

#!/bin/bash

square() {
    local num=$1
    local result=$((num * num))
    echo $result
}

number=5
result=$(square $number)
echo "Square of $number is $result"

Function with Multiple Arguments

#!/bin/bash

file_info() {
    local filepath=$1

    if [ -f "$filepath" ]; then
        echo "File: $filepath"
        echo "Size: $(wc -c < "$filepath") bytes"
        echo "Lines: $(wc -l < "$filepath") lines"
        echo "Last modified: $(stat -c %y "$filepath")"
    else
        echo "File $filepath does not exist"
        return 1
    fi
}

file_info "/etc/passwd"
file_info "nonexistent.txt"

Advanced Functions

Local vs Global Variables

#!/bin/bash

global_var="global"

demo_scope() {
    local local_var="local"
    global_var="modified global"

    echo "Inside function: local_var = $local_var"
    echo "Inside function: global_var = $global_var"
}

echo "Before call: global_var = $global_var"
demo_scope
echo "After call: global_var = $global_var"
echo "Outside function: local_var = $local_var"  # empty

Recursive Function

#!/bin/bash

factorial() {
    local n=$1

    if [ $n -le 1 ]; then
        echo 1
    else
        local prev=$(factorial $((n - 1)))
        echo $((n * prev))
    fi
}

for i in {1..5}; do
    result=$(factorial $i)
    echo "$i! = $result"
done

Function with Error Handling

#!/bin/bash

safe_mkdir() {
    local dir_path=$1

    if [ -z "$dir_path" ]; then
        echo "Error: no directory path specified" >&2
        return 1
    fi

    if [ -d "$dir_path" ]; then
        echo "Directory $dir_path already exists"
        return 0
    fi

    if mkdir -p "$dir_path" 2>/dev/null; then
        echo "Created directory $dir_path"
        return 0
    else
        echo "Error: failed to create directory $dir_path" >&2
        return 1
    fi
}

safe_mkdir "/tmp/test_dir"
safe_mkdir "/root/forbidden"  # permission error example

Arrays

Conclusion: "${arr[@]}" for safe expansion; declare -A for associative arrays (Bash 4.0+).

Basic Array Operations

Creating Arrays and Accessing Elements

#!/bin/bash

fruits=("apple" "banana" "orange" "grape")
numbers=(1 2 3 4 5)

echo "First fruit: ${fruits[0]}"
echo "Third number: ${numbers[2]}"
echo "All fruits: ${fruits[@]}"
echo "Number of fruits: ${#fruits[@]}"

Adding and Removing Elements

#!/bin/bash

colors=("red" "green" "blue")
echo "Initial array: ${colors[@]}"

colors+=("yellow")
echo "After add: ${colors[@]}"

unset 'colors[1]'  # remove "green"
echo "After delete: ${colors[@]}"

echo "Indices: ${!colors[@]}"
Initial array: red green blue
After add: red green blue yellow
After delete: red blue yellow
Indices: 0 2 3

After unset, the indices are sparse — 1 is gone. ${#colors[@]} is 3, but ${colors[1]} is empty. Looping by number skips the gap, so iterate with "${colors[@]}" or "${!colors[@]}".

Associative Arrays (Bash 4.0+)

#!/bin/bash

declare -A person

person["name"]="John Smith"
person["age"]="30"
person["city"]="New York"

echo "Name: ${person["name"]}"
echo "Age: ${person["age"]}"
echo "All keys: ${!person[@]}"
Name: John Smith
Age: 30
All keys: city age name

The key order of an associative array is not guaranteed. It has nothing to do with the order you defined them in. When order matters, keep a separate array of keys or pipe through sort.

Practical Array Usage

#!/bin/bash

log_patterns=("/var/log/*.log" "/tmp/*.log" "$HOME/*.log")

for pattern in "${log_patterns[@]}"; do
    echo "Pattern: $pattern"
    files=($pattern)

    if [ ${#files[@]} -gt 0 ] && [ -f "${files[0]}" ]; then
        for file in "${files[@]}"; do
            if [ -f "$file" ]; then
                size=$(wc -c < "$file")
                echo "  - $file ($size bytes)"
            fi
        done
    else
        echo "  - No matching files"
    fi
done

File Operations

Conclusion: IFS= read -r + redirection not pipe — loop variables survive outside the loop.

Reading and Writing Files

Various File Reading Methods

#!/bin/bash

filename="data.txt"

# Method 1: using while read
while IFS= read -r line || [ -n "$line" ]; do
    echo "Line: $line"
done < "$filename"

# Method 2: read into array
mapfile -t lines < "$filename"
for i in "${!lines[@]}"; do
    echo "Line $((i+1)): ${lines[i]}"
done

Writing to Files

#!/bin/bash

output_file="output.txt"

echo "New file content" > "$output_file"
echo "Additional line 1" >> "$output_file"

cat << EOF >> "$output_file"
Multi-line text
Line 2
Line 3
EOF

cat "$output_file"

Processing CSV Files

#!/bin/bash

csv_file="employees.csv"

cat << EOF > "$csv_file"
Name,Age,Department
Alice,30,Engineering
Bob,25,Sales
Carol,35,Management
EOF

tail -n +2 "$csv_file" | while IFS=',' read -r name age dept; do
    echo "Employee: $name (age $age) - $dept"
done
Employee: Alice (age 30) - Engineering
Employee: Bob (age 25) - Sales
Employee: Carol (age 35) - Management

This example does not modify variables on the right-hand side of the pipe, so the subshell problem does not apply. Switch to a redirect if you need to carry a total out of the loop.

Advanced File Operation Example

#!/bin/bash

source_dir="/path/to/source"
backup_dir="/path/to/backup"

backup_files() {
    local src="$1"
    local dest="$2"

    if [ ! -d "$src" ]; then
        echo "Error: source directory does not exist: $src"
        return 1
    fi

    mkdir -p "$dest"
    rsync -av --delete "$src/" "$dest/"
    echo "Backup complete: $src -> $dest"
}

log_file="/tmp/backup.log"
{
    echo "Backup started: $(date)"
    backup_files "$source_dir" "$backup_dir"
    echo "Backup finished: $(date)"
} >> "$log_file" 2>&1

Error Handling

Conclusion: Open production scripts with set -euo pipefail and trap on ERR to fail fast.

Error Handling Best Practices

Strict Error Handling with set Options

set -euo pipefail turns on three settings at once. All three exist to stop a script from carrying on past a failure nobody noticed.

#!/bin/bash

set -euo pipefail
# set -e: exit immediately when a command fails
# set -u: error on undefined variables
# set -o pipefail: error if any command in a pipeline fails

trap 'echo "Error occurred: line $LINENO" >&2' ERR

echo "Normal processing 1"

Manual Error Checking

#!/bin/bash

process_file() {
    local filename="$1"

    if [ ! -f "$filename" ]; then
        echo "Error: file '$filename' not found" >&2
        return 1
    fi

    if [ ! -r "$filename" ]; then
        echo "Error: no read permission for '$filename'" >&2
        return 1
    fi

    local line_count
    if ! line_count=$(wc -l < "$filename" 2>/dev/null); then
        echo "Error: failed to count lines" >&2
        return 1
    fi

    echo "Line count for '$filename': $line_count"
    return 0
}

if process_file "test.txt"; then
    echo "Processing completed successfully"
else
    echo "Processing failed"
    exit 1
fi

Error Handling with Logging

#!/bin/bash

LOG_FILE="/tmp/script.log"

log() {
    local level="$1"
    shift
    local message="$*"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] [$level] $message" | tee -a "$LOG_FILE"
}

handle_error() {
    local exit_code=$?
    local line_no=$1
    log "ERROR" "Script exited with error (exit code: $exit_code, line: $line_no)"
    exit $exit_code
}

trap 'handle_error $LINENO' ERR

log "INFO" "Script started"
log "INFO" "Script finished"

Real-World Script Examples

Conclusion: Backup, log monitor, organizer — three scripts on functions, arrays, and errors.

Example 1: System Backup Script

#!/bin/bash

set -euo pipefail

BACKUP_ROOT="/backup"
LOG_FILE="/var/log/backup.log"
RETENTION_DAYS=7
DATE=$(date +%Y%m%d_%H%M%S)

BACKUP_DIRS=(
    "/etc"
    "/home"
    "/var/www"
    "/usr/local/bin"
)

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

cleanup() {
    log "Removing old backups"
    find "$BACKUP_ROOT" -type f -name "backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete
    log "Old backups removed"
}

main_backup() {
    local backup_file="$BACKUP_ROOT/backup_$DATE.tar.gz"
    log "Starting backup: $backup_file"
    mkdir -p "$BACKUP_ROOT"

    if tar -czf "$backup_file" "${BACKUP_DIRS[@]}" 2>/dev/null; then
        local size=$(du -h "$backup_file" | cut -f1)
        log "Backup successful: $backup_file (size: $size)"
    else
        log "Error: backup creation failed"
        exit 1
    fi
}

if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root"
    exit 1
fi

log "System backup started"
main_backup
cleanup
log "System backup completed"

Example 2: Log Monitoring Script

#!/bin/bash

set -euo pipefail

LOG_FILE="/var/log/syslog"
ALERT_PATTERNS=("ERROR" "CRITICAL" "FAILED")
CHECK_INTERVAL=10
LAST_CHECK_FILE="$HOME/.log_monitor_last_check"  # home dir is safer than predictable /tmp path

send_alert() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] ALERT: $message" >> "/var/log/alerts.log"
    logger "LOG_MONITOR_ALERT: $message"
}

monitor_logs() {
    local last_position=0

    if [ -f "$LAST_CHECK_FILE" ]; then
        last_position=$(cat "$LAST_CHECK_FILE")
    fi

    local current_size=$(wc -c < "$LOG_FILE")

    if [ "$current_size" -gt "$last_position" ]; then
        local new_lines=$(tail -c +$((last_position + 1)) "$LOG_FILE")

        for pattern in "${ALERT_PATTERNS[@]}"; do
            if echo "$new_lines" | grep -q "$pattern"; then
                local matches=$(echo "$new_lines" | grep "$pattern")
                send_alert "Detected pattern '$pattern': $matches"
            fi
        done

        echo "$current_size" > "$LAST_CHECK_FILE"
    fi
}

echo "Starting log monitor: $LOG_FILE"
while true; do
    monitor_logs
    sleep "$CHECK_INTERVAL"
done

Example 3: File Organizer Script

#!/bin/bash

set -euo pipefail

SOURCE_DIR="$HOME/Downloads"
ORGANIZE_ROOT="$HOME/Organized"
LOG_FILE="/tmp/file_organizer.log"

declare -A FILE_TYPES=(
    ["pdf"]="Documents/PDF"
    ["doc,docx"]="Documents/Word"
    ["jpg,jpeg,png,gif"]="Images"
    ["mp4,avi,mov"]="Videos"
    ["zip,rar,7z,tar,gz"]="Archives"
)

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

get_file_type() {
    local filename="$1"
    local extension="${filename##*.}"
    extension=$(echo "$extension" | tr '[:upper:]' '[:lower:]')

    for types in "${!FILE_TYPES[@]}"; do
        if [[ ",$types," == *",$extension,"* ]]; then
            echo "${FILE_TYPES[$types]}"
            return 0
        fi
    done

    echo "Others"
}

organize_files() {
    log "Starting file organization: $SOURCE_DIR"

    if [ ! -d "$SOURCE_DIR" ]; then
        log "Error: source directory does not exist: $SOURCE_DIR"
        exit 1
    fi

    local file_count=0

    while IFS= read -r -d '' file; do
        if [ -f "$file" ]; then
            local file_type=$(get_file_type "$(basename "$file")")
            local dest_dir="$ORGANIZE_ROOT/$file_type"
            mkdir -p "$dest_dir"

            if [ -e "$dest_dir/$(basename "$file")" ]; then
                log "Skipped (name already exists): $(basename "$file")"
                continue
            fi

            mv "$file" "$dest_dir/"
            log "Moved: $(basename "$file") -> $file_type/"
            file_count=$((file_count + 1))
        fi
    done < <(find "$SOURCE_DIR" -maxdepth 1 -type f -print0)

    log "Organization complete: processed $file_count files"
}

organize_files

mv overwrites a same-named file without a word

If a file with the same name already exists at the destination, mv overwrites it silently. The script above checks with [ -e ... ] before moving, skips the file, and logs that it did.

mv -n (--no-clobber) also prevents the overwrite, but be careful putting it in a script: recent GNU coreutils (9.x) return exit code 1 when they skip, so under set -e a single name collision stops the whole run. To keep both copies instead, drop -n and use mv --backup=numbered — --backup and -n are mutually exclusive.

On the first run, replace mv with echo mv and read where each file would go. Once files have been reorganized, putting them back where they were is hard.

Common Mistakes and Pitfalls

Conclusion: Five pitfalls — expansion, scope, subshell, errors, security — each preventable.

Mistake 1: Incorrect Array Usage

NG (split by spaces)

files=("file1.txt" "file 2.txt" "file3.txt")
for file in $files; do    # split by spaces
    echo $file
done

OK (correct expansion)

files=("file1.txt" "file 2.txt" "file3.txt")
for file in "${files[@]}"; do    # expand as array
    echo "$file"
done

Use "${array[@]}" to safely expand the full array.

Mistake 2: Misunderstanding Function Scope

NG (global variable unintentionally modified)

counter=0

increment() {
    counter=$((counter + 1))    # modifies global variable
    local result=$counter
    echo $result
}

increment
echo "Global counter: $counter"  # unintended change

OK (proper scope management)

global_counter=0

increment() {
    local local_counter=$1
    local_counter=$((local_counter + 1))
    echo $local_counter
}

result=$(increment $global_counter)
global_counter=$result

Mistake 3: Pipeline Variable Modification Trap

NG (variable changes not reflected)

The right-hand side of a pipe runs in a subshell, a separate process. Variables changed there revert as soon as the pipeline ends.

count=0

cat file.txt | while IFS= read -r line; do
    count=$((count + 1))
done

echo "Lines: $count"    # stays 0 (runs in subshell)

OK (use redirection)

count=0

while IFS= read -r line; do
    count=$((count + 1))
done < file.txt

echo "Lines: $count"    # counted correctly

# or
count=$(wc -l < file.txt)

Mistake 4: Missing Error Handling

NG (errors cascade)

cp source.txt backup.txt
rm source.txt               # deletes even if copy failed

curl -o data.json http://api.example.com/data
process_data data.json      # continues even if download failed

OK (robust error handling)

set -euo pipefail

if cp source.txt backup.txt; then
    echo "Backup successful"
    rm source.txt
else
    echo "Error: backup failed" >&2
    exit 1
fi

Mistake 5: Security-Unaware Implementation

NG (security issues)

eval runs whatever string it is handed as a command. If the user types rm -rf ~, that is exactly what runs. A temp file with a predictable name leaves room for someone else to create it first (a symlink attack). A password written on the command line shows up in ps output and in shell history.

read -p "Enter command: " user_command
eval $user_command                    # arbitrary command execution (dangerous)

temp_file="/tmp/script_data.txt"      # predictable filename
echo "sensitive data" > $temp_file

mysql -u user -p'password123' -e "SELECT * FROM users"  # visible in ps output and shell history

OK (secure implementation)

# validate input
read -p "Enter filename: " filename
if [[ "$filename" =~ ^[a-zA-Z0-9._-]+$ ]]; then
    echo "Processing: $filename"
else
    echo "Error: invalid filename" >&2
    exit 1
fi

# safe temp file creation
temp_file=$(mktemp)
trap 'rm -f "$temp_file"' EXIT

# keep credentials in a config file, off the command line
# ~/.my.cnf (protect with chmod 600)
#   [client]
#   user=appuser
#   password=...
mysql --defaults-extra-file="$HOME/.my.cnf" -e "SELECT * FROM users"

Mistake 6: Performance-Unaware Implementation

NG (inefficient processing)

for file in /path/to/large/directory/*; do
    wc -l "$file"           # spawns a new process per file
done

for i in {1..1000}; do
    current_time=$(date +%s)    # runs date command every iteration
    echo "Processing $i at $current_time"
done

OK (efficient implementation)

# batch processing for efficiency
find /path/to/large/directory -name "*" -type f -exec wc -l {} +

# fetch once, reuse
start_time=$(date +%s)
for i in {1..1000}; do
    current_time=$((start_time + i))
    echo "Processing $i at $current_time"
done

Best Practices

Conclusion: Production Bash: clear names, validated input, mktemp, set -x, and ShellCheck.

Coding Standards

  • Use meaningful variable names
  • Keep functions focused on a single responsibility
  • Comment to clarify intent, not mechanics
  • Use consistent indentation (2 spaces recommended)

Security

  • Always validate user input
  • Use mktemp for temp files. Its names cannot be predicted or claimed in advance
  • Set minimal required permissions
  • Never write passwords or tokens directly in a script

Debugging

  • set -x prints each command as it runs
  • Implement appropriate logging
  • Develop and test incrementally
  • Use static analysis tools such as ShellCheck. They catch accident-prone patterns, like a missing quote, before you run anything

Summary

Mastering practical shell scripting techniques enables efficient and reliable automation. For any script that destroys something, confirm the targets with --dry-run or -print before the real run.

  • Functions improve code reusability and maintainability
  • Arrays streamline complex data processing
  • Proper error handling creates robust scripts
  • Real-world examples provide patterns you can adapt for production

Next Reading

Share this article

Next steps

Next article → Bash Arrays: Indexed and Associative Arrays Practice with quiz → Verify your understanding Practice with lesson → Run commands in the terminal