strace Basics - Tracing System Calls for Troubleshooting

strace Basics - Tracing System Calls for Troubleshooting
Reading time: About 12 min Difficulty: Intermediate

What is strace?

strace is a debugging tool that records and displays the system calls (kernel requests) and signals issued by a process in real time. It exposes raw interactions such as opening files, establishing network connections, and forking child processes — all of which are visible even without source code.

When to use strace

  • No useful log output, or error messages that reveal nothing
  • Unsure which config file the application is actually reading
  • "Permission denied" occurs but you don't know which path is being rejected
  • Network connections fail for unclear reasons

Why does strace help with troubleshooting?

Every application, no matter how complex, accesses hardware and OS resources through kernel system calls. strace monitors that exact interface — so even without source code, you can see precisely what a process tried to do and why it failed.

Kernel-level errors such as errno: ENOENT (file not found), errno: EACCES (permission denied), and errno: ECONNREFUSED (connection refused) appear directly in strace output. Even when an app only logs "An error occurred," strace leads you to the root cause.

How do you install strace?

Most distributions include strace by default. Install it with your package manager if it is missing.

# Ubuntu / Debian
sudo apt install strace

# RHEL / CentOS / Fedora
sudo dnf install strace

Verify the installation:

strace --version
strace -- version 6.x
...

Basic Usage

Trace a command directly

strace ls /tmp

A stream of output appears. The last few lines show the exit status.

execve("/bin/ls", ["ls", "/tmp"], 0x7fff... /* 20 vars */) = 0
brk(NULL)                               = 0x55d3e...
...
openat(AT_FDCWD, "/tmp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
...
+++ exited with 0 +++

Attach to a running process

sudo strace -p PID

Find the PID with ps aux or pgrep:

# Find PID by process name
pgrep nginx

# Attach
sudo strace -p 12345

Attaching with -p usually requires root privileges (not needed for your own processes). Attaching to a production service can affect its performance.

Trace child processes too

For multi-process services, -f is essential:

strace -f -p PID

How do you filter to specific system calls?

The default output is overwhelming. Use -e trace= to narrow the focus:

strace -e trace=openat ls /tmp

Common filters:

Filter Traces
openat File opens
read,write Reads and writes
connect Network connections
execve Process launches
file All file-related calls
network All network-related calls 
# Trace only file access
strace -e trace=file command

# Trace only network connections
strace -e trace=network command

How do you read strace output?

The basic format is:

syscall_name(args...) = return_value

Success:

openat(AT_FDCWD, "/etc/hosts", O_RDONLY) = 3

Failure — ENOENT (file not found):

openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file or directory)

Failure — EACCES (permission denied):

openat(AT_FDCWD, "/root/secret", O_RDONLY) = -1 EACCES (Permission denied)

Reading the return value

  • = 0 or higher: success (file descriptor number, bytes processed, etc.)
  • = -1: failure. The errno token that follows names the specific reason.

Common Troubleshooting Patterns

Pattern 1: App not reading its config file

strace -e trace=openat myapp 2>&1 | grep "ENOENT"
openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/user/.myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)

You can see exactly which paths the application tries.

Pattern 2: Pinpointing the cause of Permission denied

strace -e trace=openat myapp 2>&1 | grep "EACCES"
openat(AT_FDCWD, "/var/run/myapp.pid", O_WRONLY|O_CREAT) = -1 EACCES (Permission denied)

The exact path being rejected becomes visible.

Pattern 3: Diagnosing a connection failure

strace -e trace=network myapp 2>&1 | grep -E "connect|ECONNREFUSED"
connect(3, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)

Confirm whether the address and port match your expectations.

Pattern 4: Save output for later analysis

strace -o /tmp/trace.log -f myapp
grep ENOENT /tmp/trace.log

-o redirects output to a file, keeping stderr clean and making it easy to grep or less through later.

Practical Options Reference

Option Effect
-p PID Attach to a running process
-f Trace child processes created by fork/clone
-e trace= Filter to specific system calls
-o file Write output to a file instead of stderr
-t Prefix each line with wall-clock time
-tt Microsecond-precision timestamps
-T Show time spent in each system call
-c Print a summary of call counts, times, and errors
-s 256 Set maximum string display length (default: 32)
-q Suppress "Attaching..." messages

Checking statistics with -c

strace -c ls /tmp
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 35.14    0.000147          14        10           mmap
 18.42    0.000077          11         7           openat
 ...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000418                    48         2 total

Get a quick picture of which calls are consuming the most time.

Next Reading