Fixing "Cannot allocate memory" (ENOMEM): Overcommit and Swap

2026-06-06 Reading time: About 12 min Difficulty: Intermediate

What Is "Cannot allocate memory"?

Conclusion: A memory-allocation system call (malloc/brk/mmap/fork) was refused by the kernel and returned ENOMEM. The process is not killed — only the allocation fails. It is different from the OOM killer, and it can appear even when free shows headroom. The cause is not always low RAM; it is often a "limit" such as overcommit accounting, ulimit -v, or max_map_count.

Cannot allocate memory is the standard message for errno ENOMEM. It shows up in app logs, dmesg, or strace:

$ ./myapp
fork: Cannot allocate memory
$ python3 -c 'x = bytearray(8*1024**3)'
MemoryError
$ strace -f ./myapp 2>&1 | grep ENOMEM
mmap(NULL, 1073741824, ...) = -1 ENOMEM (Cannot allocate memory)

The key point is that this behaves differently from a forced kill by the OOM killer. The OOM killer kills a process after memory is exhausted and leaves a log. ENOMEM simply refuses the request at the moment of allocation, and the process stays alive and receives an error. That is why you can see Cannot allocate memory with no Out of memory: Killed process in dmesg.

Why Does It Appear When RAM Is Free? (How Overcommit Works)

Conclusion: Linux accounts for how much memory it has "committed" (promised) on allocation. Under vm.overcommit_memory=2 (strict mode), the moment the total commit exceeds CommitLimit, it returns ENOMEM. This "total promised amount" (Committed_AS) is separate from free physical memory, so allocation can be refused even with RAM to spare.

When a process allocates memory, real pages are mapped only on first write (demand paging). What the kernel tracks at allocation time is the "amount promised for future use" — the commit. vm.overcommit_memory decides how far that commit is allowed, in three modes:

The limit in strict mode (2) is visible in /proc/meminfo:

$ grep -i commit /proc/meminfo

CommitLimit:     6029308 kB
Committed_AS:    5980124 kB

CommitLimit is the "total you may promise," and Committed_AS is "what is currently promised." As the latter approaches the former, new allocations are rejected with ENOMEM. CommitLimit is computed as:

CommitLimit = total swap + physical RAM × (vm.overcommit_ratio / 100)

The default vm.overcommit_ratio is 50. So in mode 2, by default you can only promise "swap + half of physical RAM." Even with free RAM, once this accounting limit is hit the allocation fails — the classic "free but still fails" pattern.

What Do You Check First? (Where to Start)

Conclusion: Check, in order: (1) which operation failed (fork vs malloc/mmap), (2) whether dmesg shows OOM, (3) real headroom in free -h, (4) accounting headroom in grep -i commit /proc/meminfo, and (5) the process limit in ulimit -v. These five points almost fully determine whether real exhaustion, overcommit accounting, or a process limit is at play.

The first split is "is memory truly gone, or was the request refused by a limit?"

# 1. Any sign the OOM killer ran (if so, it is the real-exhaustion side)
$ dmesg -T | grep -i -E 'out of memory|killed process'

# 2. Actual physical memory / swap headroom
$ free -h

# 3. Overcommit accounting headroom (important in mode 2)
$ grep -i -E 'commitlimit|committed_as' /proc/meminfo

# 4. Virtual memory limit of this shell / process
$ ulimit -v

A quick decision table:

Is the Overcommit Setting the Cause? (vm.overcommit_memory)

Conclusion: If the mode is 2 and Committed_AS is pinned to CommitLimit, overcommit accounting is the culprit. Raising vm.overcommit_ratio or adding swap widens CommitLimit. But the looser the accounting, the higher the real OOM risk, so the root fix is reviewing memory usage.

Check the current mode and ratio.

$ cat /proc/sys/vm/overcommit_memory
$ cat /proc/sys/vm/overcommit_ratio

2
50

There are two ways to widen CommitLimit: raise the ratio, or add swap.

# Raise the ratio to 80% (allow promising RAM × 80% + swap)
$ sudo sysctl -w vm.overcommit_ratio=80
$ grep -i commitlimit /proc/meminfo   # confirm it applied

Make it permanent.

$ echo 'vm.overcommit_ratio = 80' | sudo tee /etc/sysctl.d/99-overcommit.conf
$ sudo sysctl --system

Is It a Process or User Limit? (ulimit -v / cgroup)

Conclusion: If ulimit -v (RLIMIT_AS) is not unlimited, that process's virtual address space is capped and returns ENOMEM. Under systemd, MemoryMax / cgroup limits cut off allocation the same way. The system as a whole may have plenty, but the per-process limit is what fails.

Check both the logged-in shell and the running process.

# Virtual memory limit of the current shell (KB; unlimited means no cap)
$ ulimit -v

# Check a running process's limit directly (by PID)
$ cat /proc/<pid>/limits | grep -i 'address space'

Max address space   2147483648  2147483648  bytes

If ulimit -v is smaller than what the app needs, that is the direct cause. If it runs as a systemd service, check the unit's limits too.

$ systemctl show -p MemoryMax -p LimitAS myapp.service

To give the service more memory, configure a drop-in.

$ sudo systemctl edit myapp.service

[Service]
MemoryMax=4G
LimitAS=infinity

$ sudo systemctl daemon-reload
$ sudo systemctl restart myapp.service

What If mmap Returns ENOMEM? (vm.max_map_count)

Conclusion: If memory has headroom but only mmap fails with Cannot allocate memory, you have likely hit vm.max_map_count (default 65530), the limit on how many memory-map areas one process can hold. This happens with Elasticsearch, many threads, or loading many shared libraries. Raise the limit to fix it.

Confirm mmap is the cause with strace.

$ strace -f -e trace=mmap ./myapp 2>&1 | grep ENOMEM

mmap(NULL, 262144, PROT_READ|PROT_WRITE, ...) = -1 ENOMEM (Cannot allocate memory)

Compare the process's current map count with the limit.

# Current number of map areas
$ wc -l < /proc/<pid>/maps
# System-wide limit
$ sysctl vm.max_map_count

65530
65530

If the count is pinned to the limit, raise it.

$ sudo sysctl -w vm.max_map_count=262144
$ echo 'vm.max_map_count = 262144' | sudo tee /etc/sysctl.d/99-max-map-count.conf
$ sudo sysctl --system

What If fork Is What Fails?

Conclusion: fork: Cannot allocate memory happens when the child needs a commit reservation as large as the parent, and overcommit accounting (mode 2) or real shortage cannot reserve it. Forking from a large process is the trigger. Switch to posix_spawn / vfork, or revisit overcommit and swap.

fork shares memory copy-on-write, but for accounting it tries to reserve "a commit for the parent's writable pages" on the child's behalf. If the parent holds several GB, commit can exceed CommitLimit at that instant and return ENOMEM.

$ ./big_parent
fork: Cannot allocate memory

There are three directions to fix it:

• Design: Do not fork+exec directly from a huge process; spawn children via posix_spawn or a small helper process
• Accounting: If overcommit is in mode 2, raise the ratio / add swap to widen CommitLimit
• Relax: vm.overcommit_memory=1 (always allow) removes the reservation check (a trade-off against OOM risk)

How Do You Fix It for Good? (Checklist)

Conclusion: Cannot allocate memory splits into two families: "real shortage" and "refused by a limit." Separate them by whether an OOM log exists, then work through overcommit accounting (CommitLimit/Committed_AS) → process limits (ulimit -v/cgroup) → max_map_count. Relaxing accounting or limits is symptomatic; the root is the balance with memory usage.

• [ ] Checked dmesg for OOM killer logs (if present, the real-exhaustion family)?
• [ ] Checked real headroom in free -h available?
• [ ] Checked the current mode with cat /proc/sys/vm/overcommit_memory?
• [ ] In mode 2, checked whether Committed_AS is near CommitLimit?
• [ ] Checked the address space limit in ulimit -v / /proc/<pid>/limits?
• [ ] Checked MemoryMax / LimitAS in systemd / cgroup / container?
• [ ] For mmap failures, compared vm.max_map_count with the map count?
• [ ] For fork failures, distinguished ENOMEM from EAGAIN?
• [ ] Before relaxing (ratio / swap / limits), investigated the real cause of high memory use?