Docker Exit Code 137 (OOMKilled): Causes and Real Fixes

The container was running fine, and then it was not. The logs cut off mid-sentence. docker ps -a shows exit code 137. No stack trace, no graceful shutdown, nothing in the application logs explaining anything.

That abruptness is the clue. Exit code 137 means the process did not decide to quit. Something killed it from outside with SIGKILL, the one signal a process cannot catch or ignore. Nine times in ten that something is the Linux OOM killer. Not always, though, and raising the memory limit without checking which case you are in is how people end up paying for 8GB containers that still die.

## Where 137 comes from

A process terminated by a signal exits with 128 + signal_number. SIGKILL is 9, so 128 + 9 = 137. There is no Docker-specific magic in it.

The decoding works for the whole family. 143 is 128 + 15, SIGTERM, a graceful shutdown request from docker stop or a rollout. 139 is 128 + 11, a segfault. So read 137 as “a SIGKILL landed here,” and then ask who sent it.

Only three senders are worth considering: the kernel’s OOM killer because the container hit its own limit, the same killer because the whole host ran dry, or a human or orchestrator that ran docker kill, blew a stop timeout, or had a liveness probe give up.

Step one: confirm it was actually an OOM kill

Do not guess. Docker records whether the OOM killer fired and it takes one command:

docker inspect <container> --format '{{.State.ExitCode}} {{.State.OOMKilled}}'

true and you have your answer: the container exceeded its limit and the kernel stepped in. false alongside a 137 points away from a container-limit OOM and toward a host-level kill or an external docker kill, and the fix is somewhere else entirely.

For something still alive and creeping toward the ceiling, watch the MEM USAGE / LIMIT column:

docker stats --no-stream

And the kernel keeps its own receipt:

dmesg -T | grep -i -E 'oom|killed process'

A line mentioning “Memory cgroup out of memory” means a cgroup-level kill, so the container hit its limit. A plain “Out of memory: Killed process” with no cgroup reference means the host ran dry. That distinction is the fork in the road for everything below.

Cause one: the limit is too low for honest work

The boring, common case. The app genuinely needs more memory than you gave it and hits the ceiling under normal load. Not a leak, just a mismatch.

docker run --memory=1g --memory-swap=1g myimage

Setting --memory-swap equal to --memory disables swap for the container, which is usually what you want. Swapping a containerized app to disk converts an OOM into a slow mysterious latency problem, which is arguably worse than the crash.

Picking the number: run it under realistic load, watch docker stats for steady-state and peak, then add 25 to 50 percent headroom. Do not eyeball it from a ten-second idle reading, because startup or one heavy request can be double the resting value.

One trap worth calling out. Runtimes do not always see your container limit. The JVM has read cgroup limits by default for years. Node’s default heap sizing has historically been derived from the machine’s memory rather than the container’s, which is why pinning it is still standard advice:

node --max-old-space-size=400 server.js

Keep that comfortably under the container limit, because the heap is not the only thing using memory in the process.

Cause two: an actual leak

Here is how to tell it apart from cause one. A too-small limit kills fast and consistently, often within seconds or minutes of the same workload. A leak kills slowly: the container runs for hours, memory climbs in a sawtooth that never fully comes back down, and eventually it crosses the line.

If you find yourself raising the limit, buying a few more hours of uptime, then raising it again, that is not a sizing problem. A bigger limit just buys a longer fuse.

Profile instead. Node takes heap snapshots a few minutes apart under load and diffs them. Python has tracemalloc and memray. The JVM dumps a heap that Eclipse MAT will happily explain. The tool matters less than the discipline of finding what holds references it should not, and the usual suspects are unbounded caches, listeners that never get removed, and connection pools without a cap.

Still set a limit as a seatbelt so a leak degrades gracefully instead of taking the host with it. A seatbelt is not a fix.

Cause three: the host ran out

Back to the case where OOMKilled was false or dmesg showed a non-cgroup kill. The container did not exceed its own limit. The machine ran out of memory and the kernel picked a victim, and yours may simply have had the highest OOM score.

This shows up constantly on CI runners and small VMs where several containers share a host with no per-container limits at all. The fix is not on the dying container. Set limits on every container so one greedy process cannot starve the rest, and give the box enough RAM for the real workload.

free -h
docker stats --no-stream

There is a build-time variant that trips people up. If docker build dies with 137, often during a bundler or type-check step, that is usually the build blowing past Docker Desktop’s VM allocation rather than any runtime limit. Raise the memory in Docker Desktop’s resource settings, or in CI give the runner more RAM and cap the build tool’s own heap.

Kubernetes: same kill, more bookkeeping

The kill is identical at the kernel level. The orchestration around it adds nuance, because two genuinely different events both surface as OOMKilled.

kubectl describe pod <pod> | grep -A5 "Last State"
kubectl get events --field-selector reason=OOMKilling

A Last State showing Reason: OOMKilled with Exit Code: 137 is the container-level kill, triggered by resources.limits.memory regardless of what else is happening in the cluster.

The biggest source of confusion is requests versus limits. The request is what the scheduler uses to place the pod, a reservation and a guaranteed floor. The limit is the hard ceiling that triggers the kill. Set the limit too low relative to real usage and you get OOMKills on a node with plenty of free RAM, which is why people stare at a half-empty node wondering what is wrong. The node’s spare memory is irrelevant to a pod hitting its own limit.

The second, separate event is node pressure. When the node runs low the kubelet does not wait for the kernel, it evicts pods proactively and chooses victims by QoS class. BestEffort pods with no requests or limits go first. Burstable next, worst offenders above their request first. Guaranteed pods, where requests equal limits, go last.

So setting requests equal to limits is not just tidy, it materially lowers the odds of being chosen when the node is squeezed. The flip side is that a BestEffort pod with no limits is both first in line for eviction and free to grow until it causes the mess. Setting limits on everything remains the cheapest reliability win in most clusters.

One behavior change worth knowing on cgroup v2 clusters: the kubelet opts into memory.oom.group, so when the OOM killer fires inside a container it kills every process in that cgroup together rather than the single fattest one. That is mostly good, because it stops you ending up with a half-dead container whose main process survived while a worker got reaped. It did surprise people when it landed, and there is now a singleProcessOOMKill kubelet option to restore the old per-process behavior if you need it.

The checklist

Set an explicit memory limit on every container and every pod. Unlimited containers are how one process takes down a host.

In Kubernetes set requests and limits both, and make them equal for anything you care about.

Size limits from observed peak plus headroom, not from a guess or a copied YAML snippet.

Tell the runtime about the limit where it does not work it out itself.

Alert before the kill. A pod sitting at 95% of its limit for an hour is something you can act on. A 3am OOMKill is not.

And when 137 does hit, check OOMKilled and dmesg before touching the memory setting. That one command tells you whether you are sizing, hunting a leak, or rescuing a starved host, and those three fixes live in completely different places.