1# Out-of-memory (OOM) system
2
3This file contains information about the systems that watch for and respond to
4out-of-memory (OOM) events.
5
6[TOC]
7
8## Behavior
9
10When the system runs out of memory and the kernel OOM thread is running, you
11should see a series of log messages like:
12
13```
14OOM: 5915.8M free (+0B) / 8072.4M total
15OOM: oom_lowmem(shortfall_bytes=524288) called
16OOM: Process mapped committed bytes:
17OOM:   proc  1043  397M 'bin/devmgr'
18OOM:   proc  2107   88M 'devhost:pci#1:8086:1916'
19OOM:   proc  1297   12M 'virtual-console'
20OOM:   proc  3496   17M 'netstack'
21OOM:   proc  4157  170M 'flutter:userpicker_device_shell'
22OOM:   proc 28708  353M 'flutter:armadillo_user_... (+3)'
23OOM:   proc 31584    9M 'dart:weather_agent'
24OOM:   proc 32093   14M 'dart:mi_dashboard.dartx'
25OOM: Finding a job to kill...
26OOM:   (skip) job  57930 'story-8cf82cb9f742d9ecc77f1d449'
27OOM:   (skip) job  37434 'story-10293ae401bc0358b3ce52d2a'
28OOM:   *KILL* job  29254 'agent'
29OOM:        + proc 32093  run 'dart:mi_dashboard.dartx'
30OOM:        = 1 running procs (1 total), 0 jobs
31OOM:   (next) job  29247 'agent'
32OOM:   (next) job  29240 'agent'
33OOM:   (next) job  29233 'agent'
34```
35
36The first line shows the current state of system memory:
37
38```
39OOM: 45.8M free (-12.4M) / 8072.4M total
40```
41
42The next section prints a list of processes that are consuming large amounts of
43memory, in no particular order:
44
45```
46OOM: Process mapped committed bytes:
47OOM:   proc  1043  397M 'bin/devmgr'
48OOM:   proc  2107   88M 'devhost:pci#1:8086:1916'
49OOM:   proc  1297   12M 'virtual-console'
50OOM:   ...
51             ^koid  ^mem
52```
53
54The next section shows the walk through the ranked job list, printing skipped
55jobs (which don't have killable process descendants), the job that will be
56killed, and the next jobs on the chopping block:
57
58```
59OOM: Finding a job to kill...
60OOM:   (skip) job  57930 'story-8cf82cb9f742d9ecc77f1d449'
61OOM:   (skip) job  37434 'story-10293ae401bc0358b3ce52d2a'
62OOM:   *KILL* job  29254 'agent'
63OOM:        + proc 32093  run 'dart:mi_dashboard.dartx'
64OOM:        = 1 running procs (1 total), 0 jobs
65OOM:   (next) job  29247 'agent'
66OOM:   (next) job  29240 'agent'
67OOM:   (next) job  29233 'agent'
68
69                   ^koid ^name
70```
71
72The `*KILL*` entry will also show all process descendants of the to-be-killed
73job.
74
75## Components
76
77### Kernel OOM thread
78
79A kernel thread that periodically checks the amount of free memory in the
80system, and kills a job if the free amount is too low (below the "redline").
81
82Use `k oom info` to see the state of the OOM thread (on the kernel console):
83
84```
85$ k oom info
86OOM info:
87  running: true
88  printing: false
89  simulating lowmem: false
90  sleep duration: 1000ms
91  redline: 50M (52428800 bytes)
92```
93
94The redline, sleep duration, and auto-start values are controlled by
95`kernel.oom.*` [kernel commandline flags](kernel_cmdline.md).
96
97The thread can be started with `k oom start` and stopped with `k oom stop`.
98
99`k oom print` will toggle a flag that prints the current free and total memory
100every time the thread wakes up.
101
102`k oom lowmem` will trigger a false low-memory event the next time the thread
103wakes up, potentially killing a job.
104
105### OOM-ranker driver
106
107TODO(dbort/maniscalco): Implement and document.
108