1.. SPDX-License-Identifier: GPL-2.0
2
3=================
4Boot-time tracing
5=================
6
7:Author: Masami Hiramatsu <mhiramat@kernel.org>
8
9Overview
10========
11
12Boot-time tracing allows users to trace boot-time process including
13device initialization with full features of ftrace including per-event
14filter and actions, histograms, kprobe-events and synthetic-events,
15and trace instances.
16Since kernel command line is not enough to control these complex features,
17this uses bootconfig file to describe tracing feature programming.
18
19Options in the Boot Config
20==========================
21
22Here is the list of available options list for boot time tracing in
23boot config file [1]_. All options are under "ftrace." or "kernel."
24prefix. See kernel parameters for the options which starts
25with "kernel." prefix [2]_.
26
27.. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>`
28.. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>`
29
30Ftrace Global Options
31---------------------
32
33Ftrace global options have "kernel." prefix in boot config, which means
34these options are passed as a part of kernel legacy command line.
35
36kernel.tp_printk
37   Output trace-event data on printk buffer too.
38
39kernel.dump_on_oops [= MODE]
40   Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer
41   on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.
42
43kernel.traceoff_on_warning
44   Stop tracing if WARN_ON() occurs.
45
46kernel.fgraph_max_depth = MAX_DEPTH
47   Set MAX_DEPTH to maximum depth of fgraph tracer.
48
49kernel.fgraph_filters = FILTER[, FILTER2...]
50   Add fgraph tracing function filters.
51
52kernel.fgraph_notraces = FILTER[, FILTER2...]
53   Add fgraph non-tracing function filters.
54
55
56Ftrace Per-instance Options
57---------------------------
58
59These options can be used for each instance including global ftrace node.
60
61ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]]
62   Enable given ftrace options.
63
64ftrace.[instance.INSTANCE.]tracing_on = 0|1
65   Enable/Disable tracing on this instance when starting boot-time tracing.
66   (you can enable it by the "traceon" event trigger action)
67
68ftrace.[instance.INSTANCE.]trace_clock = CLOCK
69   Set given CLOCK to ftrace's trace_clock.
70
71ftrace.[instance.INSTANCE.]buffer_size = SIZE
72   Configure ftrace buffer size to SIZE. You can use "KB" or "MB"
73   for that SIZE.
74
75ftrace.[instance.INSTANCE.]alloc_snapshot
76   Allocate snapshot buffer.
77
78ftrace.[instance.INSTANCE.]cpumask = CPUMASK
79   Set CPUMASK as trace cpu-mask.
80
81ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]]
82   Enable given events on boot. You can use a wild card in EVENT.
83
84ftrace.[instance.INSTANCE.]tracer = TRACER
85   Set TRACER to current tracer on boot. (e.g. function)
86
87ftrace.[instance.INSTANCE.]ftrace.filters
88   This will take an array of tracing function filter rules.
89
90ftrace.[instance.INSTANCE.]ftrace.notraces
91   This will take an array of NON-tracing function filter rules.
92
93
94Ftrace Per-Event Options
95------------------------
96
97These options are setting per-event options.
98
99ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable
100   Enable GROUP:EVENT tracing.
101
102ftrace.[instance.INSTANCE.]event.GROUP.enable
103   Enable all event tracing within GROUP.
104
105ftrace.[instance.INSTANCE.]event.enable
106   Enable all event tracing.
107
108ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER
109   Set FILTER rule to the GROUP:EVENT.
110
111ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]]
112   Set ACTIONs to the GROUP:EVENT.
113
114ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]]
115   Defines new kprobe event based on PROBEs. It is able to define
116   multiple probes on one event, but those must have same type of
117   arguments. This option is available only for the event which
118   group name is "kprobes".
119
120ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]]
121   Defines new synthetic event with FIELDs. Each field should be
122   "type varname".
123
124Note that kprobe and synthetic event definitions can be written under
125instance node, but those are also visible from other instances. So please
126take care for event name conflict.
127
128Ftrace Histogram Options
129------------------------
130
131Since it is too long to write a histogram action as a string for per-event
132action option, there are tree-style options under per-event 'hist' subkey
133for the histogram actions. For the detail of the each parameter,
134please read the event histogram document (Documentation/trace/histogram.rst)
135
136ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]keys = KEY1[, KEY2[...]]
137  Set histogram key parameters. (Mandatory)
138  The 'N' is a digit string for the multiple histogram. You can omit it
139  if there is one histogram on the event.
140
141ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]values = VAL1[, VAL2[...]]
142  Set histogram value parameters.
143
144ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]sort = SORT1[, SORT2[...]]
145  Set histogram sort parameter options.
146
147ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]size = NR_ENTRIES
148  Set histogram size (number of entries).
149
150ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]name = NAME
151  Set histogram name.
152
153ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]var.VARIABLE = EXPR
154  Define a new VARIABLE by EXPR expression.
155
156ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<pause|continue|clear>
157  Set histogram control parameter. You can set one of them.
158
159ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]event = GROUP.EVENT
160  Set histogram 'onmatch' handler matching event parameter.
161  The 'M' is a digit string for the multiple 'onmatch' handler. You can omit it
162  if there is one 'onmatch' handler on this histogram.
163
164ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]trace = EVENT[, ARG1[...]]
165  Set histogram 'trace' action for 'onmatch'.
166  EVENT must be a synthetic event name, and ARG1... are parameters
167  for that event. Mandatory if 'onmatch.event' option is set.
168
169ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmax.[M.]var = VAR
170  Set histogram 'onmax' handler variable parameter.
171
172ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onchange.[M.]var = VAR
173  Set histogram 'onchange' handler variable parameter.
174
175ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]save = ARG1[, ARG2[...]]
176  Set histogram 'save' action parameters for 'onmax' or 'onchange' handler.
177  This option or below 'snapshot' option is mandatory if 'onmax.var' or
178  'onchange.var' option is set.
179
180ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]snapshot
181  Set histogram 'snapshot' action for 'onmax' or 'onchange' handler.
182  This option or above 'save' option is mandatory if 'onmax.var' or
183  'onchange.var' option is set.
184
185ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.filter = FILTER_EXPR
186  Set histogram filter expression. You don't need 'if' in the FILTER_EXPR.
187
188Note that this 'hist' option can conflict with the per-event 'actions'
189option if the 'actions' option has a histogram action.
190
191
192When to Start
193=============
194
195All boot-time tracing options starting with ``ftrace`` will be enabled at the
196end of core_initcall. This means you can trace the events from postcore_initcall.
197Most of the subsystems and architecture dependent drivers will be initialized
198after that (arch_initcall or subsys_initcall). Thus, you can trace those with
199boot-time tracing.
200If you want to trace events before core_initcall, you can use the options
201starting with ``kernel``. Some of them will be enabled eariler than the initcall
202processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event``
203will start before the initcall.)
204
205
206Examples
207========
208
209For example, to add filter and actions for each event, define kprobe
210events, and synthetic events with histogram, write a boot config like
211below::
212
213  ftrace.event {
214        task.task_newtask {
215                filter = "pid < 128"
216                enable
217        }
218        kprobes.vfs_read {
219                probes = "vfs_read $arg1 $arg2"
220                filter = "common_pid < 200"
221                enable
222        }
223        synthetic.initcall_latency {
224                fields = "unsigned long func", "u64 lat"
225                hist {
226                        keys = func.sym, lat
227                        values = lat
228                        sort = lat
229                }
230        }
231        initcall.initcall_start.hist {
232                keys = func
233                var.ts0 = common_timestamp.usecs
234        }
235        initcall.initcall_finish.hist {
236                keys = func
237                var.lat = common_timestamp.usecs - $ts0
238                onmatch {
239                        event = initcall.initcall_start
240                        trace = initcall_latency, func, $lat
241                }
242        }
243  }
244
245Also, boot-time tracing supports "instance" node, which allows us to run
246several tracers for different purpose at once. For example, one tracer
247is for tracing functions starting with "user\_", and others tracing
248"kernel\_" functions, you can write boot config as below::
249
250  ftrace.instance {
251        foo {
252                tracer = "function"
253                ftrace.filters = "user_*"
254        }
255        bar {
256                tracer = "function"
257                ftrace.filters = "kernel_*"
258        }
259  }
260
261The instance node also accepts event nodes so that each instance
262can customize its event tracing.
263
264With the trigger action and kprobes, you can trace function-graph while
265a function is called. For example, this will trace all function calls in
266the pci_proc_init()::
267
268  ftrace {
269        tracing_on = 0
270        tracer = function_graph
271        event.kprobes {
272                start_event {
273                        probes = "pci_proc_init"
274                        actions = "traceon"
275                }
276                end_event {
277                        probes = "pci_proc_init%return"
278                        actions = "traceoff"
279                }
280        }
281  }
282
283
284This boot-time tracing also supports ftrace kernel parameters via boot
285config.
286For example, following kernel parameters::
287
288 trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"
289
290This can be written in boot config like below::
291
292  kernel {
293        trace_options = sym-addr
294        trace_event = "initcall:*"
295        tp_printk
296        trace_buf_size = 1M
297        ftrace = function
298        ftrace_filter = "vfs*"
299  }
300
301Note that parameters start with "kernel" prefix instead of "ftrace".
302