1.. _tracing:
2
3Tracing
4#######
5
6Overview
7********
8
9The tracing feature provides hooks that permits you to collect data from
10your application and allows :ref:`tools` running on a host to visualize the inner-working of
11the kernel and various subsystems.
12
13Every system has application-specific events to trace out.  Historically,
14that has implied:
15
161. Determining the application-specific payload,
172. Choosing suitable serialization-format,
183. Writing the on-target serialization code,
194. Deciding on and writing the I/O transport mechanics,
205. Writing the PC-side deserializer/parser,
216. Writing custom ad-hoc tools for filtering and presentation.
22
23An application can use one of the existing formats or define a custom format by
24overriding the macros declared in :zephyr_file:`include/zephyr/tracing/tracing.h`.
25
26Different formats, transports and host tools are available and supported in
27Zephyr.
28
29In fact, I/O varies greatly from system to system.  Therefore, it is
30instructive to create a taxonomy for I/O types when we must ensure the
31interface between payload/format (Top Layer) and the transport mechanics
32(bottom Layer) is generic and efficient enough to model these. See the
33*I/O taxonomy* section below.
34
35Named Trace Events
36******************
37
38Although the user can extend any of the supported serialization formats
39to enable additional tracing functions (or provide their own backend), zephyr
40also provides one generic named tracing function for convenience purposes,
41as well as to demonstrate how tracing frameworks could be extended.
42
43Users can generate a custom trace event by calling
44:c:func:`sys_trace_named_event`, which takes an event name as well as two
45arbitrary 4 byte arguments. Tracing backends may truncate the provided event
46name if it is too long for the serialization format they support.
47
48Serialization Formats
49**********************
50
51.. _ctf:
52
53Common Trace Format (CTF) Support
54=================================
55
56Common Trace Format, CTF, is an open format and language to describe trace
57formats. This enables tool reuse, of which line-textual (babeltrace) and
58graphical (TraceCompass) variants already exist.
59
60CTF should look familiar to C programmers but adds stronger typing.
61See `CTF - A Flexible, High-performance Binary Trace Format
62<https://diamon.org/ctf/>`_.
63
64
65CTF allows us to formally describe application specific payload and the
66serialization format, which enables common infrastructure for host tools
67and parsers and tools for filtering and presentation.
68
69
70A Generic Interface
71--------------------
72
73In CTF, an event is serialized to a packet containing one or more fields.
74As seen from *I/O taxonomy* section below, a bottom layer may:
75
76- perform actions at transaction-start (e.g. mutex-lock),
77- process each field in some way (e.g. sync-push emit, concat, enqueue to
78  thread-bound FIFO),
79- perform actions at transaction-stop (e.g. mutex-release, emit of concat
80  buffer).
81
82CTF Top-Layer Example
83----------------------
84
85The CTF_EVENT macro will serialize each argument to a field::
86
87  /* Example for illustration */
88  static inline void ctf_top_foo(uint32_t thread_id, ctf_bounded_string_t name)
89  {
90    CTF_EVENT(
91      CTF_LITERAL(uint8_t, 42),
92      thread_id,
93      name,
94      "hello, I was emitted from function: ",
95      __func__  /* __func__ is standard since C99 */
96    );
97  }
98
99How to serialize and emit fields as well as handling alignment, can be done
100internally and statically at compile-time in the bottom-layer.
101
102
103The CTF top layer is enabled using the configuration option
104:kconfig:option:`CONFIG_TRACING_CTF` and can be used with the different transport
105backends both in synchronous and asynchronous modes.
106
107.. _tools:
108
109Tracing Tools
110*************
111
112Zephyr includes support for several popular tracing tools, presented below in alphabetical order.
113
114Percepio Tracealyzer Support
115============================
116
117Zephyr includes support for `Percepio Tracealyzer`_ that offers trace visualization for
118simplified analysis, report generation and other analysis features. Tracealyzer allows for trace
119streaming over various interfaces and also snapshot tracing, where the events are kept in a RAM
120buffer.
121
122.. _Percepio Tracealyzer: https://percepio.com/tracealyzer
123
124.. figure:: percepio_tracealyzer.png
125    :align: center
126    :alt: Percepio Tracealyzer
127    :figclass: align-center
128    :width: 80%
129
130Zephyr kernel events are captured automatically when Tracealyzer tracing is enabled.
131Tracealyzer also provides extensive support for application logging, where you call the tracing
132library from your application code. This lets you visualize kernel events and application events
133together, for example as data plots or state diagrams on logged variables.
134Learn more in the Tracealyzer User Manual provided with the application.
135
136Percepio TraceRecorder and Stream Ports
137---------------------------------------
138The tracing library for Tracealyzer (TraceRecorder) is included in the Zephyr manifest and
139provided under the same license (Apache 2.0). This is enabled by adding the following
140configuration options in your prj.conf:
141
142.. code-block:: cfg
143
144    CONFIG_TRACING=y
145    CONFIG_PERCEPIO_TRACERECORDER=y
146
147Or using menuconfig:
148
149* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
150* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
151  :guilabel:`Percepio Tracealyzer`
152
153Some additional settings are needed to configure TraceRecorder. The most important configuration
154is to select the right "stream port". This specifies how to output the trace data.
155As of July 2024, the following stream ports are available in the Zephyr configuration system:
156
157* **Ring Buffer**: The trace data is kept in a circular RAM buffer.
158* **RTT**: Trace streaming via Segger RTT on J-Link debug probes.
159* **ITM**: Trace streaming via the ITM function on Arm Cortex-M devices.
160* **Semihost**: For tracing on QEMU. Streams the trace data to a host file.
161
162Select the stream port in menuconfig under
163:menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`.
164
165Or simply add one of the following options in your prj.conf:
166
167.. code-block:: cfg
168
169    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER=y
170    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT=y
171    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ITM=y
172    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ZEPHYR_SEMIHOST=y
173
174Make sure to only include ONE of these configuration options.
175
176The stream port modules have individual configuration options. In menuconfig these are found
177under :menuselection:`Modules --> percepio --> TraceRecorder --> (Stream Port) Config`.
178The most important options for each stream port are described below.
179
180Tracealyzer Snapshot Tracing (Ring Buffer)
181------------------------------------------
182
183The "Ring Buffer" stream port keeps the trace data in a RAM buffer on the device.
184By default, this is a circular buffer, meaning that it always contains the most recent data.
185This is used to dump "snapshots" of the trace data, e.g. by using the debugger. This usually only
186allows for short traces, unless you have megabytes of RAM to spare, so it is not suitable for
187profiling. However, it can be quite useful for debugging in combination with breakpoints.
188For example, if you set a breakpoint in an error handler, a snapshot trace can show the sequence
189of events leading up to the error. Snapshot tracing is also easy to begin with, since it doesn't
190depend on any particular debug probe or other development tool.
191
192To use the Ring Buffer option, make sure to have the following configuration options in your
193prj.cnf:
194
195.. code-block:: cfg
196
197    CONFIG_TRACING=y
198    CONFIG_PERCEPIO_TRACERECORDER=y
199    CONFIG_PERCEPIO_TRC_START_MODE_START=y
200    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER=y
201    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RINGBUFFER_SIZE=<size in bytes>
202
203Or if using menuconfig:
204
205* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
206* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
207  :guilabel:`Percepio Tracealyzer`
208* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
209  :guilabel:`Start`
210* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
211  :guilabel:`Ring Buffer`
212* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Ring Buffer Config --> Buffer Size`,
213  set the buffer size in bytes.
214
215The default buffer size can be reduced if you are tight on RAM, or increased if you have RAM to
216spare and want longer traces. You may also optimize the Tracing Configuration settings to get
217longer traces by filtering out less important events.
218In menuconfig, see
219:menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Configuration`.
220
221To view the trace data, the easiest way is to start your debugger (west debug) and run the
222following GDB command::
223
224    dump binary value trace.bin *RecorderDataPtr
225
226The resulting file is typically found in the root of the build folder, unless a different path is
227specified. Open this file in Tracealyzer by selecting :menuselection:`File --> Open --> Open File`.
228
229Tracealyzer Streaming with SEGGER RTT
230-------------------------------------
231
232Tracealyzer has built-in support for SEGGER RTT to receive trace data using a J-Link probe.
233This allows for recording very long traces. To configure Zephyr for RTT streaming to Tracealyzer,
234add the following configuration options in your prj.cnf:
235
236.. code-block:: cfg
237
238    CONFIG_TRACING=y
239    CONFIG_PERCEPIO_TRACERECORDER=y
240    CONFIG_PERCEPIO_TRC_START_MODE_START_FROM_HOST=y
241    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT=y
242    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT_UP_BUFFER_SIZE=<size in bytes>
243
244Or if using menuconfig:
245
246* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
247* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
248  :guilabel:`Percepio Tracealyzer`
249* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
250  :guilabel:`Start From Host`
251* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
252  :guilabel:`RTT`
253* Under :menuselection:`Modules --> percepio --> TraceRecorder --> RTT Config`, set the size of the
254  RTT "up" buffer in bytes.
255
256The setting :guilabel:`RTT buffer size up` sets the size of the RTT transmission buffer. This is important
257for throughput. By default this buffer is quite large, 5000 bytes, to give decent performance
258also on onboard J-Link debuggers (they are not as fast as the stand-alone probes).
259If you are tight on RAM, you may consider reducing this setting. If using a regular J-Link probe
260it is often sufficient with a much smaller buffer, e.g. 1 KB or less.
261
262Learn more about RTT streaming in the Tracealyzer User Manual.
263See Creating and Loading Traces -> Percepio TraceRecorder -> Using TraceRecorder v4.6 or later ->
264Stream ports (or search for RTT).
265
266Tracealyzer Streaming with Arm ITM
267----------------------------------
268
269This stream port is for Arm Cortex-M devices featuring the ITM unit. It is recommended to use a
270fast debug probe that allows for SWO speeds of 10 MHz or higher. To use this stream port,
271apply the following configuration options:
272
273.. code-block:: cfg
274
275    CONFIG_TRACING=y
276    CONFIG_PERCEPIO_TRACERECORDER=y
277    CONFIG_PERCEPIO_TRC_START_MODE_START=y
278    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ITM=y
279    CONFIG_PERCEPIO_TRC_CFG_ITM_PORT=1
280
281Or if using menuconfig:
282
283* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
284* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
285  :guilabel:`Percepio Tracealyzer`
286* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
287  :guilabel:`Start`
288* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
289  :guilabel:`ITM`
290* Under :menuselection:`Modules --> percepio --> TraceRecorder --> ITM Config`, set the ITM port to
291  1.
292
293The main setting for the ITM stream port is the ITM port (0-31). A dedicated channel is needed
294for Tracealyzer. Port 0 is usually reserved for printf logging, so channel 1 is used by default.
295
296The option :guilabel:`Use internal buffer` should typically remain disabled. It buffers the data in RAM
297before transmission and defers the data transmission to the periodic TzCtrl thread.
298
299The host-side setup depends on what debug probe you are using. Learn more in the Tracealyzer
300User Manual.
301See :menuselection:`Creating and Loading Traces --> Percepio TraceRecorder --> Using TraceRecorder v4.6 or later --> Stream ports (or search for ITM)`.
302
303Tracealyzer Streaming from QEMU (Semihost)
304------------------------------------------
305
306This stream port is designed for Zephyr tracing in QEMU. This can be an easy way to get started
307with tracing and try out streaming trace without needing a fast debug probe. The data is streamed
308to a host file using semihosting. To use this option, apply the following configuration options:
309
310.. code-block:: cfg
311
312    CONFIG_SEMIHOST=y
313    CONFIG_TRACING=y
314    CONFIG_PERCEPIO_TRACERECORDER=y
315    CONFIG_PERCEPIO_TRC_START_MODE_START=y
316    CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_ZEPHYR_SEMIHOST=y
317
318Using menuconfig
319
320* Enable :menuselection:`General Architecture Options --> Semihosting support for Arm and RISC-V targets`
321* Enable :menuselection:`Subsystems and OS Services --> Tracing Support`
322* Under :menuselection:`Subsystems and OS Services --> Tracing Support --> Tracing Format`, select
323  :guilabel:`Percepio Tracealyzer`
324* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Recorder Start Mode`, select
325  :guilabel:`Start`
326* Under :menuselection:`Modules --> percepio --> TraceRecorder --> Stream Port`, select
327  :guilabel:`Semihost`
328
329By default, the resulting trace file is found in :file:`./trace.psf` in the root of the build folder,
330unless a different path is specified. Open this file in `Percepio Tracealyzer`_ by selecting
331:menuselection:`File --> Open --> Open File`.
332
333Recorder Start Mode
334-------------------
335
336You may have noticed the :guilabel:`Recorder Start Mode` option in the Tracealyzer examples above.
337This decides when the tracing starts. With the option :guilabel:`Start`, the tracing begins directly
338at startup, once the TraceRecorder library has been initialized. This is recommended when using the
339Ring Buffer and Semihost stream ports.
340
341For streaming via RTT or ITM you may also use :guilabel:`Start From Host` or
342:guilabel:`Start Await Host`. Both listens for start commands from the Tracealyzer application. The
343latter option, :guilabel:`Start Await Host`, causes the TraceRecorder initialization to block until
344the start command is received from the Tracealyzer application.
345
346Custom Stream Ports for Tracealyzer
347-----------------------------------
348
349The stream ports are small modules within TraceRecorder that define what functions to call to
350output the trace data and (optionally) how to read start/stop commands from Tracealyzer.
351It is fairly easy to make custom stream ports to implement your own data transport and
352Tracealyzer can receive trace streams over various interfaces, including files, sockets,
353COM ports, named pipes and more. Note that additional stream port modules are available in the
354TraceRecorder repo (e.g. lwIP), although they might require modifications to work with Zephyr.
355
356Learning More
357-------------
358
359Learn more about how to get started in the `Tracealyzer Getting Started Guides`_.
360
361.. _Tracealyzer Getting Started Guides: https://percepio.com/tracealyzer/gettingstarted/
362
363
364Percepio View for Zephyr
365========================
366Percepio View is a free-of-charge tracing tool based on `Percepio Tracealyzer`_, intended for
367debugging and verification of Zephyr applications.
368
369.. figure:: percepio_view.webp
370    :align: center
371    :alt: Percepio View
372    :figclass: align-center
373    :width: 80%
374
375Percepio View can be used side-by-side with a traditional debugger and complements your debugger
376by visualising the real-time execution of threads, ISRs, syscalls and your own “User Events”.
377
378.. figure:: percepio_view_user_event.webp
379    :align: center
380    :alt: Percepio View User Events
381    :figclass: align-center
382    :width: 80%
383
384
385To learn more about Percepio View, how to get started and upgrade options, check out
386`Percepio's product page
387<https://traceviewer.io/get-view/?target=zephyr>`_.
388
389Percepio View provides snapshot tracing, meaning the data is stored to a ring-buffer in target RAM
390and is saved to host using the regular debugger connection.
391For trace streaming support, Percepio offers (paid-for) upgrades to Percepio Profile or
392Percepio Tracealyzer. No modifications of the Zephyr source code are needed, only enabling the
393TraceRecorder library in Kconfig. Percepio View runs on Windows and Linux hosts.
394
395
396SEGGER SystemView Support
397=========================
398
399Zephyr provides built-in support for `SEGGER SystemView`_ that can be enabled in
400any application for platforms that have the required hardware support.
401
402The payload and format used with SystemView is custom to the application and
403relies on RTT as a transport. Newer versions of SystemView support other
404transports, for example UART or using snapshot mode (both still not
405supported in Zephyr).
406
407To enable tracing support with `SEGGER SystemView`_ add the
408:ref:`snippet-rtt-tracing` to your build command:
409
410    .. zephyr-app-commands::
411        :zephyr-app: samples/synchronization
412        :board: <board>
413        :snippets: rtt-tracing
414        :goals: build
415        :compact:
416
417SystemView can also be used for post-mortem tracing, which can be enabled with
418:kconfig:option:`CONFIG_SEGGER_SYSVIEW_POST_MORTEM_MODE`. In this mode, a debugger can
419be attached after the system has crashed using ``west attach`` after which the
420latest data from the internal RAM buffer can be loaded into SystemView.
421
422.. figure:: segger_systemview.png
423    :align: center
424    :alt: SEGGER SystemView
425    :figclass: align-center
426    :width: 80%
427
428.. _SEGGER SystemView: https://www.segger.com/products/development-tools/systemview/
429
430
431Recent versions of `SEGGER SystemView`_ come with an API translation table for
432Zephyr which is incomplete and does not match the current level of support
433available in Zephyr. To use the latest Zephyr API description table, copy the
434file available in the tree to your local configuration directory to override the
435builtin table::
436
437        # On Linux and MacOS
438        cp $ZEPHYR_BASE/subsys/tracing/sysview/SYSVIEW_Zephyr.txt ~/.config/SEGGER/
439
440TraceCompass
441=============
442
443TraceCompass is an open source tool that visualizes CTF events such as thread
444scheduling and interrupts, and is helpful to find unintended interactions and
445resource conflicts on complex systems.
446
447See also the presentation by Ericsson,
448`Advanced Trouble-shooting Of Real-time Systems
449<https://wiki.eclipse.org/images/0/0e/TechTalkOnlineDemoFeb2017_v1.pdf>`_.
450
451
452User-Defined Tracing
453====================
454
455This tracing format allows the user to define functions to perform any work desired
456when a task is switched in or out, when an interrupt is entered or exited, and when the cpu
457is idle.
458
459Examples include:
460- simple toggling of GPIO for external scope tracing while minimizing extra cpu load
461- generating/outputting trace data in a non-standard or proprietary format that can
462not be supported by the other tracing systems
463
464The following functions can be defined by the user:
465
466.. code-block:: c
467
468   void sys_trace_thread_create_user(struct k_thread *thread);
469   void sys_trace_thread_abort_user(struct k_thread *thread);
470   void sys_trace_thread_suspend_user(struct k_thread *thread);
471   void sys_trace_thread_resume_user(struct k_thread *thread);
472   void sys_trace_thread_name_set_user(struct k_thread *thread);
473   void sys_trace_thread_switched_in_user(struct k_thread *thread);
474   void sys_trace_thread_switched_out_user(struct k_thread *thread);
475   void sys_trace_thread_info_user(struct k_thread *thread);
476   void sys_trace_thread_sched_ready_user(struct k_thread *thread);
477   void sys_trace_thread_pend_user(struct k_thread *thread);
478   void sys_trace_thread_priority_set_user(struct k_thread *thread, int prio);
479   void sys_trace_isr_enter_user(int nested_interrupts);
480   void sys_trace_isr_exit_user(int nested_interrupts);
481   void sys_trace_idle_user();
482
483Enable this format with the :kconfig:option:`CONFIG_TRACING_USER` option.
484
485Transport Backends
486******************
487
488The following backends are currently supported:
489
490* UART
491* USB
492* File (Using the native port with POSIX architecture based targets)
493* RTT (With SystemView)
494* RAM (buffer to be retrieved by a debugger)
495
496Using Tracing
497*************
498
499The sample :zephyr_file:`samples/subsys/tracing` demonstrates tracing with
500different formats and backends.
501
502To get started, the simplest way is to use the CTF format with the :ref:`native_sim <native_sim>`
503port, build the sample as follows:
504
505.. zephyr-app-commands::
506   :tool: all
507   :zephyr-app: samples/subsys/tracing
508   :board: native_sim
509   :gen-args: -DCONF_FILE=prj_native_ctf.conf
510   :goals: build
511
512You can then run the resulting binary with the option ``-trace-file`` to generate
513the tracing data::
514
515    mkdir data
516    cp $ZEPHYR_BASE/subsys/tracing/ctf/tsdl/metadata data/
517    ./build/zephyr/zephyr.exe -trace-file=data/channel0_0
518
519The resulting CTF output can be visualized using babeltrace or TraceCompass
520by pointing the tool to the ``data`` directory with the metadata and trace files.
521
522Using RAM backend
523=================
524
525For devices that do not have available I/O for tracing such as USB or UART but have
526enough RAM to collect trace data, the ram backend can be enabled with configuration
527:kconfig:option:`CONFIG_TRACING_BACKEND_RAM`.
528Adjust :kconfig:option:`CONFIG_RAM_TRACING_BUFFER_SIZE` to be able to record enough traces for your needs.
529Then thanks to a runtime debugger such as gdb this buffer can be fetched from the target
530to an host computer::
531
532    (gdb) dump binary memory data/channel0_0 <ram_tracing_start> <ram_tracing_end>
533
534The resulting channel0_0 file have to be placed in a directory with the ``metadata``
535file like the other backend.
536
537Future LTTng Inspiration
538************************
539
540Currently, the top-layer provided here is quite simple and bare-bones,
541and needlessly copied from Zephyr's Segger SystemView debug module.
542
543For an OS like Zephyr, it would make sense to draw inspiration from
544Linux's LTTng and change the top-layer to serialize to the same format.
545Doing this would enable direct reuse of TraceCompass' canned analyses
546for Linux.  Alternatively, LTTng-analyses in TraceCompass could be
547customized to Zephyr.  It is ongoing work to enable TraceCompass
548visibility of Zephyr in a target-agnostic and open source way.
549
550
551I/O Taxonomy
552=============
553
554- Atomic Push/Produce/Write/Enqueue:
555
556  - synchronous:
557                  means data-transmission has completed with the return of the
558                  call.
559
560  - asynchronous:
561                  means data-transmission is pending or ongoing with the return
562                  of the call. Usually, interrupts/callbacks/signals or polling
563                  is used to determine completion.
564
565  - buffered:
566                  means data-transmissions are copied and grouped together to
567                  form a larger ones. Usually for amortizing overhead (burst
568                  dequeue) or jitter-mitigation (steady dequeue).
569
570  Examples:
571    - sync  unbuffered
572        E.g. PIO via GPIOs having steady stream, no extra FIFO memory needed.
573        Low jitter but may be less efficient (can't amortize the overhead of
574        writing).
575
576    - sync  buffered
577        E.g. ``fwrite()`` or enqueuing into FIFO.
578        Blockingly burst the FIFO when its buffer-waterlevel exceeds threshold.
579        Jitter due to bursts may lead to missed deadlines.
580
581    - async unbuffered
582        E.g. DMA, or zero-copying in shared memory.
583        Be careful of data hazards, race conditions, etc!
584
585    - async buffered
586        E.g. enqueuing into FIFO.
587
588
589
590- Atomic Pull/Consume/Read/Dequeue:
591
592  - synchronous:
593                  means data-reception has completed with the return of the call.
594
595  - asynchronous:
596                  means data-reception is pending or ongoing with the return of
597                  the call. Usually, interrupts/callbacks/signals or polling is
598                  used to determine completion.
599
600  - buffered:
601                  means data is copied-in in larger chunks than request-size.
602                  Usually for amortizing wait-time.
603
604  Examples:
605    - sync  unbuffered
606        E.g. Blocking read-call, ``fread()`` or SPI-read, zero-copying in shared
607        memory.
608
609    - sync  buffered
610        E.g. Blocking read-call with caching applied.
611        Makes sense if read pattern exhibits spatial locality.
612
613    - async unbuffered
614        E.g. zero-copying in shared memory.
615        Be careful of data hazards, race conditions, etc!
616
617    - async buffered
618        E.g. ``aio_read()`` or DMA.
619
620
621
622Unfortunately, I/O may not be atomic and may, therefore, require locking.
623Locking may not be needed if multiple independent channels are available.
624
625  - The system has non-atomic write and one shared channel
626        E.g. UART. Locking required.
627
628        ``lock(); emit(a); emit(b); emit(c); release();``
629
630  - The system has non-atomic write but many channels
631        E.g. Multi-UART. Lock-free if the bottom-layer maps each Zephyr
632        thread+ISR to its own channel, thus alleviating races as each
633        thread is sequentially consistent with itself.
634
635        ``emit(a,thread_id); emit(b,thread_id); emit(c,thread_id);``
636
637  - The system has atomic write     but one shared channel
638        E.g. ``native_sim`` or board with DMA. May or may not need locking.
639
640        ``emit(a ## b ## c); /* Concat to buffer */``
641
642        ``lock(); emit(a); emit(b); emit(c); release(); /* No extra mem */``
643
644  - The system has atomic write     and many channels
645        E.g. native_sim or board with multi-channel DMA. Lock-free.
646
647        ``emit(a ## b ## c, thread_id);``
648
649
650Object tracking
651***************
652
653The kernel can also maintain lists of objects that can be used to track
654their usage. Currently, the following lists can be enabled::
655
656  struct k_timer *_track_list_k_timer;
657  struct k_mem_slab *_track_list_k_mem_slab;
658  struct k_sem *_track_list_k_sem;
659  struct k_mutex *_track_list_k_mutex;
660  struct k_stack *_track_list_k_stack;
661  struct k_msgq *_track_list_k_msgq;
662  struct k_mbox *_track_list_k_mbox;
663  struct k_pipe *_track_list_k_pipe;
664  struct k_queue *_track_list_k_queue;
665  struct k_event *_track_list_k_event;
666
667Those global variables are the head of each list - they can be traversed
668with the help of macro ``SYS_PORT_TRACK_NEXT``. For instance, to traverse
669all initialized mutexes, one can write::
670
671  struct k_mutex *cur = _track_list_k_mutex;
672  while (cur != NULL) {
673    /* Do something */
674
675    cur = SYS_PORT_TRACK_NEXT(cur);
676  }
677
678To enable object tracking, enable :kconfig:option:`CONFIG_TRACING_OBJECT_TRACKING`.
679Note that each list can be enabled or disabled via their tracing
680configuration. For example, to disable tracking of semaphores, one can
681disable :kconfig:option:`CONFIG_TRACING_SEMAPHORE`.
682
683Object tracking is behind tracing configuration as it currently leverages
684tracing infrastructure to perform the tracking.
685
686API
687***
688
689
690Common
691======
692
693.. doxygengroup:: subsys_tracing_apis
694
695Threads
696=======
697
698.. doxygengroup:: subsys_tracing_apis_thread
699
700Work Queues
701===========
702
703.. doxygengroup:: subsys_tracing_apis_work
704
705Poll
706====
707
708.. doxygengroup:: subsys_tracing_apis_poll
709
710Semaphore
711=========
712
713.. doxygengroup:: subsys_tracing_apis_sem
714
715Mutex
716=====
717
718.. doxygengroup:: subsys_tracing_apis_mutex
719
720Condition Variables
721===================
722
723.. doxygengroup:: subsys_tracing_apis_condvar
724
725Queues
726======
727
728.. doxygengroup:: subsys_tracing_apis_queue
729
730FIFO
731====
732
733.. doxygengroup:: subsys_tracing_apis_fifo
734
735LIFO
736====
737.. doxygengroup:: subsys_tracing_apis_lifo
738
739Stacks
740======
741
742.. doxygengroup:: subsys_tracing_apis_stack
743
744Message Queues
745==============
746
747.. doxygengroup:: subsys_tracing_apis_msgq
748
749Mailbox
750=======
751
752.. doxygengroup:: subsys_tracing_apis_mbox
753
754Pipes
755======
756
757.. doxygengroup:: subsys_tracing_apis_pipe
758
759Heaps
760=====
761
762.. doxygengroup:: subsys_tracing_apis_heap
763
764Memory Slabs
765============
766
767.. doxygengroup:: subsys_tracing_apis_mslab
768
769Timers
770======
771
772.. doxygengroup:: subsys_tracing_apis_timer
773
774Object tracking
775===============
776
777.. doxygengroup:: subsys_tracing_object_tracking
778
779Syscalls
780========
781
782.. doxygengroup:: subsys_tracing_apis_syscall
783
784Network tracing
785===============
786
787.. doxygengroup:: subsys_tracing_apis_net
788
789Network socket tracing
790======================
791
792.. doxygengroup:: subsys_tracing_apis_socket
793