1.. SPDX-License-Identifier: GPL-2.0
2
3=====================
4Devices and Protocols
5=====================
6
7The type of CXL device (Memory, Accelerator, etc) dictates many configuration steps. This section
8covers some basic background on device types and on-device resources used by the platform and OS
9which impact configuration.
10
11Protocols
12=========
13
14There are three core protocols to CXL.  For the purpose of this documentation,
15we will only discuss very high level definitions as the specific hardware
16details are largely abstracted away from Linux.  See the CXL specification
17for more details.
18
19CXL.io
20------
21The basic interaction protocol, similar to PCIe configuration mechanisms.
22Typically used for initialization, configuration, and I/O access for anything
23other than memory (CXL.mem) or cache (CXL.cache) operations.
24
25The Linux CXL driver exposes access to .io functionalty via the various sysfs
26interfaces and /dev/cxl/ devices (which exposes direct access to device
27mailboxes).
28
29CXL.cache
30---------
31The mechanism by which a device may coherently access and cache host memory.
32
33Largely transparent to Linux once configured.
34
35CXL.mem
36---------
37The mechanism by which the CPU may coherently access and cache device memory.
38
39Largely transparent to Linux once configured.
40
41
42Device Types
43============
44
45Type-1
46------
47
48A Type-1 CXL device:
49
50* Supports cxl.io and cxl.cache protocols
51* Implements a fully coherent cache
52* Allows Device-to-Host coherence and Host-to-Device snoops.
53* Does NOT have host-managed device memory (HDM)
54
55Typical examples of type-1 devices is a Smart NIC - which may want to
56directly operate on host-memory (DMA) to store incoming packets. These
57devices largely rely on CPU-attached memory.
58
59Type-2
60------
61
62A Type-2 CXL Device:
63
64* Supports cxl.io, cxl.cache, and cxl.mem protocols
65* Optionally implements coherent cache and Host-Managed Device Memory
66* Is typically an accelerator device with high bandwidth memory.
67
68The primary difference between a type-1 and type-2 device is the presence
69of host-managed device memory, which allows the device to operate on a
70local memory bank - while the CPU still has coherent DMA to the same memory.
71
72This allows things like GPUs to expose their memory via DAX devices or file
73descriptors, allows drivers and programs direct access to device memory
74rather than use block-transfer semantics.
75
76Type-3
77------
78
79A Type-3 CXL Device
80
81* Supports cxl.io and cxl.mem
82* Implements Host-Managed Device Memory
83* May provide either Volatile or Persistent memory capacity (or both).
84
85A basic example of a type-3 device is a simple memory expander, whose
86local memory capacity is exposed to the CPU for access directly via
87basic coherent DMA.
88
89Switch
90------
91
92A CXL switch is a device capable of routing any CXL (and by extension, PCIe)
93protocol between an upstream, downstream, or peer devices.  Many devices, such
94as Multi-Logical Devices, imply the presence of switching in some manner.
95
96Logical Devices and Heads
97-------------------------
98
99A CXL device may present one or more "Logical Devices" to one or more hosts
100(via physical "Heads").
101
102A Single-Logical Device (SLD) is a device which presents a single device to
103one or more heads.
104
105A Multi-Logical Device (MLD) is a device which may present multiple devices
106to one or more upstream devices.
107
108A Single-Headed Device exposes only a single physical connection.
109
110A Multi-Headed Device exposes multiple physical connections.
111
112MHSLD
113~~~~~
114A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical
115device to multiple heads which may be connected to one or more discrete
116hosts.  An example of this would be a simple memory-pool which may be
117statically configured (prior to boot) to expose portions of its memory
118to Linux via :doc:`CEDT <../platform/acpi/cedt>`.
119
120MHMLD
121~~~~~
122A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical
123devices to multiple heads which may be connected to one or more discrete
124hosts.  An example of this would be a Dynamic Capacity Device or which
125may be configured at runtime to expose portions of its memory to Linux.
126
127Example Devices
128===============
129
130Memory Expander
131---------------
132The simplest form of Type-3 device is a memory expander.  A memory expander
133exposes Host-Managed Device Memory (HDM) to Linux.  This memory may be
134Volatile or Non-Volatile (Persistent).
135
136Memory Expanders will typically be considered a form of Single-Headed,
137Single-Logical Device - as its form factor will typically be an add-in-card
138(AIC) or some other similar form-factor.
139
140The Linux CXL driver provides support for static or dynamic configuration of
141basic memory expanders.  The platform may program decoders prior to OS init
142(e.g. auto-decoders), or the user may program the fabric if the platform
143defers these operations to the OS.
144
145Multiple Memory Expanders may be added to an external chassis and exposed to
146a host via a head attached to a CXL switch.  This is a "memory pool", and
147would be considered an MHSLD or MHMLD depending on the management capabilities
148provided by the switch platform.
149
150As of v6.14, Linux does not provide a formalized interface to manage non-DCD
151MHSLD or MHMLD devices.
152
153Dynamic Capacity Device (DCD)
154-----------------------------
155
156A Dynamic Capacity Device is a Type-3 device which provides dynamic management
157of memory capacity. The basic premise of a DCD to provide an allocator-like
158interface for physical memory capacity to a "Fabric Manager" (an external,
159privileged host with privileges to change configurations for other hosts).
160
161A DCD manages "Memory Extents", which may be volatile or persistent. Extents
162may also be exclusive to a single host or shared across multiple hosts.
163
164As of v6.14, Linux does not provide a formalized interface to manage DCD
165devices, however there is active work on LKML targeting future release.
166