1.. SPDX-License-Identifier: GPL-2.0
2.. include:: <isonum.txt>
3
4==================================
5CXL Access Coordinates Computation
6==================================
7
8Latency and Bandwidth Calculation
9=================================
10A memory region performance coordinates (latency and bandwidth) are typically
11provided via ACPI tables :doc:`SRAT <../platform/acpi/srat>` and
12:doc:`HMAT <../platform/acpi/hmat>`. However, the platform firmware (BIOS) is
13not able to annotate those for CXL devices that are hot-plugged since they do
14not exist during platform firmware initialization. The CXL driver can compute
15the performance coordinates by retrieving data from several components.
16
17The :doc:`SRAT <../platform/acpi/srat>` provides a Generic Port Affinity
18subtable that ties a proximity domain to a device handle, which in this case
19would be the CXL hostbridge. Using this association, the performance
20coordinates for the Generic Port can be retrieved from the
21:doc:`HMAT <../platform/acpi/hmat>` subtable. This piece represents the
22performance coordinates between a CPU and a Generic Port (CXL hostbridge).
23
24The :doc:`CDAT <../platform/cdat>` provides the performance coordinates for
25the CXL device itself. That is the bandwidth and latency to access that device's
26memory region. The DSMAS subtable provides a DSMADHandle that is tied to a
27Device Physical Address (DPA) range. The DSLBIS subtable provides the
28performance coordinates that's tied to a DSMADhandle and this ties the two
29table entries together to provide the performance coordinates for each DPA
30region. For example, if a device exports a DRAM region and a PMEM region,
31then there would be different performance characteristsics for each of those
32regions.
33
34If there's a CXL switch in the topology, then the performance coordinates for the
35switch is provided by SSLBIS subtable. This provides the bandwidth and latency
36for traversing the switch between the switch upstream port and the switch
37downstream port that points to the endpoint device.
38
39Simple topology example::
40
41 GP0/HB0/ACPI0016-0
42        RP0
43         |
44         | L0
45         |
46     SW 0 / USP0
47     SW 0 / DSP0
48         |
49         | L1
50         |
51        EP0
52
53In this example, there is a CXL switch between an endpoint and a root port.
54Latency in this example is calculated as such:
55L(EP0) - Latency from EP0 CDAT DSMAS+DSLBIS
56L(L1) - Link latency between EP0 and SW0DSP0
57L(SW0) - Latency for the switch from SW0 CDAT SSLBIS.
58L(L0) - Link latency between SW0 and RP0
59L(RP0) - Latency from root port to CPU via SRAT and HMAT (Generic Port).
60Total read and write latencies are the sum of all these parts.
61
62Bandwidth in this example is calculated as such:
63B(EP0) - Bandwidth from EP0 CDAT DSMAS+DSLBIS
64B(L1) - Link bandwidth between EP0 and SW0DSP0
65B(SW0) - Bandwidth for the switch from SW0 CDAT SSLBIS.
66B(L0) - Link bandwidth between SW0 and RP0
67B(RP0) - Bandwidth from root port to CPU via SRAT and HMAT (Generic Port).
68The total read and write bandwidth is the min() of all these parts.
69
70To calculate the link bandwidth:
71LinkOperatingFrequency (GT/s) is the current negotiated link speed.
72DataRatePerLink (MB/s) = LinkOperatingFrequency / 8
73Bandwidth (MB/s) = PCIeCurrentLinkWidth * DataRatePerLink
74Where PCIeCurrentLinkWidth is the number of lanes in the link.
75
76To calculate the link latency:
77LinkLatency (picoseconds) = FlitSize / LinkBandwidth (MB/s)
78
79See `CXL Memory Device SW Guide r1.0 <https://www.intel.com/content/www/us/en/content-details/643805/cxl-memory-device-software-guide.html>`_,
80section 2.11.3 and 2.11.4 for details.
81
82In the end, the access coordinates for a constructed memory region is calculated from one
83or more memory partitions from each of the CXL device(s).
84
85Shared Upstream Link Calculation
86================================
87For certain CXL region construction with endpoints behind CXL switches (SW) or
88Root Ports (RP), there is the possibility of the total bandwidth for all
89the endpoints behind a switch being more than the switch upstream link.
90A similar situation can occur within the host, upstream of the root ports.
91The CXL driver performs an additional pass after all the targets have
92arrived for a region in order to recalculate the bandwidths with possible
93upstream link being a limiting factor in mind.
94
95The algorithm assumes the configuration is a symmetric topology as that
96maximizes performance. When asymmetric topology is detected, the calculation
97is aborted. An asymmetric topology is detected during topology walk where the
98number of RPs detected as a grandparent is not equal to the number of devices
99iterated in the same iteration loop. The assumption is made that subtle
100asymmetry in properties does not happen and all paths to EPs are equal.
101
102There can be multiple switches under an RP. There can be multiple RPs under
103a CXL Host Bridge (HB). There can be multiple HBs under a CXL Fixed Memory
104Window Structure (CFMWS) in the :doc:`CEDT <../platform/acpi/cedt>`.
105
106An example hierarchy::
107
108                CFMWS 0
109                  |
110         _________|_________
111        |                   |
112    ACPI0017-0          ACPI0017-1
113 GP0/HB0/ACPI0016-0   GP1/HB1/ACPI0016-1
114    |          |        |           |
115   RP0        RP1      RP2         RP3
116    |          |        |           |
117  SW 0       SW 1     SW 2        SW 3
118  |   |      |   |    |   |       |   |
119 EP0 EP1    EP2 EP3  EP4  EP5    EP6 EP7
120
121Computation for the example hierarchy:
122
123Min (GP0 to CPU BW,
124     Min(SW 0 Upstream Link to RP0 BW,
125         Min(SW0SSLBIS for SW0DSP0 (EP0), EP0 DSLBIS, EP0 Upstream Link) +
126         Min(SW0SSLBIS for SW0DSP1 (EP1), EP1 DSLBIS, EP1 Upstream link)) +
127     Min(SW 1 Upstream Link to RP1 BW,
128         Min(SW1SSLBIS for SW1DSP0 (EP2), EP2 DSLBIS, EP2 Upstream Link) +
129         Min(SW1SSLBIS for SW1DSP1 (EP3), EP3 DSLBIS, EP3 Upstream link))) +
130Min (GP1 to CPU BW,
131     Min(SW 2 Upstream Link to RP2 BW,
132         Min(SW2SSLBIS for SW2DSP0 (EP4), EP4 DSLBIS, EP4 Upstream Link) +
133         Min(SW2SSLBIS for SW2DSP1 (EP5), EP5 DSLBIS, EP5 Upstream link)) +
134     Min(SW 3 Upstream Link to RP3 BW,
135         Min(SW3SSLBIS for SW3DSP0 (EP6), EP6 DSLBIS, EP6 Upstream Link) +
136         Min(SW3SSLBIS for SW3DSP1 (EP7), EP7 DSLBIS, EP7 Upstream link))))
137
138The calculation starts at cxl_region_shared_upstream_perf_update(). A xarray
139is created to collect all the endpoint bandwidths via the
140cxl_endpoint_gather_bandwidth() function. The min() of bandwidth from the
141endpoint CDAT and the upstream link bandwidth is calculated. If the endpoint
142has a CXL switch as a parent, then min() of calculated bandwidth and the
143bandwidth from the SSLBIS for the switch downstream port that is associated
144with the endpoint is calculated. The final bandwidth is stored in a
145'struct cxl_perf_ctx' in the xarray indexed by a device pointer. If the
146endpoint is direct attached to a root port (RP), the device pointer would be an
147RP device. If the endpoint is behind a switch, the device pointer would be the
148upstream device of the parent switch.
149
150At the next stage, the code walks through one or more switches if they exist
151in the topology. For endpoints directly attached to RPs, this step is skipped.
152If there is another switch upstream, the code takes the min() of the current
153gathered bandwidth and the upstream link bandwidth. If there's a switch
154upstream, then the SSLBIS of the upstream switch.
155
156Once the topology walk reaches the RP, whether it's direct attached endpoints
157or walking through the switch(es), cxl_rp_gather_bandwidth() is called. At
158this point all the bandwidths are aggregated per each host bridge, which is
159also the index for the resulting xarray.
160
161The next step is to take the min() of the per host bridge bandwidth and the
162bandwidth from the Generic Port (GP). The bandwidths for the GP are retrieved
163via ACPI tables (:doc:`SRAT <../platform/acpi/srat>` and
164:doc:`HMAT <../platform/acpi/hmat>`). The minimum bandwidth are aggregated
165under the same ACPI0017 device to form a new xarray.
166
167Finally, the cxl_region_update_bandwidth() is called and the aggregated
168bandwidth from all the members of the last xarray is updated for the
169access coordinates residing in the cxl region (cxlr) context.
170
171QTG ID
172======
173Each :doc:`CEDT <../platform/acpi/cedt>` has a QTG ID field. This field provides
174the ID that associates with a QoS Throttling Group (QTG) for the CFMWS window.
175Once the access coordinates are calculated, an ACPI Device Specific Method can
176be issued to the ACPI0016 device to retrieve the QTG ID depends on the access
177coordinates provided. The QTG ID for the device can be used as guidance to match
178to the CFMWS to setup the best Linux root decoder for the device performance.
179