1.. _hv_rdt:
2
3RDT Allocation Feature Supported by Hypervisor
4##############################################
5
6The ACRN hypervisor uses RDT (Resource Director Technology) allocation features
7such as CAT (Cache Allocation Technology) and MBA (Memory Bandwidth
8Allocation) to control VMs which may be over-utilizing cache resources or
9memory bandwidth relative to their priorities. By setting limits to critical
10resources, ACRN can optimize RTVM performance over regular VMs. In ACRN, the
11CAT and MBA are configured via the "VM-Configuration". The resources
12allocated for VMs are determined in the VM configuration (:ref:`rdt_vm_configuration`).
13
14For further details on the Intel RDT, refer to `Intel 64 and IA-32 Architectures Software Developer's Manual (SDM), Volume 3, (Section 17.19 Intel Resource Director Technology Allocation Features) <https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html>`_.
15
16
17Objective of CAT
18****************
19The CAT feature in the hypervisor can isolate the cache for a VM from other
20VMs. It can also isolate cache usage between VMX root and non-root
21modes. Generally, certain cache resources are allocated for the
22RT VMs in order to reduce performance interference through the shared
23cache access from the neighbor VMs.
24
25The figure below shows that with CAT, the cache ways can be isolated vs
26the default where high priority VMs can be impacted by a noisy neighbor.
27
28   .. figure:: images/cat-objective.png
29      :align: center
30
31CAT Support in ACRN
32===================
33On x86 platforms that support CAT, the ACRN hypervisor automatically enables
34support and by default shares the cache ways equally between all VMs.
35This is done by setting the max cache mask in the MSR_IA32_type_MASK_n (where
36type: L2 or L3) MSR that corresponds to each CLOS and then setting the
37IA32_PQR_ASSOC MSR to CLOS 0. (Note that CLOS, or Class of Service, is a
38resource allocator.) The user can check the cache capabilities such as cache
39mask and max supported CLOS as described in :ref:`rdt_detection_capabilities`
40and then program the IA32_type_MASK_n and IA32_PQR_ASSOC MSR with a
41CLOS ID, to select a cache mask to take effect. These configurations can be
42done in scenario XML file under ``FEATURES`` section as shown in the below example.
43ACRN uses VMCS MSR loads on every VM Entry/VM Exit for non-root and root modes
44to enforce the settings.
45
46   .. code-block:: none
47      :emphasize-lines: 2,4
48
49      <RDT>
50            <RDT_ENABLED>y</RDT_ENABLED>
51            <CDP_ENABLED>n</CDP_ENABLED>
52            <CLOS_MASK>0xF</CLOS_MASK>
53
54Once the cache mask is set of each individual CPU, the respective CLOS ID
55needs to be set in the scenario XML file under ``VM`` section. If user desires
56to use CDP feature, ``CDP_ENABLED`` should be set to ``y``.
57
58   .. code-block:: none
59      :emphasize-lines: 2
60
61      <clos>
62            <vcpu_clos>0</vcpu_clos>
63
64.. note::
65   ACRN takes the lowest common CLOS max value between the supported
66   resources as maximum supported CLOS ID. For example, if max CLOS
67   supported by L3 is 16 and MBA is 8, ACRN programs MAX_PLATFORM_CLOS_NUM
68   to 8. ACRN recommends to have consistent capabilities across all RDT
69   resources by using a common subset CLOS. This is done in order to minimize
70   misconfiguration errors.
71
72
73Objective of MBA
74****************
75The Memory Bandwidth Allocation (MBA) feature provides indirect and
76approximate control over memory bandwidth that's available per core. It
77provides a method to control VMs which may be over-utilizing bandwidth
78relative to their priorities and thus improves the performance of high
79priority VMs. MBA introduces a programmable request rate controller (PRRC)
80between cores and high-speed interconnect. Throttling values can be
81programmed via MSRs to the PRRC to limit bandwidth availability.
82
83The following figure shows memory bandwidth impact without MBA which causes
84bottlenecks for high priority VMs vs with MBA support:
85
86.. figure:: images/no_mba_objective.png
87   :align: center
88   :name: without-mba-support
89
90   Without MBA Support
91
92.. figure:: images/mba_objective.png
93   :align: center
94   :name: with-mba-support
95
96   With MBA Support
97
98
99MBA Support in ACRN
100===================
101On x86 platforms that support MBA, the ACRN hypervisor automatically enables
102support and by default sets no limits to the memory bandwidth access by VMs.
103This is done by setting a 0 mba delay value in the MSR_IA32_MBA_MASK_n MSR
104that corresponds to each CLOS and then setting IA32_PQR_ASSOC MSR with CLOS
1050. To select a delay to take effect for restricting memory bandwidth,
106users can check the MBA capabilities such as mba delay values and
107max supported CLOS as described in :ref:`rdt_detection_capabilities` and
108then program the IA32_MBA_MASK_n and IA32_PQR_ASSOC MSR with the CLOS ID.
109These configurations can be done in scenario XML file under ``FEATURES`` section
110as shown in the below example. ACRN uses VMCS MSR loads on every VM Entry/VM Exit
111for non-root and root modes to enforce the settings.
112
113   .. code-block:: none
114      :emphasize-lines: 2,5
115
116      <RDT>
117            <RDT_ENABLED>y</RDT_ENABLED>
118            <CDP_ENABLED>n</CDP_ENABLED>
119            <CLOS_MASK></CLOS_MASK>
120            <MBA_DELAY>0</MBA_DELAY>
121
122Once the cache mask is set of each individual CPU, the respective CLOS ID
123needs to be set in the scenario XML file under ``VM`` section.
124
125   .. code-block:: none
126      :emphasize-lines: 2
127
128      <clos>
129            <vcpu_clos>0</vcpu_clos>
130
131.. note::
132   ACRN takes the lowest common CLOS max value between the supported
133   resources as maximum supported CLOS ID. For example, if max CLOS
134   supported by L3 is 16 and MBA is 8, ACRN programs MAX_PLATFORM_CLOS_NUM
135   to 8. ACRN recommends to have consistent capabilities across all RDT
136   resources by using a common subset CLOS. This is done in order to minimize
137   misconfiguration errors.
138
139
140CAT and MBA High-Level Design in ACRN
141*************************************
142
143Data Structures
144===============
145The below figure shows the RDT data structure to store enumerated resources.
146
147   .. figure:: images/mba_data_structures.png
148      :align: center
149
150Enabling CAT, MBA Software Flow
151===============================
152
153The hypervisor enumerates RDT capabilities and sets up mask arrays; it also
154sets up CLOS for VMs and the hypervisor itself per the "vm configuration"(:ref:`rdt_vm_configuration`).
155
156- The RDT capabilities are enumerated on the bootstrap processor (BSP) during
157  the pCPU pre-initialize stage. The global data structure ``res_cap_info``
158  stores the capabilities of the supported resources.
159
160- If CAT or/and MBA is supported, then setup masks array on all APs at the
161  pCPU post-initialize stage. The mask values are written to
162  IA32_type_MASK_n. Refer to :ref:`rdt_detection_capabilities` for details
163  on identifying values to program the mask/delay MRSs and the max CLOS.
164
165- If CAT or/and MBA is supported, the CLOS of a **VM** will be stored into
166  its vCPU ``msr_store_area`` data structure guest part. It will be loaded
167  to MSR IA32_PQR_ASSOC at each VM entry.
168
169- If CAT or/and MBA is supported, the CLOS of **hypervisor** is stored for
170  all VMs, in their vCPU ``msr_store_area`` data structure host part. It will
171  be loaded to MSR IA32_PQR_ASSOC at each VM exit.
172
173The figure below shows the high level overview of RDT resource flow in the
174ACRN hypervisor.
175
176   .. figure:: images/cat_mba_software_flow.png
177      :align: center
178