1.. _vt-d-hld:
2
3VT-d
4####
5
6Intel Virtualization Technology for Directed I/O, or VT-d, provides
7hardware support for I/O device virtualization. It extends the
8protection and isolation properties of VMs for I/O operations.
9
10VT-d provides the following main functions:
11
12-  **DMA remapping**: for supporting address translations for DMA from
13   devices.
14
15-  **Interrupt remapping**: for supporting isolation and routing of
16   interrupts from devices and external interrupt controllers to the
17   appropriate VMs.
18
19-  **Interrupt posting**: for supporting direct delivery of virtual
20   interrupts from devices and external controllers to virtual
21   processors.
22
23The ACRN hypervisor supports DMA remapping that provides address translation
24capability for PCI passthrough devices, and second-level translation,
25which applies to requests-without-PASID. ACRN does not support
26First-level/nested translation.
27
28DMAR Engines Discovery
29**********************
30
31DMA Remapping Report ACPI Table
32===============================
33
34For generic platforms, the ACRN hypervisor retrieves DMAR information from
35the ACPI table and then parses the DMAR reporting structure to discover the
36number of DMA-remapping hardware units present in the platform as well as
37the devices under the scope of a remapping hardware unit, as shown in
38:numref:`dma-remap-report`:
39
40.. figure:: images/vt-d-image90.png
41   :align: center
42   :name: dma-remap-report
43
44   DMA Remapping Reporting Structure
45
46Pre-Parsed DMAR Information
47===========================
48
49For specific platforms, the ACRN hypervisor uses pre-parsed DMA remapping
50reporting information directly to save hypervisor bootup time.
51
52DMA Remapping
53*************
54
55DMA remapping hardware is used to isolate device access to memory,
56enabling each device in the system to be assigned to a specific domain
57through a distinct set of paging structures.
58
59Domains
60=======
61
62A domain is abstractly defined as an isolated environment in the
63platform, to which a subset of the host physical memory is allocated.
64The memory resource of a domain is specified by the address translation
65tables.
66
67Device-to-Domain Mapping Structure
68==================================
69
70VT-d hardware uses root-table and context-tables to build the mapping
71between devices and domains as shown in :numref:`vt-d-mapping`.
72
73.. figure:: images/vt-d-image44.png
74   :align: center
75   :name: vt-d-mapping
76
77   Device-to-Domain Mapping Structures
78
79The root-table is 4-KByte in size and contains 256 root-entries to cover
80the PCI bus number space (0-255). Each root-entry contains a
81context-table pointer to reference the context-table for devices on the
82bus identified by the root-entry, if the present flag of the root-entry
83is set.
84
85Each context-table contains 256 entries, with each entry corresponding
86to a PCI device function on the bus. For a PCI device, the device and
87function numbers (8-bits) are used to index into the context-table. Each
88context-entry contains a Second-level Page-table Pointer, which provides
89the host physical address of the address translation structure in system
90memory to be used for remapping requests-without-PASID processed through
91the context-entry.
92
93For a given Bus, Device, and Function combination as shown in
94:numref:`bdf-passthru`, a passthrough device can be associated with the
95address translation structures for a domain.
96
97.. figure:: images/vt-d-image19.png
98   :align: center
99   :name: bdf-passthru
100
101   BDF Format of Passthrough Device
102
103Refer to the `VT-d spec`_ for more details on device-to-domain
104mapping structures.
105
106.. _VT-d spec:
107   https://cdrdv2.intel.com/v1/dl/getContent/671081
108
109Address Translation Structures
110==============================
111
112For ACRN, the EPT table of a domain is used as the address translation
113structures for the devices assigned to the domain, as shown in
114:numref:`vt-d-DMA`.
115
116.. figure:: images/vt-d-image40.png
117   :align: center
118   :name: vt-d-DMA
119
120   DMA Remapping Diagram
121
122When the device attempts to access system memory, the DMA remapping hardware
123intercepts the access and utilizes the EPT table of the domain to determine
124whether the access is allowed. It then translates the DMA address according
125to the EPT table from the guest physical address (GPA) to the host physical
126address (HPA).
127
128Domains and Memory Isolation
129============================
130
131DMA operations do not exist inside the hypervisor, so ACRN doesn't
132create a domain for the hypervisor. No DMA operations from passthrough
133devices can access the hypervisor memory.
134
135ACRN treats each virtual machine (VM) as a separate domain. For a VM,
136an EPT table exists for Normal world; an EPT table for Secure world might
137also exist. Secure world can access Normal world's memory, but Normal
138world cannot access Secure world's memory.
139
140Service VM domain
141   The Service VM domain is created when the hypervisor creates the Service VM.
142
143   IOMMU uses the Normal world's EPT table of the Service VM as the address
144   translation structures for the devices in the Service VM domain. The Normal
145   world's EPT table of the Service VM doesn't include the memory resource of
146   the hypervisor and Secure worlds (if any exists). The devices in the
147   Service VM domain can't access the memory belonging to the hypervisor or
148   Secure worlds.
149
150Other domains
151   Other VM domains are created when the hypervisor creates User VMs,
152   one domain for each User VM.
153
154   IOMMU uses the Normal world's EPT table of a VM as the address
155   translation structures for the devices in the domain. The Normal world's
156   EPT table of the VM allows devices to access only the memory
157   allocated for the Normal world of the VM.
158
159Page-Walk Coherency
160===================
161
162For the VT-d hardware, which doesn't support page-walk coherency, the
163hypervisor needs to make sure the updates of VT-d tables are synced in
164memory:
165
166-  Device-to-Domain Mapping Structures, including Root-entries and
167   Context-entries.
168
169-  The EPT table of a VM.
170
171ACRN flushes the related cache line after these structures are updated
172if the VT-d hardware doesn't support page-walk coherency.
173
174Super-Page Support
175==================
176
177The ACRN VT-d reuses the EPT table as the address translation table. VT-d
178capability or super-page support should be identical with the usage of the
179EPT table.
180
181Snoop Control
182=============
183
184If VT-d hardware supports Snoop Control, VT-d can control the
185ability to ignore the "no-snoop attribute" in PCIe transactions.
186
187The following table shows the snoop behavior of a DMA operation controlled by
188the following:
189
190-  Snoop Control capability of VT-d DMAR unit
191-  The setting of SNP filed in leaf PTE
192-  No-snoop attribute in PCIe request
193
194.. list-table::
195   :widths: 25 25 25 25
196   :header-rows: 1
197
198   * - Snoop Control capability of VT-d
199     - SNP filed in leaf PTE
200     - No-snoop attribute in request
201     - Snoop behavior
202
203   * - 0
204     - 0 (must be 0)
205     - no snoop
206     - No snoop
207
208   * - 0
209     - 0 (must be 0)
210     - snoop
211     - Snoop
212
213   * - 1
214     - 1
215     - snoop / no snoop
216     - Snoop
217
218   * - 1
219     - 0
220     - no snoop
221     - No snoop
222
223   * - 1
224     - 0
225     - snoop
226     - Snoop
227
228If VT-d DMAR units do not support Snoop Control, then the SNP Bit (bit 11)
229of leaf PTEs of the EPT is not set since the field is treated as reserved (0)
230by the VT-d hardware implementations of not supporting Snoop Control.
231
232The VT-d DMAR unit of the Intel integrated graphics device doesn't support
233Snoop Control. The ACRN hypervisor uses the same copy of EPT as the
234secondary address translation table for a VM. When the DMAR unit for the
235Intel integrated graphics device is enabled, the SNP Bit cannot be set in
236the leaf PTEs of the EPT.
237
238No matter if ACRN enables or disables Snoop Control, the DMA operations of
239passthrough devices behave correctly from the guest's point of view. ACRN
240disables Snoop Control in VT-d DMAR engines that simplify the implementation.
241Also, since the snoop behavior of PCIe transactions can be controlled by
242guest drivers, some devices may take advantage of the NO_SNOOP_ATTRIBUTE
243of PCIe transactions for better performance when snoop is not needed.
244
245The driver is responsible for configuring the correct attribute in PCIe
246transactions. Otherwise, the corresponding device may not work properly.
247
248Initialization
249**************
250
251During hypervisor initialization, it registers DMAR units on the
252platform according to the reparsed information or DMAR table. There may
253be multiple DMAR units on the platform; ACRN allows some of the DMAR
254units to be ignored. If some DMAR units are marked as ignored, they
255would not be enabled.
256
257The hypervisor creates the Service VM domain using the Normal World's EPT table
258of the Service VM as the address translation table when creating the Service VM.
259All PCI devices on the platform are added to the Service VM domain. Then enable
260DMAR translation for DMAR units if they are not marked as ignored.
261
262.. _device-assignment:
263
264Device Assignment
265*****************
266
267All devices are initially added to the Service VM domain. To assign a device
268means to assign the device to a User VM. The device is removed from the
269Service VM domain and is added to the VM domain related to the User VM, which
270changes the address translation table from the EPT of the Service VM to the EPT
271of the User VM for the device.
272
273To unassign a device means to unassign the device from a User VM. The
274device is removed from the VM domain related to the User VM and then added
275back to the Service VM domain; this changes the address translation table from
276the EPT of the User VM to the EPT of the Service VM for the device.
277
278Power Management Support for S3
279*******************************
280
281During platform S3 suspend and resume, the VT-d register values are
282lost. ACRN VT-d provides APIs that are called during S3 suspend and resume.
283
284During S3 suspend, some register values are saved in the memory, and
285DMAR translation is disabled. During S3 resume, the register values
286saved are restored. The Root table address register is set. The DMAR
287translation is enabled.
288
289All operations for S3 suspend and resume are performed on all DMAR
290units on the platform, except for the DMAR units marked ignored.
291
292Error Handling
293**************
294
295ACRN VT-d supports DMA remapping error reporting. ACRN VT-d requests an
296IRQ / vector for DMAR error reporting. A DMAR fault handler is
297registered for the IRQ. DMAR units support reporting fault events via an MSI.
298When a fault event occurs, an MSI is generated, and the DMAR fault
299handler reports the error event.
300
301Data Structures and Interfaces
302******************************
303
304Initialization and Deinitialization
305===================================
306
307The following APIs are provided during initialization and
308deinitialization:
309
310.. doxygenfunction:: init_iommu
311   :project: Project ACRN
312
313Runtime
314=======
315
316The following API are provided during runtime:
317
318.. doxygenfunction:: create_iommu_domain
319   :project: Project ACRN
320
321.. doxygenfunction:: destroy_iommu_domain
322   :project: Project ACRN
323
324.. doxygenfunction:: suspend_iommu
325   :project: Project ACRN
326
327.. doxygenfunction:: resume_iommu
328   :project: Project ACRN
329
330.. doxygenfunction:: move_pt_device
331   :project: Project ACRN
332