1.. _hld-io-emulation: 2 3I/O Emulation High-Level Design 4############################### 5 6As discussed in :ref:`intro-io-emulation`, there are multiple ways and 7places to handle I/O emulation, including HV, Service VM Kernel HSM, and Service VM 8user-land Device Model (acrn-dm). 9 10I/O emulation in the hypervisor provides these functionalities: 11 12- Maintain lists of port I/O or MMIO handlers in the hypervisor for 13 emulating trapped I/O accesses in a certain range. 14 15- Forward I/O accesses to Service VM when they cannot be handled by the 16 hypervisor by any registered handlers. 17 18:numref:`io-control-flow` illustrates the main control flow steps of I/O emulation 19inside the hypervisor: 20 211. Trap and decode I/O access by VM exits and decode the access from 22 exit qualification or by invoking the instruction decoder. 23 242. If the range of the I/O access overlaps with any registered handler, 25 call that handler if it completely covers the range of the 26 access, or ignore the access if the access crosses the boundary. 27 283. If the range of the I/O access does not overlap the range of any I/O 29 handler, deliver an I/O request to Service VM. 30 31.. figure:: images/ioem-image101.png 32 :align: center 33 :name: io-control-flow 34 35 Control Flow of I/O Emulation in the Hypervisor 36 37I/O emulation does not rely on any calibration data. 38 39Trap Path 40********* 41 42Port I/O accesses are trapped by VM exits with the basic exit reason 43"I/O instruction". The port address to be accessed, size, and direction 44(read or write) are fetched from the VM exit qualification. For writes 45the value to be written to the I/O port is fetched from guest registers 46al, ax or eax, depending on the access size. 47 48MMIO accesses are trapped by VM exits with the basic exit reason "EPT 49violation". The instruction emulator is invoked to decode the 50instruction that triggers the VM exit to get the memory address being 51accessed, size, direction (read or write), and the involved register. 52 53The I/O bitmaps and EPT are used to configure the addresses that will 54trigger VM exits when accessed by a VM. Refer to 55:ref:`io-mmio-emulation` for details. 56 57I/O Emulation in the Hypervisor 58******************************* 59 60When a port I/O or MMIO access is trapped, the hypervisor first checks 61whether the to-be-accessed address falls in the range of any registered 62handler, and calls the handler when such a handler exists. 63 64Handler Management 65================== 66 67Each VM has two lists of I/O handlers, one for port I/O and the other 68for MMIO. Each element of the list contains a memory range and a pointer 69to the handler which emulates the accesses falling in the range. See 70:ref:`io-handler-init` for descriptions of the related data structures. 71 72The I/O handlers are registered on VM creation and never changed until 73the destruction of that VM, when the handlers are unregistered. If 74multiple handlers are registered for the same address, the one 75registered later wins. See :ref:`io-handler-init` for the interfaces 76used to register and unregister I/O handlers. 77 78I/O Dispatching 79=============== 80 81When a port I/O or MMIO access is trapped, the hypervisor first walks 82through the corresponding I/O handler list in the reverse order of 83registration, looking for a proper handler to emulate the access. The 84following cases exist: 85 86- If a handler whose range overlaps the range of the I/O access is 87 found, 88 89 - If the range of the I/O access falls completely in the range the 90 handler can emulate, that handler is called. 91 92 - Otherwise it is implied that the access crosses the boundary of 93 multiple devices which the hypervisor does not emulate. Thus 94 no handler is called and no I/O request will be delivered to 95 Service VM. I/O reads get all 1's and I/O writes are dropped. 96 97- If the range of the I/O access does not overlap with any range of the 98 handlers, the I/O access is delivered to Service VM as an I/O request 99 for further processing. 100 101I/O Requests 102************ 103 104An I/O request is delivered to Service VM vCPU 0 if the hypervisor does not 105find any handler that overlaps the range of a trapped I/O access. This 106section describes the initialization of the I/O request mechanism and 107how an I/O access is emulated via I/O requests in the hypervisor. 108 109Initialization 110============== 111 112For each User VM the hypervisor shares a page with Service VM to exchange I/O 113requests. The 4-KByte page consists of 16 256-Byte slots, indexed by 114vCPU ID. It is required for the DM to allocate and set up the request 115buffer on VM creation, otherwise I/O accesses from User VM cannot be 116emulated by Service VM, and all I/O accesses not handled by the I/O handlers in 117the hypervisor will be dropped (reads get all 1's). 118 119Refer to the following sections for details on I/O requests and the 120initialization of the I/O request buffer. 121 122Types of I/O Requests 123===================== 124 125There are four types of I/O requests: 126 127.. list-table:: 128 :widths: 50 50 129 :header-rows: 1 130 131 * - I/O Request Type 132 - Description 133 134 * - PIO 135 - A port I/O access. 136 137 * - MMIO 138 - An MMIO access to a GPA with no mapping in EPT. 139 140 * - PCI 141 - A PCI configuration space access. 142 143 * - WP 144 - An MMIO access to a GPA with a read-only mapping in EPT. 145 146 147For port I/O accesses, the hypervisor will always deliver an I/O request 148of type PIO to Service VM. For MMIO accesses, the hypervisor will deliver an 149I/O request of either MMIO or WP, depending on the mapping of the 150accessed address (in GPA) in the EPT of the vCPU. The hypervisor will 151never deliver any I/O request of type PCI, but will handle such I/O 152requests in the same ways as port I/O accesses on their completion. 153 154Refer to :ref:`io-structs-interfaces` for a detailed description of the 155data held by each type of I/O request. 156 157I/O Request State Transitions 158============================= 159 160Each slot in the I/O request buffer is managed by a finite state machine 161with four states. The following figure illustrates the state transitions 162and the events that trigger them. 163 164.. figure:: images/ioem-image92.png 165 :align: center 166 167 State Transition of I/O Requests 168 169The four states are: 170 171FREE 172 The I/O request slot is not used and new I/O requests can be 173 delivered. This is the initial state on User VM creation. 174 175PENDING 176 The I/O request slot is occupied with an I/O request pending 177 to be processed by Service VM. 178 179PROCESSING 180 The I/O request has been dispatched to a client but the 181 client has not finished handling it yet. 182 183COMPLETE 184 The client has completed the I/O request but the hypervisor 185 has not consumed the results yet. 186 187The contents of an I/O request slot are owned by the hypervisor when the 188state of an I/O request slot is FREE or COMPLETE. In such cases Service VM can 189only access the state of that slot. Similarly the contents are owned by 190Service VM when the state is PENDING or PROCESSING, when the hypervisor can 191only access the state of that slot. 192 193The states are transferred as follow: 194 1951. To deliver an I/O request, the hypervisor takes the slot 196 corresponding to the vCPU triggering the I/O access, fills the 197 contents, changes the state to PENDING and notifies Service VM via 198 upcall. 199 2002. On upcalls, Service VM dispatches each I/O request in the PENDING state to 201 clients and changes the state to PROCESSING. 202 2033. The client assigned an I/O request changes the state to COMPLETE 204 after it completes the emulation of the I/O request. A hypercall 205 is made to notify the hypervisor on I/O request completion after 206 the state change. 207 2084. The hypervisor finishes the post-work of a I/O request after it is 209 notified on its completion and change the state back to FREE. 210 211States are accessed using atomic operations to avoid getting unexpected 212states on one core when it is written on another. 213 214Note that there is no state to represent a 'failed' I/O request. Service VM 215should return all 1's for reads and ignore writes whenever it cannot 216handle the I/O request, and change the state of the request to COMPLETE. 217 218Post-Work 219========= 220 221After an I/O request is completed, some more work needs to be done for 222I/O reads to update guest registers accordingly. The 223hypervisor re-enters the vCPU thread every time a vCPU is scheduled back 224in, rather than switching to where the vCPU is scheduled out. As a result, 225post-work is introduced for this purpose. 226 227The hypervisor pauses a vCPU before an I/O request is delivered to Service VM. 228Once the I/O request emulation is completed, a client notifies the 229hypervisor by a hypercall. The hypervisor will pick up that request, do 230the post-work, and resume the guest vCPU. The post-work takes care of 231updating the vCPU guest state to reflect the effect of the I/O reads. 232 233.. figure:: images/ioem-image100.png 234 :align: center 235 236 Workflow of MMIO I/O Request Completion 237 238The figure above illustrates the workflow to complete an I/O 239request for MMIO. Once the I/O request is completed, Service VM makes a 240hypercall to notify the hypervisor which resumes the User VM vCPU triggering 241the access after requesting post-work on that vCPU. After the User VM vCPU 242resumes, it does the post-work first to update the guest registers if 243the access reads an address, changes the state of the corresponding I/O 244request slot to FREE, and continues execution of the vCPU. 245 246.. figure:: images/ioem-image106.png 247 :align: center 248 :name: port-io-completion 249 250 Workflow of Port I/O Request Completion 251 252Completion of a port I/O request (shown in :numref:`port-io-completion` 253above) is 254similar to the MMIO case, except the post-work is done before resuming 255the vCPU. This is because the post-work for port I/O reads need to update 256the general register eax of the vCPU, while the post-work for MMIO reads 257need further emulation of the trapped instruction. This is much more 258complex and may impact the performance of the Service VM. 259 260.. _io-structs-interfaces: 261 262Data Structures and Interfaces 263****************************** 264 265External Interfaces 266=================== 267 268The following structures represent an I/O request. *struct acrn_io_request* 269is the main structure and the others are detailed representations of I/O 270requests of different kinds. 271 272.. doxygenstruct:: acrn_mmio_request 273 :project: Project ACRN 274 275.. doxygenstruct:: acrn_pio_request 276 :project: Project ACRN 277 278.. doxygenstruct:: acrn_pci_request 279 :project: Project ACRN 280 281.. doxygenstruct:: acrn_io_request 282 :project: Project ACRN 283 284For hypercalls related to I/O emulation, refer to `I/O Emulation in the Hypervisor`_. 285 286.. _io-handler-init: 287 288Initialization and Deinitialization 289=================================== 290 291The following structure represents a port I/O handler: 292 293.. doxygenstruct:: vm_io_handler_desc 294 :project: Project ACRN 295 296The following structure represents an MMIO handler. 297 298.. doxygenstruct:: mem_io_node 299 :project: Project ACRN 300 301The following APIs are provided to initialize, deinitialize or configure 302I/O bitmaps and register or unregister I/O handlers: 303 304.. doxygenfunction:: allow_guest_pio_access 305 :project: Project ACRN 306 307.. doxygenfunction:: register_pio_emulation_handler 308 :project: Project ACRN 309 310.. doxygenfunction:: register_mmio_emulation_handler 311 :project: Project ACRN 312 313I/O Emulation 314============= 315 316The following APIs are provided for I/O emulation at runtime: 317 318.. doxygenfunction:: acrn_insert_request 319 :project: Project ACRN 320 321.. doxygenfunction:: pio_instr_vmexit_handler 322 :project: Project ACRN 323 324.. doxygenfunction:: ept_violation_vmexit_handler 325 :project: Project ACRN 326
