1.. SPDX-License-Identifier: GPL-2.0-only
2.. include:: <isonum.txt>
3
4=====================
5VFIO Mediated devices
6=====================
7
8:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
9:Author: Neo Jia <cjia@nvidia.com>
10:Author: Kirti Wankhede <kwankhede@nvidia.com>
11
12
13
14Virtual Function I/O (VFIO) Mediated devices[1]
15===============================================
16
17The number of use cases for virtualizing DMA devices that do not have built-in
18SR_IOV capability is increasing. Previously, to virtualize such devices,
19developers had to create their own management interfaces and APIs, and then
20integrate them with user space software. To simplify integration with user space
21software, we have identified common requirements and a unified management
22interface for such devices.
23
24The VFIO driver framework provides unified APIs for direct device access. It is
25an IOMMU/device-agnostic framework for exposing direct device access to user
26space in a secure, IOMMU-protected environment. This framework is used for
27multiple devices, such as GPUs, network adapters, and compute accelerators. With
28direct device access, virtual machines or user space applications have direct
29access to the physical device. This framework is reused for mediated devices.
30
31The mediated core driver provides a common interface for mediated device
32management that can be used by drivers of different devices. This module
33provides a generic interface to perform these operations:
34
35* Create and destroy a mediated device
36* Add a mediated device to and remove it from a mediated bus driver
37* Add a mediated device to and remove it from an IOMMU group
38
39The mediated core driver also provides an interface to register a bus driver.
40For example, the mediated VFIO mdev driver is designed for mediated devices and
41supports VFIO APIs. The mediated bus driver adds a mediated device to and
42removes it from a VFIO group.
43
44The following high-level block diagram shows the main components and interfaces
45in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
46devices as examples, as these devices are the first devices to use this module::
47
48     +---------------+
49     |               |
50     | +-----------+ |  mdev_register_driver() +--------------+
51     | |           | +<------------------------+              |
52     | |  mdev     | |                         |              |
53     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
54     | |  driver   | |     probe()/remove()    |              |    APIs
55     | |           | |                         +--------------+
56     | +-----------+ |
57     |               |
58     |  MDEV CORE    |
59     |   MODULE      |
60     |   mdev.ko     |
61     | +-----------+ |  mdev_register_parent() +--------------+
62     | |           | +<------------------------+              |
63     | |           | |                         | ccw_device.ko|<-> physical
64     | |           | +------------------------>+              |    device
65     | |           | |        callbacks        +--------------+
66     | | Physical  | |
67     | |  device   | |  mdev_register_parent() +--------------+
68     | | interface | |<------------------------+              |
69     | |           | |                         |  i915.ko     |<-> physical
70     | |           | +------------------------>+              |    device
71     | |           | |        callbacks        +--------------+
72     | +-----------+ |
73     +---------------+
74
75
76Registration Interfaces
77=======================
78
79The mediated core driver provides the following types of registration
80interfaces:
81
82* Registration interface for a mediated bus driver
83* Physical device driver interface
84
85Registration Interface for a Mediated Bus Driver
86------------------------------------------------
87
88The registration interface for a mediated device driver provides the following
89structure to represent a mediated device's driver::
90
91     /*
92      * struct mdev_driver [2] - Mediated device's driver
93      * @probe: called when new device created
94      * @remove: called when device removed
95      * @driver: device driver structure
96      */
97     struct mdev_driver {
98	     int  (*probe)  (struct mdev_device *dev);
99	     void (*remove) (struct mdev_device *dev);
100	     unsigned int (*get_available)(struct mdev_type *mtype);
101	     ssize_t (*show_description)(struct mdev_type *mtype, char *buf);
102	     struct device_driver    driver;
103     };
104
105A mediated bus driver for mdev should use this structure in the function calls
106to register and unregister itself with the core driver:
107
108* Register::
109
110    int mdev_register_driver(struct mdev_driver *drv);
111
112* Unregister::
113
114    void mdev_unregister_driver(struct mdev_driver *drv);
115
116The mediated bus driver's probe function should create a vfio_device on top of
117the mdev_device and connect it to an appropriate implementation of
118vfio_device_ops.
119
120When a driver wants to add the GUID creation sysfs to an existing device it has
121probe'd to then it should call::
122
123    int mdev_register_parent(struct mdev_parent *parent, struct device *dev,
124			struct mdev_driver *mdev_driver);
125
126This will provide the 'mdev_supported_types/XX/create' files which can then be
127used to trigger the creation of a mdev_device. The created mdev_device will be
128attached to the specified driver.
129
130When the driver needs to remove itself it calls::
131
132    void mdev_unregister_parent(struct mdev_parent *parent);
133
134Which will unbind and destroy all the created mdevs and remove the sysfs files.
135
136Mediated Device Management Interface Through sysfs
137==================================================
138
139The management interface through sysfs enables user space software, such as
140libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
141This management interface provides flexibility to the underlying physical
142device's driver to support features such as:
143
144* Mediated device hot plug
145* Multiple mediated devices in a single virtual machine
146* Multiple mediated devices from different physical devices
147
148Links in the mdev_bus Class Directory
149-------------------------------------
150The /sys/class/mdev_bus/ directory contains links to devices that are registered
151with the mdev core driver.
152
153Directories and files under the sysfs for Each Physical Device
154--------------------------------------------------------------
155
156::
157
158  |- [parent physical device]
159  |--- Vendor-specific-attributes [optional]
160  |--- [mdev_supported_types]
161  |     |--- [<type-id>]
162  |     |   |--- create
163  |     |   |--- name
164  |     |   |--- available_instances
165  |     |   |--- device_api
166  |     |   |--- description
167  |     |   |--- [devices]
168  |     |--- [<type-id>]
169  |     |   |--- create
170  |     |   |--- name
171  |     |   |--- available_instances
172  |     |   |--- device_api
173  |     |   |--- description
174  |     |   |--- [devices]
175  |     |--- [<type-id>]
176  |          |--- create
177  |          |--- name
178  |          |--- available_instances
179  |          |--- device_api
180  |          |--- description
181  |          |--- [devices]
182
183* [mdev_supported_types]
184
185  The list of currently supported mediated device types and their details.
186
187  [<type-id>], device_api, and available_instances are mandatory attributes
188  that should be provided by vendor driver.
189
190* [<type-id>]
191
192  The [<type-id>] name is created by adding the device driver string as a prefix
193  to the string provided by the vendor driver. This format of this name is as
194  follows::
195
196	sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
197
198* device_api
199
200  This attribute shows which device API is being created, for example,
201  "vfio-pci" for a PCI device.
202
203* available_instances
204
205  This attribute shows the number of devices of type <type-id> that can be
206  created.
207
208* [device]
209
210  This directory contains links to the devices of type <type-id> that have been
211  created.
212
213* name
214
215  This attribute shows a human readable name.
216
217* description
218
219  This attribute can show brief features/description of the type. This is an
220  optional attribute.
221
222Directories and Files Under the sysfs for Each mdev Device
223----------------------------------------------------------
224
225::
226
227  |- [parent phy device]
228  |--- [$MDEV_UUID]
229         |--- remove
230         |--- mdev_type {link to its type}
231         |--- vendor-specific-attributes [optional]
232
233* remove (write only)
234
235Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
236fail the remove() callback if that device is active and the vendor driver
237doesn't support hot unplug.
238
239Example::
240
241	# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
242
243Mediated device Hot plug
244------------------------
245
246Mediated devices can be created and assigned at runtime. The procedure to hot
247plug a mediated device is the same as the procedure to hot plug a PCI device.
248
249Translation APIs for Mediated Devices
250=====================================
251
252The following APIs are provided for translating user pfn to host pfn in a VFIO
253driver::
254
255	int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
256				  int npage, int prot, struct page **pages);
257
258	void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,
259				    int npage);
260
261These functions call back into the back-end IOMMU module by using the pin_pages
262and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
263these callbacks are supported in the TYPE1 IOMMU module. To enable them for
264other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
265these two callback functions.
266
267References
268==========
269
2701. See Documentation/driver-api/vfio.rst for more information on VFIO.
2712. struct mdev_driver in include/linux/mdev.h
2723. struct mdev_parent_ops in include/linux/mdev.h
2734. struct vfio_iommu_driver_ops in include/linux/vfio.h
274