1Dom0less
2========
3
4"Dom0less" is a set of Xen features that enable the deployment of a Xen
5system without an control domain (often referred to as "dom0"). Each
6feature can be used independently from the others, unless otherwise
7stated.
8
9Booting Multiple Domains from Device Tree
10-----------------------------------------
11
12This feature enables Xen to create a set of DomUs at boot time.
13Information about the DomUs to be created by Xen is passed to the
14hypervisor via Device Tree. Specifically, the existing Device Tree based
15Multiboot specification has been extended to allow for multiple domains
16to be passed to Xen. See docs/misc/arm/device-tree/booting.txt for more
17information about the Multiboot specification and how to use it.
18
19Currently, a control domain ("dom0") is still required to manage the DomU
20domains, but the system can start also without dom0 if the Device Tree
21doesn't specify the dom0 kernel and it declares one or more domUs.
22Instead of waiting for the control domain (when declared) to be fully
23booted and the Xen tools to become available, domains created by Xen
24this way are started right away in parallel. Hence, their boot time is
25typically much shorter.
26
27
28Configuration
29-------------
30
31### Loading binaries into memory ###
32
33U-Boot needs to load not just Xen, the device tree binary, the dom0 kernel and
34ramdisk. It also needs to load the kernel and ramdisk of any additional domains
35to boot. For example if this is the bootcmd for Xen and Dom0:
36
37    tftpb 0x1280000 xen.dtb
38    tftpb 0x0x80000 xen-Image
39    tftpb 0x1400000 xen.ub
40    tftpb 0x9000000 xen-rootfs.cpio.gz.u-boot
41
42    bootm 0x1400000 0x9000000 0x1280000
43
44If we want to add one DomU with Image-DomU as the DomU kernel
45and ramdisk-DomU as DomU ramdisk:
46
47    tftpb 0x1280000 xen.dtb
48    tftpb 0x80000 xen-Image
49    tftpb 0x1400000 xen.ub
50    tftpb 0x9000000 xen-rootfs.cpio.gz.u-boot
51
52    tftpb 0x2000000 Image-DomU
53    tftpb 0x3000000 ramdisk-DomU
54
55    bootm 0x1400000 0x9000000 0x1280000
56
57
58### Device Tree configuration ###
59
60In addition to loading the necessary binaries, we also need to advertise
61the presence of the additional VM and its configuration. It is done via
62device tree adding a node under /chosen as follows:
63
64    domU1 {
65        #address-cells = <1>;
66        #size-cells = <1>;
67        compatible = "xen,domain";
68        memory = <0 0x20000>;
69        cpus = <1>;
70        vpl011;
71
72        module@2000000 {
73            compatible = "multiboot,kernel", "multiboot,module";
74            reg = <0x2000000 0xffffff>;
75            bootargs = "console=ttyAMA0";
76        };
77
78        module@30000000 {
79            compatible = "multiboot,ramdisk", "multiboot,module";
80            reg = <0x3000000 0xffffff>;
81        };
82    };
83
84Where memory is the memory of the VM in KBs, cpus is the number of
85cpus. module@2000000 and module@3000000 advertise where the kernel and
86ramdisk are in memory.
87
88Note: the size specified should exactly match the size of the Kernel/initramfs.
89Otherwise, they may be unusable in Xen (for instance if they are compressed).
90
91See docs/misc/arm/device-tree/booting.txt for more information.
92
93PV Drivers
94----------
95
96It is possible to use PV drivers with dom0less guests with some
97restrictions:
98
99- dom0less domUs that want to use PV drivers support should have the
100  "xen,enhanced" property set under their device tree nodes (see
101  docs/misc/arm/device-tree/booting.txt)
102- a dom0 must be present (or another domain with enough privileges to
103  run the toolstack)
104- after dom0 is booted, the utility "init-dom0less" must be run
105- do not run "init-dom0less" while creating other guests with xl
106
107After the execution of init-dom0less, it is possible to use "xl" to
108hotplug PV drivers to dom0less guests. E.g. xl network-attach domU.
109
110The implementation works as follows:
111- Xen allocates the xenstore event channel for each dom0less domU that
112  has the "xen,enhanced" property, and sets HVM_PARAM_STORE_EVTCHN
113- Xen allocates the xenstore page and sets HVM_PARAM_STORE_PFN as well
114  as the connection status to XENSTORE_RECONNECT.
115- Dom0less domU kernels check that HVM_PARAM_STORE_PFN is set to
116  ~0ULL (invalid) or the connection status is *not* XENSTORE_CONNECTED.
117    - Old kernels will continue without xenstore support (Note: some old
118      buggy kernels might crash because they don't check the validity of
119      HVM_PARAM_STORE_PFN before using it! Disable "xen,enhanced" in
120      those cases)
121    - New kernels will wait for a notification on the xenstore event
122      channel (HVM_PARAM_STORE_EVTCHN) before continuing with the
123      initialization
124- Once dom0 is booted, init-dom0less is executed:
125    - it gets the xenstore shared page from HVM_PARAM_STORE_PFN
126    - it calls xs_introduce_domain
127- Xenstored notices the new domain, initializes interfaces as usual, and
128  sends an event channel notification to the domain using the xenstore
129  event channel (HVM_PARAM_STORE_EVTCHN)
130- The Linux domU kernel receives the event channel notification, checks
131  HVM_PARAM_STORE_PFN and the connection status again and continue with
132  the initialization
133
134
135Limitations
136-----------
137
138Domains started by Xen at boot time currently have the following
139limitations:
140
141- They cannot be properly shutdown or rebooted using xl. If one of them
142  crashes, the whole platform should be rebooted.
143
144- Some xl operations might not work as expected. xl is meant to be used
145  with domains that have been created by it. Using xl with domains
146  started by Xen at boot might not work as expected.
147
148- The GIC version is the native version. In absence of other
149  information, the GIC version exposed to the domains started by Xen at
150  boot is the same as the native GIC version.
151
152- Pinning vCPUs of domains started by Xen at boot can be
153  done from the control domain, using `xl vcpu-pin` as usual. It is not
154  currently possible to configure vCPU pinning without a control domain.
155  However, the NULL scheduler can be selected by passing `sched=null` to
156  the Xen command line. The NULL scheduler automatically assigns and
157  pins vCPUs to pCPUs, but the vCPU-pCPU assignments cannot be
158  configured.
159
160Notes
161-----
162
163- 'xl console' command will not attach to the domain's console in case
164  of dom0less. DomU are domains created by Xen (similar to Dom0) and
165  therefore they are all managed by Xen and some of the commands may not work.
166
167  A user is allowed to configure the key sequence to switch input.
168  Pressing the Xen "conswitch" (Ctrl-A by default) three times
169  switches input in case of dom0less mode.
170
171- Domains created by Xen will have no name at boot. Domain-0 has a name
172  thanks to the helper xen-init-dom0 called at boot by the initscript.
173  If you want to setup DomU name, then you will have to create the xenstore
174  node associated. By default DomU names are shown as '(null)' in the
175  xl domains list.
176