xref: /docs/misc/arm/device-tree/booting.txt

Dom0 kernel and ramdisk modules
================================

Xen is passed the dom0 kernel and initrd via a reference in the /chosen
node of the device tree.

Each node contains the following properties:

- compatible

	Must always include at least the generic compatiblity string:

		"multiboot,module"

	Optionally a more specific compatible string may be used in
	addition to the above. One of:

	- "multiboot,kernel"	-- the dom0 kernel
	- "multiboot,ramdisk"	-- the dom0 ramdisk
	- "xen,xsm-policy"	-- XSM policy blob

	It is normally recommended to include a more specific
	compatible string (if one applies) in addition to the generic
	string (which must always be present).

	Xen will assume that the first module which lacks a more
	specific compatible string is a "multiboot,kernel".

	Xen will examine each module, starting from the second
	module that lacks a specific compatible string.  Xen will
	check each such module for the XSM Magic number:

	- For a module which has the XSM Magic number: it will be
	  treated by Xen as if its compatible string was
	  "xen,xsm-policy";

	- For a module which does not have the XSM Magic: the second
	  module lacking a compatible string will be treated by Xen as
	  if its compatible string was "multiboot,ramdisk"; for the
	  third and subsequent modules which lack a specific
	  compatible string, Xen will not apply any special treatment.

	This means if the ramdisk module is present and does not have the
	compatible string "multiboot,ramdisk", then it must always be the
	second module.

	Note: This XSM Magic detection behavior was introduced by Xen 4.7.
	Xen 4.6 (and downwards) still requires the XSM module to have the
	compatible string "xen,xsm-policy".

	Xen 4.4 supported a different set of legacy compatible strings
	which remain supported such that systems supporting both 4.4
	and later can use a single DTB.
	However when booting Xen using UEFI, the legacy compatible
	strings are not supported.

	- "xen,multiboot-module" equivalent to "multiboot,module"
	- "xen,linux-zimage"     equivalent to "multiboot,kernel"
	- "xen,linux-initrd"     equivalent to "multiboot,ramdisk"

	For compatibility with Xen 4.4 the more specific "xen,linux-*"
	names are non-optional and must be included.

- reg

	Specifies the physical address of the module in RAM and the
	length of the module.

- bootargs (optional)

	Command line associated with this module. See below for the
	priority of this field vs. other mechanisms of specifying the
	bootargs for the kernel.

- xen,uefi-binary (UEFI boot only)

	String property that specifies the file name to be loaded by the UEFI
	boot for this module. If this is specified, there is no need to specify
	the reg property because it will be created by the UEFI stub on boot.
	This option is needed only when UEFI boot is used, the node needs to be
	compatible with multiboot,kernel or multiboot,ramdisk.

Examples
========

A boot module of unspecified type:

	module@0xc0000000 {
		compatible = "multiboot,module";
		reg = <0xc0000000 0x1234>;
		bootargs = "...";
	};

A boot module containing a ramdisk:

	module@0xd0000000 {
		compatible = "multiboot,ramdisk", "multiboot,module";
		reg = <0xd0000000 0x5678>;
	};

The previous examples are compatible with Xen 4.5+ only.

To be compatible with Xen 4.4 as well use the legacy names:

	module@0xd0000000 {
		compatible = "xen,linux-initrd", "xen,multiboot-module";
		reg = <0xd0000000 0x5678>;
	};

Command lines
=============

Xen also checks for properties directly under /chosen to find suitable command
lines for Xen and Dom0. The logic is the following:

 - If xen,xen-bootargs is present, it will be used for Xen.
 - If xen,dom0-bootargs is present, it will be used for Dom0.
 - If xen,xen-bootargs is _not_ present, but xen,dom0-bootargs is,
   bootargs will be used for Xen.
 - If a kernel boot module is present and has a bootargs property then
   the top-level bootargs will used for Xen.
 - If no Xen specific properties are present, bootargs is for Dom0.
 - If xen,xen-bootargs is present, but xen,dom0-bootargs is missing,
   bootargs will be used for Dom0.

Most of these cases is to make booting with Xen-unaware bootloaders easier.
For those you would hardcode the Xen commandline in the DTB under
/chosen/xen,xen-bootargs and would let the bootloader set the Dom0 command
line by writing bootargs (as for native Linux).
A Xen-aware bootloader would set xen,xen-bootargs for Xen, xen,dom0-bootargs
for Dom0 and bootargs for native Linux.


UEFI boot and DT
================

When Xen is booted using UEFI, it doesn't read the configuration file if any
multiboot module is specified. To force Xen to load the configuration file, the
boolean property xen,uefi-cfg-load must be declared in the /chosen node.


Creating Multiple Domains directly from Xen
===========================================

It is possible to have Xen create other domains, in addition to dom0,
out of the information provided via device tree. A kernel and initrd
(optional) need to be specified for each guest.

For each domain to be created there needs to be one node under /chosen
with the following properties:

- compatible

    For domUs: "xen,domain"

- memory

	A 64-bit integer specifying the amount of kilobytes of RAM to
    allocate to the guest.

- cpus

    An integer specifying the number of vcpus to allocate to the guest.

- llc-colors
    A string specifying the LLC color configuration for the guest.
    Refer to docs/misc/cache_coloring.rst for syntax. This option is applicable
    only to Arm64 guests.

- capabilities
    Optional.  A 32-bit integer representing a bit field of domain capabilities
    for a disaggregated system.  A traditional dom0 has all of these
    capabilities, and a domU has none of them.

    0x1 DOMAIN_CAPS_CONTROL  - A privileged, control domain
    0x2 DOMAIN_CAPS_HARDWARE - The hardware domain - there can be only 1
    0x4 DOMAIN_CAPS_XENSTORE - The xenstore domain - there can be only 1

    The default is no capabilities.

- vpl011

    An empty property to enable/disable a virtual pl011 for the guest to
    use. The virtual pl011 uses SPI number 0 (see GUEST_VPL011_SPI).
    Please note that the SPI used for the virtual pl011 could clash with the
    physical SPI of a physical device assigned to the guest.

- nr_spis

    Optional. A 32-bit integer specifying the number of SPIs (Shared
    Peripheral Interrupts) to allocate for the domain. If nr_spis is
    missing, the max number of SPIs supported by the physical GIC is
    used, or GUEST_VPL011_SPI+1 if vpl011 is enabled, whichever is
    greater.

- #address-cells and #size-cells

    Both #address-cells and #size-cells need to be specified because
    both sub-nodes (described shortly) have reg properties.

- direct-map

    Only available when statically allocated memory is used for the domain.
    An empty property to request the memory of the domain to be
    direct-map (guest physical address == physical address).

- domain-cpupool

    Optional. Handle to a xen,cpupool device tree node that identifies the
    cpupool where the guest will be started at boot.

- sve

    Optional. The `sve` property enables Arm SVE usage for the domain and sets
    the maximum SVE vector length, the option is applicable only to Arm64
    guests.
    A value equal to 0 disables the feature, this is the default value.
    Specifying this property with no value, means that the SVE vector length
    will be set equal to the maximum vector length supported by the platform.
    Values above 0 explicitly set the maximum SVE vector length for the domain,
    allowed values are from 128 to maximum 2048, being multiple of 128.
    Please note that when the user explicitly specifies the value, if that value
    is above the hardware supported maximum SVE vector length, the domain
    creation will fail and the system will stop, the same will occur if the
    option is provided with a non zero value, but the platform doesn't support
    SVE.

- trap-unmapped-accesses

    Optional. An integer that configures handling of accesses to unmapped
    address ranges.
    If set to 0, guest accesses will read all bits as ones, e.g 0xFFFFFFFF
    for a 32bit access and writes will be ignored.
    If set to 1, guest accesses will trap.

    The default is 1 when trap-unmapped-accesses is absent.

- xen,enhanced

    A string property. Possible property values are:

    - "enabled" (or missing property value)
    Xen PV interfaces, including grant-table and xenstore, will be
    enabled for the VM.

    - "legacy"
    Same as above, but the way the xenstore page is allocated is not
    compatible with static-mem guests. On the other hand, it works with
    older Linux kernels.

    - "disabled"
    Xen PV interfaces are disabled.

    - "no-xenstore"
    All default Xen PV interfaces, including grant-table will be enabled but
    xenstore will be disabled for the VM.

    If the xen,enhanced property is present with no value, it defaults
    to "enabled". If the xen,enhanced property is not present, PV
    interfaces are disabled.

    In the future other possible property values might be added to
    enable only selected interfaces.

- xen,domain-p2m-mem-mb

    Optional. A 32-bit integer specifying the amount of megabytes of RAM
    used for the domain P2M pool. This is in-sync with the shadow_memory
    option in xl.cfg. Leaving this field empty in device tree will lead to
    the default size of domain P2M pool, i.e. 1MB per guest vCPU plus 4KB
    per MB of guest RAM plus 512KB for guest extended regions.

- max_grant_version

    Optional. A 32-bit integer specifying the maximum grant table version
    the domain is allowed to use (valid values are 1 or 2). If this property
    is missing, the value specified by Xen command line parameter gnttab=max-ver
    (or its default value if unspecified, i.e. 1) is used.

- max_grant_frames

    Optional. A 32-bit integer specifying the maximum number of grant frames
    the domain is allowed to have. If this property is missing, the value
    specified by Xen command line parameter gnttab_max_frames (or its default
    value if unspecified, i.e. 64) is used.

- max_maptrack_frames

    Optional. A 32-bit integer specifying the maximum number of grant maptrack
    frames the domain is allowed to have. If this property is missing, the
    value specified by Xen command line parameter gnttab_max_maptrack_frames
    (or its default value if unspecified, i.e. 1024) is used.

- passthrough

    A string property specifying whether IOMMU mappings are enabled for the
    domain and hence whether it will be enabled for passthrough hardware.
    Possible property values are:

    - "enabled"
    IOMMU mappings are enabled for the domain. Note that this option is the
    default if the user provides the device partial passthrough device tree
    for the domain.

    - "disabled"
    IOMMU mappings are disabled for the domain and so hardware may not be
    passed through. This option is the default if this property is missing
    and the user does not provide the device partial device tree for the domain.

Under the "xen,domain" compatible node, one or more sub-nodes are present
for the DomU kernel and ramdisk.

The kernel sub-node has the following properties:

- compatible

    "multiboot,kernel", "multiboot,module"

- reg

    Specifies the physical address of the kernel in RAM and its
    length.

- bootargs (optional)

    Command line parameters for the guest kernel.

- xen,uefi-binary (UEFI boot only)

    String property that specifies the file name to be loaded by the UEFI boot
    for this module. If this is specified, there is no need to specify the reg
    property because it will be created by the UEFI stub on boot.
    This option is needed only when UEFI boot is used.

The ramdisk sub-node has the following properties:

- compatible

    "multiboot,ramdisk", "multiboot,module"

- reg

    Specifies the physical address of the ramdisk in RAM and its
    length.

- xen,uefi-binary (UEFI boot only)

    String property that specifies the file name to be loaded by the UEFI boot
    for this module. If this is specified, there is no need to specify the reg
    property because it will be created by the UEFI stub on boot.
    This option is needed only when UEFI boot is used.

Under the "xen,domain" compatible node, it is possible optionally to add
one or more sub-nodes to configure vCPU affinity. The vCPU affinity
sub-node has the following properties:

- compatible

    "xen,vcpu"

- id

    A 32-bit integer that specifies the vCPU id. 0 is the first vCPU.
    The last vCPU is cpus-1, where "cpus" is the number of vCPUs
    specified with the "cpus" property under the "xen,domain" node.
    Each "xen,vcpu" node must have a unique vCPU id.

- hard-affinity

    Optional. A string specifying the hard affinity configuration for the
    vCPU: a comma-separated list of pCPUs or ranges of pCPUs is used.
    Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive
    on both sides. The numbers refer to logical pCPU ids.


Example
=======

chosen {
    domU1 {
        compatible = "xen,domain";
        #address-cells = <0x2>;
        #size-cells = <0x1>;
        memory = <0 131072>;
        cpus = <2>;
        vpl011;

        vcpu0 {
            compatible = "xen,vcpu";
            id = <0>;
            hard-affinity = "0-3";
        };

        vcpu1 {
            compatible = "xen,vcpu";
            id = <1>;
            hard-affinity = "1,4-7";
        };

        module@0x4a000000 {
            compatible = "multiboot,kernel", "multiboot,module";
            reg = <0x0 0x4a000000 0xffffff>;
            bootargs = "console=ttyAMA0 init=/bin/sh";
        };

        module@0x4b000000 {
            compatible = "multiboot,ramdisk", "multiboot,module";
            reg = <0x0 0x4b000000 0xffffff>;
        };
    };

    domU2 {
        compatible = "xen,domain";
        #address-cells = <0x2>;
        #size-cells = <0x1>;
        memory = <0 65536>;
        cpus = <1>;

        module@0x4c000000 {
            compatible = "multiboot,kernel", "multiboot,module";
            reg = <0x0 0x4c000000 0xffffff>;
            bootargs = "console=ttyAMA0 init=/bin/sh";
        };

        module@0x4d000000 {
            compatible = "multiboot,ramdisk", "multiboot,module";
            reg = <0x0 0x4d000000 0xffffff>;
        };
    };
};


Device Assignment
=================

Device Assignment (Passthrough) is supported by adding another module,
alongside the kernel and ramdisk, with the device tree fragment
corresponding to the device node to assign to the guest.

The dtb sub-node should have the following properties:

- compatible

    "multiboot,device-tree" and "multiboot,module"

- reg

    Specifies the physical address of the device tree binary fragment
    RAM and its length.

- xen,uefi-binary (UEFI boot only)

    String property that specifies the file name to be loaded by the UEFI boot
    for this module. If this is specified, there is no need to specify the reg
    property because it will be created by the UEFI stub on boot.
    This option is needed only when UEFI boot is used.

As an example:

        module@0xc000000 {
            compatible = "multiboot,device-tree", "multiboot,module";
            reg = <0x0 0xc000000 0xffffff>;
        };

The DTB fragment is loaded at 0xc000000 in the example above. It should
follow the convention explained in docs/misc/arm/passthrough.txt. The
DTB fragment will be added to the guest device tree, so that the guest
kernel will be able to discover the device.


Static Allocation
=============

Static Allocation refers to system or sub-system(domains) for which memory
areas are pre-defined by configuration using physical address ranges.

Memory can be statically allocated to a domain using the property "xen,static-
mem" defined in the domain configuration. The number of cells for the address
and the size must be defined respectively by the parent node properties
"#address-cells" and "#size-cells".

The property 'memory' is still needed and should match the amount of memory
given to the guest. Currently, it either comes from static memory or lets Xen
allocate from heap. *Mixing* is not supported.

The static memory will be mapped in the guest at the usual guest memory
addresses (GUEST_RAM0_BASE, GUEST_RAM1_BASE) defined by
xen/include/public/arch-arm.h.

Below is an example on how to specify the static memory region in the
device-tree:

    / {
        chosen {
            #address-cells = <0x1>;
            #size-cells = <0x1>;
            ...
            domU1 {
                compatible = "xen,domain";
                cpus = <2>;
                memory = <0x0 0x80000>;
                xen,static-mem = <0x30000000 0x20000000>;
                ...
            };
        };
    };

This will reserve a 512MB region starting at the host physical address
0x30000000 to be exclusively used by DomU1.

Static Event Channel
====================
The event channel communication will be established statically between two
domains (dom0 and domU also). Event channel connection information between
domains will be passed to Xen via the device tree node. The event channel
will be created and established in Xen before the domain started. The domain
does not need to do any operation to establish a connection. Domain only
needs hypercall EVTCHNOP_send(local port) to send notifications to the
remote guest.

There is no need to describe the static event channel info in the domU device
tree. Static event channels are only useful in fully static configurations,
and in those configurations, the domU device tree dynamically generated by Xen
is not needed.

To enable the event-channel interface for domU guests include the
xen,enhanced = "no-xenstore" property in the domU Xen device tree node.

Under the "xen,domain" compatible node for domU, there needs to be sub-nodes
with compatible "xen,evtchn" that describe the event channel connection
between two domUs. For dom0, there needs to be sub-nodes with compatible
"xen,evtchn" under the chosen node.

The static event channel node has the following properties:

- compatible

    "xen,evtchn"

- xen,evtchn

    The property is tuples of two numbers
    (local-evtchn link-to-foreign-evtchn) where:

    local-evtchn is an integer value that will be used to allocate local port
    for a domain to send and receive event notifications to/from the remote
    domain. Maximum supported value is 2^17 for FIFO ABI and 4096 for 2L ABI.
    It is recommended to use low event channel IDs.

    link-to-foreign-evtchn is a single phandle to a remote evtchn to which
    local-evtchn will be connected.

Example
=======

chosen {

    /* One sub-node per local event channel. This sub-node is for Dom0. */
    ec1: evtchn@1 {
         compatible = "xen,evtchn-v1";
         /* local-evtchn link-to-foreign-evtchn */
         xen,evtchn = <0xa &ec2>;
    };

    domU1 {
        compatible = "xen,domain";
        #address-cells = <0x2>;
        #size-cells = <0x1>;
        xen,enhanced = "no-xenstore";

        /* One sub-node per local event channel */
        ec2: evtchn@2 {
            compatible = "xen,evtchn-v1";
            /* local-evtchn link-to-foreign-evtchn */
            xen,evtchn = <0xa &ec1>;
        };

        ec3: evtchn@3 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0xb &ec5>;
        };

        ec4: evtchn@4 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0xc &ec6>;
        };
    };

    domU2 {
        compatible = "xen,domain";
        #address-cells = <0x2>;
        #size-cells = <0x1>;
        xen,enhanced = "no-xenstore";

        /* One sub-node per local event channel */
        ec5: evtchn@5 {
            compatible = "xen,evtchn-v1";
            /* local-evtchn link-to-foreign-evtchn */
            xen,evtchn = <0xb &ec3>;
        };

        ec6: evtchn@6 {
            compatible = "xen,evtchn-v1";
            xen,evtchn = <0xd &ec4>;
        };
    };
};

Static Heap Memory
==================

The static heap memory refers to parts of RAM reserved in the beginning of
boot time for heap. The memory is reserved by configuration in the device
tree using physical address ranges.

The static heap memory declared in the device tree defines the memory areas
that will be reserved to be used exclusively as heap.

- For Arm32, since there are separated heaps, the static heap will be used
for both domheap and xenheap. The admin should make sure that the static
heap region should contain enough memory below 4GB to cater 32-bit DMA.

- For Arm64, since there is a single heap, the defined static heap areas
shall always go to the heap allocator.

The static heap memory is an optional feature and can be enabled by adding
below device tree property.

- xen,static-heap

    Property under the top-level "chosen" node. It specifies the address
    and size of Xen static heap memory. Number of address and size cells
    for the "xen,static-heap" property is determined by the root node
    "#address-cells" and "#size-cells". Note that at least a 64KB alignment
    is required.

Below is an example on how to specify the static heap in device tree:

    / {
        #address-cells = <0x2>;
        #size-cells = <0x2>;
        ...
        chosen {
            xen,static-heap = <0x0 0x30000000 0x0 0x40000000>;
            ...
        };
    };

RAM starting from the host physical address 0x30000000 of 1GB size will
be reserved as static heap.

Static Shared Memory
====================

The static shared memory device tree nodes allow users to statically set up
shared memory on dom0less system, enabling domains to do shm-based
communication.

- compatible

    "xen,domain-shared-memory-v1"

- xen,shm-id

    An arbitrary string that represents the unique identifier of the shared
    memory region, with a strict limit on the number of characters(\0 included),
    `MAX_SHM_ID_LENGTH(16)`. e.g. "xen,shm-id = "my-shared-mem-1"".

- xen,shared-mem

    An array takes a physical address, which is the base address of the
    shared memory region in host physical address space, a size, and a guest
    physical address, as the target address of the mapping.
    e.g. xen,shared-mem = < [host physical address] [guest address] [size] >;
    Note that if a domain is direct-mapped, i.e. the Dom0 and the Dom0less
    DomUs with `direct-map` device tree property, the static shared memory
    should also be direct-mapped (host physical address == guest address).

    It shall also meet the following criteria:
    1) If the SHM ID matches with an existing region, the address range of the
    region shall also exactly match.
    2) If the SHM ID does not match with any other existing region, it should
    also not overlap with any other regions.

    The number of cells for the host address (and size) is the same as the
    guest pseudo-physical address and they are inherited from the parent node.

    Host physical address is optional, when missing Xen decides the location.
    e.g. xen,shared-mem = < [guest address] [size] >;

- role (Optional)

    A string property specifying the ownership of a shared memory region,
    the value must be one of the following: "owner", or "borrower"
    A shared memory region could be explicitly backed by one domain, which is
    called "owner domain", and all the other domains who are also sharing
    this region are called "borrower domain".
    If not specified, the default value is "borrower" and owner is
    DOMID_IO, a system domain.

As an example:

chosen {
    #address-cells = <0x1>;
    #size-cells = <0x1>;
    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";

    ......

    /* this is for Dom0 */
    dom0-shared-mem@10000000 {
        compatible = "xen,domain-shared-memory-v1";
        role = "owner";
        xen,shm-id = "my-shared-mem-0";
        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
    };

    domU1 {
        compatible = "xen,domain";
        #address-cells = <0x1>;
        #size-cells = <0x1>;
        memory = <0 131072>;
        cpus = <2>;
        vpl011;

        /*
         * shared memory region "my-shared-mem-0" is shared
         * between Dom0 and DomU1.
         */
        domU1-shared-mem@10000000 {
            compatible = "xen,domain-shared-memory-v1";
            role = "borrower";
            xen,shm-id = "my-shared-mem-0";
            xen,shared-mem = <0x10000000 0x50000000 0x10000000>;
        };

        /*
         * shared memory region "my-shared-mem-1" is shared between
         * DomU1 and DomU2.
         */
        domU1-shared-mem@50000000 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = "my-shared-mem-1";
            xen,shared-mem = <0x50000000 0x60000000 0x20000000>;
        };

        /*
         * shared memory region "my-shared-mem-2" is shared between
         * DomU1 and DomU2.
         */
        domU1-shared-mem-2 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = "my-shared-mem-2";
            role = "owner";
            xen,shared-mem = <0x80000000 0x20000000>;
        };

        ......

    };

    domU2 {
        compatible = "xen,domain";
        #address-cells = <0x1>;
        #size-cells = <0x1>;
        memory = <0 65536>;
        cpus = <1>;

        /*
         * shared memory region "my-shared-mem-1" is shared between
         * domU1 and domU2.
         */
        domU2-shared-mem@50000000 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = "my-shared-mem-1";
            xen,shared-mem = <0x50000000 0x70000000 0x20000000>;
        };

        domU2-shared-mem-2 {
            compatible = "xen,domain-shared-memory-v1";
            xen,shm-id = "my-shared-mem-2";
            role = "borrower";
            xen,shared-mem = <0x90000000 0x20000000>;
        };

        ......
    };
};

This is an example with two static shared memory regions.

For the static shared memory region identified as "my-shared-mem-0", host
physical address starting at 0x10000000 of 256MB will be reserved to be
shared between Dom0 and DomU1. It will get mapped at 0x10000000 in Dom0 guest
physical address space, and at 0x50000000 in DomU1 guest physical address space.
Dom0 is explicitly defined as the owner domain, and DomU1 is the borrower domain.

For the static shared memory region identified as "my-shared-mem-1", host
physical address starting at 0x50000000 of 512MB will be reserved to be
shared between DomU1 and DomU2. It will get mapped at 0x60000000 in DomU1 guest
physical address space, and at 0x70000000 in DomU2 guest physical address space.
DomU1 and DomU2 are both the borrower domain, the owner domain is the default
owner domain DOMID_IO.

For the static shared memory region "my-shared-mem-2", since host physical
address is not provided by user, Xen will automatically allocate 512MB
from heap as static shared memory to be shared between DomU1 and DomU2.
The automatically allocated static shared memory will get mapped at
0x80000000 in DomU1 guest physical address space, and at 0x90000000 in DomU2
guest physical address space. DomU1 is explicitly defined as the owner domain,
and DomU2 is the borrower domain.