1.. SPDX-License-Identifier: BSD-3-Clause 2.. SPDX-FileCopyrightText: Copyright TF-RMM Contributors. 3 4############### 5EL0 apps in RMM 6############### 7 8**** 9Goal 10**** 11 12The goal of introducing EL0 app support in TF-RMM is to improve security by 13minimizing the amount of code running in privileged mode. This is achieved by 14deprivileging parts of RMM to run at EL0 using the Virtualization Host 15Extensions (VHE). This approach is especially desirable for code pulled in from 16external projects (like Mbed TLS), as such components are often complex and 17sensitive. RMM apps now run at EL0 using the EL2&0 translation regime, with the 18CPU configured such that an eret instruction in EL2 transitions directly to EL0, 19and the target EL of an SVC instruction is EL2. Additionally, each EL0 app has 20its own isolated address space, which not only improves fault isolation but also 21helps mitigate speculation-based side channels, allowing secrets within apps to 22be better protected from speculative execution. This isolation mechanism also 23lays the groundwork for future side-channel mitigations and facilitates their 24deployment. Overall, this scheme is especially useful for sand-boxing complex 25and sensitive code at EL0 and increasing the robustness of RMM. 26 27|RMM app EL0| 28 29This design document covers the various aspects of the implementation of the EL0 30app support in RMM. 31 32 33********************* 34Building app binaries 35********************* 36 37The RMM app's code is in the ``app`` directory, and they may link to 38libraries from the ``lib`` directory. RMM apps use a common linker script that 39contains a few predefined section. The RMM app cmake file first compiles and 40links the RMM app elf binary. Then the python script ``app/gen_app_bin.py`` 41is called that extracts the section contents along with address and size 42information. Then this information is used to generate a bin file. The bin file 43is a concatenation of the following: 44 45+----------------+-------------------------------------+ 46| 1 page size of | 8 bytes padding (explained later) | 47| header +-------------------------------------+ 48| | magic | 49| +-------------------------------------+ 50| | header version | 51| +-------------------------------------+ 52| | app_name | 53| +-------------------------------------+ 54| | app id | 55| +-------------------------------------+ 56| | app_len | 57| +-------------------------------------+ 58| | section 1 offset | 59| +-------------------------------------+ 60| | section 1 size | 61| +-------------------------------------+ 62| | ... | 63| +-------------------------------------+ 64| | section n offset | 65| +-------------------------------------+ 66| | section n size | 67| +-------------------------------------+ 68| | magic | 69| +-------------------------------------+ 70| | Padding | 71+----------------+-------------------------------------+ 72| content of section 1 | 73+------------------------------------------------------+ 74| ... | 75+------------------------------------------------------+ 76| content of section n | 77+------------------------------------------------------+ 78 79The number and order of sections in an RMM app elf/binary is hardcoded in the 80linker script, the python script that does the extraction of section content, 81and in the header definition. 82 83The RMM app bin size is padded to be a multiple of page size. 84 85************************ 86Building final RMM image 87************************ 88 89The RMM core code is built and linked using its own linker script. Then objcopy 90is called on the resulting elf to extract the sections to an img file. Then the 91bin files for the RMM apps are built as described above. The final RMM image 92is created concatenating the following elements by the python script 93``app/bundle_app_rmm.py``: 94 95* The first RMM app bin. The first 8 byte padding of the header is updated to a 96 ``BL`` (branch with link) instruction, followed by 4 bytes of padding. This 97 allows the offset of the first meaningful field - the section offsets in the 98 app header - to remain consistent with its position in the original app 99 binary. The ``BL`` instruction branches to the start of the text section in 100 the RMM core img. The ``BL`` instruction uses ``PC`` relative offset to encode 101 the branch target, so the instruction can be calculated when the bundled image 102 is assembled. The return value that is saved in the Link Register by the 103 ``BL`` instruction is used by RMM core to determine the memory address of the 104 header of the first app in the bundle. 105* The rest of the RMM app bins (if any) unmodified. 106* The RMM core img file aligned to 64KB offset to meet ELF loader requirements. 107 108The packaged RMM binary must be loaded at a 64KB-aligned physical address to 109ensure that the RMM core itself is also 64KB-aligned. This alignment requirement 110stems from the use of a 1:1 virtual-to-physical address mapping for the RMM 111core. In contrast, EL0 apps run at virtual addresses determined at compile time, 112which are already 64KB-aligned. As a result, the physical address alignment of 113app binaries is not significant. 114 115The number of RMM apps expected by RMM core is hardcoded in RMM core. 116The RMM app headers are parsed by the RMM core during the coldboot phase. The 117RMM app headers contain a magic value, so if less than the expected number of 118RMM apps is added to the final bin that can be detected, then boot fails in that 119case. If more than the hardcoded number of RMM apps is present, RMM boot fails 120as well. 121 122|RMM bin generation| 123 124**************** 125RMM App Instance 126**************** 127 128In this document app instance refers to an execution context, along with the 129necessary memory that is allocated for it. In operating system terms, a thread 130is analogous to an EL0 App instance. 131 132To manage and run an RMM app instance, RMM core needs memory for the following 133purposes: 134 1351. RMM app context: A single page of memory for use by the context switch logic 136 to back up registers. This memory page acts as app's CPU context storage when 137 it is not scheduled. This page is mapped in the APP VA space as well, with 138 permissions that it is only accessible by privileged execution (i.e. EL2). 139 The mapping is necessary because, on exit from an RMM app to the RMM core, 140 the RMM app's virtual address (VA) space is still active and set up in the 141 High VA region, while the RMM core's VA space is not yet available. To switch 142 back to RMM Core High VA space, the general purpose registers need to be 143 backed up beforehand. To enable this, the app context page is mapped at the 144 beginning of the app’s VA space (without UNPRIV access). This fixed mapping 145 allows the context switch code to use a hardcoded address for the register 146 backup. Consequently, the EL0 app’s accessible address space begins at a 4KB 147 offset from the start of the High VA region 148 1492. A memory page that is used to exchange RMM app specific data between the 150 RMM app and the RMM app caller mapped both in EL2 and in EL0. (Also 151 referenced as shared page by this document.) 152 1533. Metadata for the RMM app (``struct app_data_cfg``). This includes 154 155 * xlat library structures including the pointer to the pagetable for the RMM 156 app. 157 * entry point 158 * PA of the app_reg_ctx 159 * PA of the shared page 160 1614. Page containing the pagetable for the EL0 memory mappings (currently the RMM 162 app has a limited 2MB VA space) 163 1645. Stack for RMM app. 165 1666. Heap for RMM app. 167 168In the current implementation there are 3 possible locations where the above 169mentioned data is stored: 170 171a. In statically allocated per-cpu buffers. 172b. In the rec structure for a REC and in the rec aux pages 173c. In the pdev structure for a PDEV object in the pdev aux pages 174 175In case of *b.* and *c.* the memory pages are delegated by the host. 176 177Note that this introduces a limitation on the stack/heap of the RMM apps, as the 178max count of aux pages is limited, and RMM core is already using some of them. 179 180**************** 181RMM App VA space 182**************** 183 184The RMM app va space (as it is configured in the xlat library) contains the 185following regions: 186 187 * Single page for the RMM app context (Transient) 188 * RMM app text 189 * RMM app RO data 190 * RMM app data 191 * RMM app BSS 192 * RMM app Shared (Transient) 193 * RMM app heap (Transient) 194 * RMM app stack (Transient) 195 196The pages that are mapped in the transient region are private to the RMM app 197instance. As all the other regions are common, it is possible to initialise the 198xlat_ctx_cfg used by apps during RMM boot. 199 200************************** 201Initialising RMM app pages 202************************** 203 204The pages that are instance specific (`RMM App Instance`_) needs to be initialised 205on creating an RMM app instance. 206 207As it is possible that the pages are delegated by the host, and the pages need 208to be mapped dynamically using slot-buffer framework, a new slot is added to the 209slot buffer. When initialising such pages, the page under initialisation is 210mapped into this new slot, and unmapped after initialising. 211 212The ``app_init_data`` function receives an array of physical addresses of 213the pages to be used for the RMM app. In case of initialisation the code checks 214whether the PA is in the range of the RMM core's rw PA range. If so, it can 215write to it directly. If not, the slot buffer mechanism is used as described 216above. 217 218Mappings when using AUX pages for RMM app RW memory: 219 220|RMM app memory aux| 221 222Mappings when using statically allocated pages for RMM app RW memory: 223 224|RMM app memory static| 225 226********************* 227RMM App Service Calls 228********************* 229 230Since the apps are running at EL0, it cannot perform some privileged operations 231like communicate with EL3 or copy data from NS host buffer etc. RMM provides 232some services for RMM apps (like printing). The services are required to have 233the signature 234 235 236 .. code-block:: bash 237 238 typedef uint64_t (*app_service_func)(struct app_data_cfg *app_data, 239 unsigned long arg0, 240 unsigned long arg1, 241 unsigned long arg2, 242 unsigned long arg3); 243 244Services are expected to be thread safe. 245 246Some services can call other apps to perform its functionality (like PRNG). But 247this kind of nesting is limited to one other app to avoid interdependencies 248between apps. 249 250Services are stored in a function pointer array. There is a single array in the 251system. 252 253The RMM app that calls a service must do an SVC with a predefined immediate 254value. The index of the service to be executed is selected by the value of the 255X0 register. 256 257The el0_app framework calls app_enter in a loop. It exits the loop 258if the SVC immediate value reports function return or yield (in future). 259 260******************** 261Debugging RMM on FVP 262******************** 263 264The RMM core text section offset is changed due to the app binaries being 265prepended before the RMM core img. The ``bundle_app_rmm.py`` script prints the 266text section offset in the binary which should be added to the RMM load address 267when loading the elf in the debugger for symbols. The script output is in the 268file ``<build_dir>/<build_type>/bundle_app_out.txt``. The offset is also printed 269out during the build process. 270 271The RMM apps are linked to the VA address that they are going to be loaded by 272RMM core, so the RMM app symbols needs to be loaded in the debugger with offset 273``0x0`` 274 275******************* 276RMM Fake Host Build 277******************* 278 279In case of the fake host build, the applications are compiled as a standalone 280elf file. The RMM core is compiled to the elf file ``rmm_core.elf``. The path 281to the RMM app elf files are passed to ``rmm_core.elf`` as a command line 282parameter, along with the ID of the RMM apps. The first time the main RMM 283process calls ``app_init_data`` the process is forked, the image of the 284requested RMM app is loaded in the new process, and a named-pipe connection is 285established between the two processes. When the RMM app process is created, for 286each ``app_init_data`` (including the first call) a new thread is created. The 287main thread in the RMM app process is responsible for dispatching the RMM app 288calls and returns between the main RMM process and the RMM app thread. 289 290There is no shared memory between the main and the RMM app processes, memory 291sharing is emulated by sending over the content of the main process's "shared 292page" to the RMM app thread on an ``app_run`` call, and sending back the content 293of the RMM app thread's "shared page" on RMM app function return. 294 295|RMM app host| 296 297The help of the main process: 298 299 .. code-block:: 300 301 $ rmm_core.elf --help 302 Run RMM on the host 303 304 Usage: rmm_core.elf [-h|--help] [app_id app_elf [...]] 305 306 Arguments: 307 -h, --help print this message and exit. 308 app_id Integer value of app id of the app. 309 app_elf path to the app's elf file. 310 311 312An example call: 313 314 .. code-block:: bash 315 316 $ Debug/rmm_core.elf \ 317 103 Debug/rmm_app_random.elf \ 318 211 Debug/rmm_app_attestation.elf 319 320************************* 321RMM Apps currently in RMM 322************************* 323 324Currently there are 2 RMM apps in RMM: 325 326 - attestation: Used for doing attestation related tasks, like attestation 327 token generation and hash calculation 328 - random: A small application with a pseudo random number generator, either 329 seeded with a true random sequence, or with input from an already initialised 330 random RMM app. 331 332Currently both apps have their own build of Mbed TLS linked to them. 333 334There is an instance of random number RMM app initialised for each of the CPUs. 335There is also an attestation app initialised for each CPU for doing RIM 336extension, and there is an RMM app instance initialised for each REC, to do 337measurement extend and attestation token generation. 338 339The attestation RMM app uses the random RMM app via a service. The attestation 340RMM app reaches the EL3 interface (for requesting realm attestation key 341and platform token) via an RMM app service. 342 343The layout of the RMM apps can be seen on the image below: 344 345|RMM app layout| 346 347Note: To be continued with DA related RMM apps 348 349************************ 350Directory layout of apps 351************************ 352 353 .. code-block:: bash 354 355 . 356 └── app 357 ├── attestation 358 │ ├── el0_app 359 │ │ ├── mbedtls_attest # Library (EL0) 360 │ │ ├── qcbor # Library (EL0) 361 │ │ ├── src # Implementation of the attestation RMM app (EL0) 362 │ │ └── t_cose # Library (EL0) 363 │ └── rmm_stub # Helper function for using the Attestation RMM app (EL2) 364 ├── common 365 │ ├── el0_app # Common code that is used by multiple apps (EL0) 366 │ ├── framework # Framework for calling an app and provide app services (EL2) 367 │ ├── include 368 │ └── rmm_svc # Implementation of RMM services provided to apps (EL2) 369 └── random 370 ├── el0_app 371 │ ├── mbedtls_random # Library (EL0) 372 │ └── src # Implementation of the Random RMM app (EL0) 373 └── rmm_stub # Helper function for using the Random RMM app (EL2) 374 375********************************************** 376Proposed Enhancements to the RMM App Framework 377********************************************** 378 379#. Shared Memory for ``fake_host`` 380 381 Implement support for shared memory to enable communication between processes 382 within the ``fake_host`` environment. 383 384#. Configurable App ID 385 386 Currently, the App ID is hardcoded. There is a suggestion to derive it from a 387 header generated as part of the configuration. This approach can be explored 388 further. 389 390#. App BSS Allocation 391 392 Consider whether the App BSS should be allocated dynamically. Alternatively, 393 RMM could reserve a dedicated section for app BSS and allocate memory from 394 there. 395 396#. Common Binary Generation Logic 397 398 The logic for generating app bin files can be consolidated or reused. 399 400#. Avoid Unnecessary Remapping of Shared Buffers 401 402 When shared buffers are located in auxiliary granules, remapping may not be 403 needed if the granules are already mapped. 404 405#. Unify VBAR Usage 406 407 Currently, separate VBARs are set up for the app and RMM. Evaluate whether a 408 single VBAR could be used to simplify the design. 409 410#. Lightweight App Threading in ``fake_host`` 411 412 Replace full PThreads with a lighter-weight threading model for running apps 413 in the ``fake_host`` test environment. 414 415#. Enable FEAT_PAN/PAN3 for EL0 416 417 Ensure that the appropriate Privileged Access Never (PAN) features (including 418 PAN3) are correctly set when executing code at EL0. 419 420#. Hardening ``app_services`` 421 422 The ``app_services`` interface should be hardened. This includes: 423 424 - Clear documentation of function arguments and return codes 425 - Proper validation of input parameters 426 427#. Harden ``rmm_stub`` and Define Conventions 428 429 The ``rmm_stub`` must validate all return values from EL0 apps. Also, define 430 a consistent convention for parameter passing between the EL0 app and the 431 stub to support future extensibility. 432 433#. Optional TLBI on EL0 Entry 434 435 In certain cases, Translation Lookaside Buffer Invalidate (TLBI) may not be 436 needed when entering an EL0 app. This optimization is already noted in the 437 code and should be validated and documented. 438 439#. Post-Build App Repackaging 440 441 Explore the possibility of repackaging EL0 apps after the main RMM binary is 442 built. 443 444#. Unmap heap pages allocated to app from RMM high VA space. 445 446 When heap is allocated from REC aux granules, the aux granules are always 447 mapped when the REC is in use and the app is not running. Even if app is 448 running stack and pages table page not needed to be mapped, and unless some 449 data is being shared between core and app via heap, heap pages are not needed 450 to be mapped 451 452.. |RMM app EL0| image:: ./diagrams/rmm_app_EL0.drawio.png 453.. |RMM bin generation| image:: ./diagrams/rmm_bin_generation.drawio.png 454.. |RMM app memory aux| image:: ./diagrams/rmm_app_memory_aux.drawio.png 455.. |RMM app memory static| image:: ./diagrams/rmm_app_memory_static.drawio.png 456.. |RMM app host| image:: ./diagrams/rmm_app_on_host.drawio.png 457.. |RMM app layout| image:: ./diagrams/rmm_app_layout.drawio.png 458
