1.. SPDX-License-Identifier: GPL-2.0
2
3==============
4Memory Hotplug
5==============
6The final phase of surfacing CXL memory to the kernel page allocator is for
7the `DAX` driver to surface a `Driver Managed` memory region via the
8memory-hotplug component.
9
10There are four major configurations to consider:
11
121) Default Online Behavior (on/off and zone)
132) Hotplug Memory Block size
143) Memory Map Resource location
154) Driver-Managed Memory Designation
16
17Default Online Behavior
18=======================
19The default-online behavior of hotplug memory is dictated by the following,
20in order of precedence:
21
22- :code:`CONFIG_MHP_DEFAULT_ONLINE_TYPE` Build Configuration
23- :code:`memhp_default_state` Boot parameter
24- :code:`/sys/devices/system/memory/auto_online_blocks` value
25
26These dictate whether hotplugged memory blocks arrive in one of three states:
27
281) Offline
292) Online in :code:`ZONE_NORMAL`
303) Online in :code:`ZONE_MOVABLE`
31
32:code:`ZONE_NORMAL` implies this capacity may be used for almost any allocation,
33while :code:`ZONE_MOVABLE` implies this capacity should only be used for
34migratable allocations.
35
36:code:`ZONE_MOVABLE` attempts to retain the hotplug-ability of a memory block
37so that it the entire region may be hot-unplugged at a later time.  Any capacity
38onlined into :code:`ZONE_NORMAL` should be considered permanently attached to
39the page allocator.
40
41Hotplug Memory Block Size
42=========================
43By default, on most architectures, the Hotplug Memory Block Size is either
44128MB or 256MB.  On x86, the block size increases up to 2GB as total memory
45capacity exceeds 64GB.  As of v6.15, Linux does not take into account the
46size and alignment of the ACPI CEDT CFMWS regions (see Early Boot docs) when
47deciding the Hotplug Memory Block Size.
48
49Memory Map
50==========
51The location of :code:`struct folio` allocations to represent the hotplugged
52memory capacity are dictated by the following system settings:
53
54- :code:`/sys_module/memory_hotplug/parameters/memmap_on_memory`
55- :code:`/sys/bus/dax/devices/daxN.Y/memmap_on_memory`
56
57If both of these parameters are set to true, :code:`struct folio` for this
58capacity will be carved out of the memory block being onlined.  This has
59performance implications if the memory is particularly high-latency and
60its :code:`struct folio` becomes hotly contended.
61
62If either parameter is set to false, :code:`struct folio` for this capacity
63will be allocated from the local node of the processor running the hotplug
64procedure.  This capacity will be allocated from :code:`ZONE_NORMAL` on
65that node, as it is a :code:`GFP_KERNEL` allocation.
66
67Systems with extremely large amounts of :code:`ZONE_MOVABLE` memory (e.g.
68CXL memory pools) must ensure that there is sufficient local
69:code:`ZONE_NORMAL` capacity to host the memory map for the hotplugged capacity.
70
71Driver Managed Memory
72=====================
73The DAX driver surfaces this memory to memory-hotplug as "Driver Managed". This
74is not a configurable setting, but it's important to note that driver managed
75memory is explicitly excluded from use during kexec.  This is required to ensure
76any reset or out-of-band operations that the CXL device may be subject to during
77a functional system-reboot (such as a reset-on-probe) will not cause portions of
78the kexec kernel to be overwritten.
79