1##################
2RSE to RSE routing
3##################
4
5In multi-RSE implementations, RSE uses a routing protocol to allow messages to
6be sent from any RSE to any other RSE, regardless of if there is a physical link
7between them. This may involve an intermediate RSE forwarding the message.
8
9A routing table is used to indicate which is the next-hop for any message. This
10table is indexed by the destination RSE and contains an index indicating which
11hardware link should be used to send the message.
12
13The table is located in OTP, which allows for it to be provisioned once the chip
14topology is known. This also allows BL1_2 code to be identical across all RSEs
15in a system.
16
17The routing table is automatically constructed by the build system. Each
18platform must specify a .tgf (trivial graph format) file which describes the
19system topology, which is then used by the provisioning scripts to calculate the
20next-hop for each RSE to every other RSE. This generates individual provisioning
21bundles for each RSE containing their routing tables.
22
23***********************
24RSE System Topology Map
25***********************
26
27The system topology map is described as a .tgf file. The MULTI_RSE_TOPOLOGY_FILE
28cmake variable configures which system topology map tgf file will be used by
29the build system. This variable should be set in RSE subplatform default config
30for any subplatform which contains multiple RSEs.
31
32Documentation of the TGF format can be found at
33https://en.wikipedia.org/wiki/Trivial_Graph_Format#Format.
34
35In the system topology map, each node represents an RSE. The Label portion of
36the node is optional, but it is recommended to use it to describe which RSE the
37node represents. Each edge represents a physical link between the corresponding
38RSEs. The edges are directional, with the first index representing the RSE
39whose link IDs are described by the edge (henceforth referred to as the Local
40RSE) and the second index representing the RSE which is being communicated with
41(henceforth referred to as the Remote RSE).
42
43An example graph node line is::
44
45    0 Primary RSE
46
47Because knowing that a link exists does not encode which link should be used,
48the link ID is encoded as the name of the edge. The format of this name must be
49"Send <n> Receive <m>" where <n> is the index of the link which is used by the
50Local RSE to send to the Remote RSE, and <m> is the index of the link which is
51used by the Local RSE to receive from the Remote RSE.
52
53An example graph edge line is::
54
55    0 1 Send 1 Receive 2
56
57In this example, the Local RSE has index 0 and the Remote RSE has index 1. If
58the Local RSE wants to send a message to the Remote RSE, it should send the
59message via the link with index 1. If the Local RSE wants to listen for messages
60from the Remote RSE, it should listen on the link with index 2.
61
62The following is an example topology map for a system with 4 RSEs, which are all
63connected to each other::
64
65    0 RSE 0
66    1 RSE 1
67    2 RSE 2
68    3 RSE 3
69
70    0 1 Send 1 Receive 1
71    0 2 Send 2 Receive 2
72    0 3 Send 3 Receive 3
73
74    1 0 Send 0 Receive 0
75    1 2 Send 2 Receive 2
76    1 3 Send 3 Receive 3
77
78    2 0 Send 0 Receive 0
79    2 1 Send 1 Receive 1
80    2 3 Send 3 Receive 3
81
82    3 0 Send 0 Receive 0
83    3 1 Send 1 Receive 1
84    3 2 Send 2 Receive 2
85
86The following is a "hub-and-spoke" system topology with 5 RSEs::
87
88    0 Hub
89    1 Spoke 0
90    2 Spoke 1
91    3 Spoke 2
92    4 Spoke 3
93
94    0 1 Send 0 Receive 0
95    0 2 Send 1 Receive 1
96    0 3 Send 2 Receive 2
97    0 4 Send 3 Receive 3
98
99    1 0 Send 0 Receive 0
100    2 0 Send 0 Receive 0
101    3 0 Send 0 Receive 0
102    4 0 Send 0 Receive 0
103
104*****************
105RSE Routing Table
106*****************
107
108The RSE routing table is provisioned into OTP. It is provisioned as two distinct
109tables, as "send" table and a "receive" table. Each table entry is a uint32_t,
110which is used as an index into the link table in order to decide which link
111to send/receive on. The amount of elements in the routing table is the same as
112the number of RSEs in the system.
113
114In order to find the link that should be used, the destination RSE should be
115used as an index in the correct routing table. The value in the routing table
116should then be used to select the link.
117
118The size of the routing tables is 4 bytes * 2 * the number of RSEs in the
119system.
120
121The value in the table which corresponds to the current RSE must be 0. It is
122invalid to send a message to the same RSE that is sending the message. 0 is also
123a valid link index.
124
125Each RSE in the system is provisioned with different routing tables. The routing
126tables for each RSE are generated by running a shortest-path algorithm on the
127graph between the RSE whose table is being generated and every other RSE.
128
129The link table for RSE 2 in the first example above (fully connected) is::
130
131    Send    = {0, 1, 0, 3}
132    Receive = {0, 1, 0, 3}
133
134The link table for RSE 1 in the Second example above (hub-and-spoke) is::
135
136    Send    = {0, 0, 0, 0}
137    Receive = {0, 0, 0, 0}
138
139--------------
140
141*Copyright (c) 2024, Arm Limited. All rights reserved.*
142