1# Unit Testing
2This directory contains a basic unit test framework for the SCP firmware.
3Please note that this is under development and subject to change.
4
5## Background
6SCP-firmware provides few unit tests for core framework under framework/test
7directory. However this can not be used to implement unit tests for other
8components such as various modules. Hence a generic unit test framework was
9needed. This directory provides necessary unit test framework that can be
10used for this case.
11
12This unit test framework is implemented using
13[Unity](http://www.throwtheswitch.org/unity) and
14[CMock](http://www.throwtheswitch.org/cmock) components.
15
16It is recommended to read the documentation available on those links before
17you proceed below.
18
19The build system is implemented using CMake and it uses CTest feature of
20CMake to run the tests, although it is not necessary.
21
22---
23# Quick Start: Building and executing sample test
24
25- Pre-requisite
26
27  * Unity and CMock:
28
29    This framework depends on external components Unity and CMock
30
31    Execute below instructions to get the git submodules
32
33    ```sh
34    $ git submodule update --init --recursive
35    ```
36   * CMake:
37
38    Read doc/cmake_readme.md for overall setup for cmake (If not done before)
39    after that execute below instruction in this directory.
40
41- Build and run unit tests
42
43    ```sh
44    $ make -f Makefile.cmake mod_test
45     ```
46
47    These unit tests are also included in the test target, alongside framework tests.
48
49    ```sh
50    $ make -f Makefile.cmake test
51     ```
52---
53# Description
54
55### Components
56```
57unit_test
58        ├── cfg.yml
59        ├── CMakeLists.txt
60        ├── unity_mocks
61        │   ├── arch_helpers.h
62        │   └── mocks
63        ├── utils
64        │   └── zip_coverage_report.sh
65        ├── gm.rb
66        └── user_guide.md
67contrib
68        └── cmock
69
70
71- cfg.yml: Default configuration for generating Mocks.
72- CMakeLists.txt:
73  Toplevel CMakeLists file for building unit test framework and test cases.
74- unity_mocks: This contains generated mocks for framework component.
75- utils: This script files will generate code coverage reports in the form of xml & html.
76- gm.rb: A ruby script to generate mocks for passed header file.
77- user_guide.md: This file.
78- cmock: CMock and unity source code.
79```
80# Writing Unit Tests
81
82Documents available at [Unity](http://www.throwtheswitch.org/unity) explains
83the main concepts. However as an example, below description explains how a
84simple test is implemented for source code ```module/scmi/test/mod_scmi_unit_test.c```
85
86Below is the simple implementation of an unit test for the function
87```scmi_base_discover_sub_vendor_handler``` available
88in the ```module/scmi/test/mod_scmi_unit_test.c```
89
90```
91#include "unity.h"
92
93void setUp (void) {} /* Is run before every test, put unit init calls here. */
94void tearDown (void) {} /* Is run after every test, put unit clean-up calls here. */
95
96void test_TheFirst(void)
97{
98    TEST_IGNORE_MESSAGE("Hello world!"); /* Ignore this test but print a message. */
99}
100void scmi_test01(void)
101{
102        int status;
103        status = FWK_SUCCESS;
104        TEST_ASSERT_EQUAL(status, FWK_SUCCESS);
105}
106
107void scmi_test02(void)
108{
109        int status;
110        status = FWK_SUCCESS;
111        TEST_ASSERT_EQUAL(status, FWK_SUCCESS);
112
113}
114
115int scmi_test_main(void)
116{
117    UNITY_BEGIN();
118    RUN_TEST(test_TheFirst);
119    RUN_TEST(scmi_test01);
120    RUN_TEST(scmi_test02);
121    return UNITY_END();
122
123}
124
125#if !defined(TEST_ON_TARGET)
126int main(void)
127{
128    return scmi_test_main();
129}
130#endif
131
132```
133Sadly above code is not complete but gives enough idea about how Unity
134can be used. However from SCP-Firmware point of view there are various problems
135that requires some work before above code can be useful. Please see below:
136
137```
138#include "unity.h"
139
140#include <mod_scmi.h>
141#include <internal/mod_scmi.h>
142#include <fwk_module_idx.h>
143#include <fwk_element.h>
144
145#include UNIT_TEST_SRC
146
147void setUp(void)
148{
149}
150
151void tearDown(void)
152{
153}
154
155void test_function_scmi_base_discover_sub_vendor_handler(void)
156{
157    fwk_id_t ex = FWK_ID_ELEMENT_INIT(
158        FAKE_MODULE_ID,
159        FAKE_SERVICE_IDX_OSPM);
160
161    TEST_ASSERT_EQUAL(
162        FWK_SUCCESS,
163        scmi_base_discover_sub_vendor_handler(ex, NULL)
164        );
165}
166
167int main(void) {
168    UNITY_BEGIN();
169    RUN_TEST(test_function_scmi_base_discover_sub_vendor_handler);
170    return UNITY_END();
171}
172
173```
174As you can see above, few extra lines been added in the code.
175The important line here, is the inclusion of module source code
176in the unit test source file.
177
178```
179#include UNIT_TEST_SRC // equivalent to #include <module/scmi/mod_scmi.c>
180```
181
182This is necessary because we would like to bring various global objects
183and ```static``` members and methods in unit tests scope which otherwise
184would not be available for the function call
185```scmi_base_discover_sub_vendor_handler```. Also note, use of
186```TEST_ASSERT_EQUAL``` which mark test *FAIL* or *PASS* based on return value
187of the function under test.
188
189Sadly, above code is still incomplete because the function under test
190```scmi_base_discover_sub_vendor_handler``` calls various other calls
191which are not available for this unit test code. One way to make
192these function available is to use
193[CMock](http://www.throwtheswitch.org/cmock) and generate the mocked
194version of the functions on which ```scmi_base_discover_sub_vendor_handler```
195is dependent.
196
197For example to generate mocked version of the functions available
198in the ```framework/include/fwk_module.h``` Use below command
199
200```sh
201${SCP_ROOT}/unit_test/gm.rb  -m ${SCP_ROOT}/unit_test/unity_mocks/mocks/
202-f fwk_module.h
203```
204and for ```internal``` version of this file
205```framework/include/internal/fwk_module.h```
206
207```sh
208${SCP_ROOT}/unit_test/gm.rb -m ${SCP_ROOT}/unit_test/unity_mocks/mocks/
209-f internal/fwk_module.h -dinternal -s_internal
210```
211
212It's recommended to export the path to gm.rb to PATH for ease of use, but
213a relative reference to gm.rb from within the current directory would work
214fine.
215
216Apart from above mocked code, we need to setup few global objects as well
217the complete code can be found in:
218
219```
220└── module
221    └── scmi
222        ├── CMakeLists.txt
223        ├── include
224        │   ├── internal
225        │   │   ├── mod_scmi.h
226        │   │   ├── scmi_base.h
227        │   │   └── scmi.h
228        │   ├── mod_scmi.h
229        │   ├── mod_scmi_header.h
230        │   └── mod_scmi_std.h
231        ├── Module.cmake
232        ├── src
233        │   ├── Makefile
234        │   ├── mod_scmi.c
235        │   └── mod_scmi_base.c
236        └── test
237            ├── fwk_module_idx.h
238            ├── mocks
239            │   ├── Mockmod_scmi_extra.c
240            │   ├── Mockmod_scmi_extra.h
241            │   └── .clang-format
242            ├── mod_scmi_extra.h
243            ├── mod_scmi_unit_test.c
244            ├── CMakeLists.txt
245            └── fwk_module_idx.h
246
247```
248
249One of the important point to note above is the presence of
250```mod_scmi_extra.h``` and its generated mock version
251```Mockmod_scmi_extra.h```. This is needed because CMock can
252not mock functions which are declared as function pointers in the
253interface headers. So this work around helps to generate the mock
254functions for these declarations.
255
256When generating a new mock, a .clang-format file is placed in the
257same directory so that style checks don't evaluate the autogenerated
258code contained within. This will also exclude any other files contained
259in the directory, so mocks should be placed in seperated mock directories
260from hand-written code.
261
262> **Note:**
263> Modules depend on **external header files** must include the required
264>headers under `module/<module_path>/test/ext/`
265>
266>Make sure to update CMakeLists.txt of the unit test to add the `ext` folder
267>to the unit test includes.
268> CMakeLists.txt:
269>```cmake
270>       ...
271>       set(MODULE_UT_INC ${CMAKE_CURRENT_LIST_DIR}/ext)
272>       ...
273>```
274>*Header files inside the `ext` folder can be a modified version for testing
275>purposes.*
276>
277>`optee/mbx` unit test is a good example for such case.
278
279## Adding test for new modules
280
281A template of minimum required files is provided as a reference for new modules.
282
283See unit_test/template/test
284
2851. Duplicate existing reference test directory as a starting point.
286
2872. Modify *.cmake file for our specific test case:
288    a. Change TEST_MODULE to name of module
289    b. Add directories of other modules used in test to OTHER_MODULE_INC
290    c. Replace framework sources with mocks using replace_with_mock
291    d. Anything else
292
2933. Populate config data, typically using a platform's config as starting point
294
2954. Modify ```fwk_module_idx.h``` manual mock
296
2975. Mock any APIs, by mocking the ```mod_{}_extra.h``` file
298
2996. Write SetUp() to initialise ctx structures, bind apis etc.
300
3017. Write test functions
302
3038. Modify ```unit_test/CMakeLists.txt``` to append test under
304    ```#Append new unit tests below here```
305
3069. The test should be ready to run under ```make -f Makefile.cmake mod_test```
307
308    **Note:** The common framework specific mocks should be generated under
309```unit_test/unity_mocks/mocks``` and module specific mock should be
310generated under module specific unit test directory e.g.
311```module/new_module/test/mocks```
312
313## fwk_core mock workaround
314
315CMock generates the fwk_run_main_loop mock as a returning function,
316because ```noreturn``` is stripped out via cfg.yml. Compilation therefore
317complains when this function terminates. To work around this, an infinite
318for loop needs to be appended to the function manual when updating
319fwk_core's mock.
320
321## Unit testing style guidelines
322
323For the addition and changes for Unit Testing, it is preferable to follow
324the guidelines below:
325- The format for the functions' names being tested should be as follow:
326  ```utest_``` followed by the function being tested and a pass or fail
327  expectation, for example: ```utest_function_being_tested_init_success```.
328- One test scenario for each test case. For example, if testing one case for a
329  function, have a test function for that case only.
330- Name the test functions according to the test being performed.
331
332### Executing Tests on target hardware or FVP models
333
334All above describes how to build and execute unit tests on host machine.
335However, if needed, tests can be executed on hardware/models as well.
336
337Note that to build and execute tests on target, a few more steps are required:
338
3391. Since Unity/CMock requires stdio for output the log messages
340a bare minimum scp_ramfw is needed for the target under test as example
341see product/juno/scp_ut
342
3432. The target firmware must include module/ut to execute the test required
344
3453. Follow TEST_ON_TARGET and TEST_ON_HOST in test/module/scmi/CMakeLists.txt
346to understand setup needed building test that can be executed on target
347
3484. Platform must provide definition for plat_execute_all_tests which
349is called by module/ut. See example product/juno/scp_ut/tests_entry.c
350
351## Unit testing requirement guidelines
352
353Unit Testing (UT) has been introduced with the aim of improving the quality and
354reliability of the code.
355In this introductory phase, we suggest that contributors become familiar with
356the unit testing in SCP-firmware.
357In the meantime, while we are not enforcing contibutors to add unit testing on
358their additions or modifications, we encourage them to attempt adding UT
359whenever compatible with their development.
360We foresee that UT will become a mandatory requirement later in the future for
361contributions into SCP-firmware project.
362