1# Unit Testing
2This directory contains a basic unit test framework for the SCP firmware.
3Please note that this is under development and subject to change.
4
5## Background
6SCP-firmware provides few unit tests for core framework under framework/test
7directory. However this can not be used to implement unit tests for other
8components such as various modules. Hence a generic unit test framework was
9needed. This directory provides necessary unit test framework that can be
10used for this case.
11
12This unit test framework is implemented using
13[Unity](http://www.throwtheswitch.org/unity) and
14[CMock](http://www.throwtheswitch.org/cmock) components.
15
16It is recommended to read the documentation available on those links before
17you proceed below.
18
19The build system is implemented using CMake and it uses CTest feature of
20CMake to run the tests, although it is not necessary.
21
22---
23# Quick Start: Building and executing sample test
24
25- Pre-requisite
26
27  * Unity and CMock:
28
29    This framework depends on external components Unity and CMock
30
31    Execute below instructions to get the git submodules
32
33    ```sh
34    $ git submodule update --init --recursive
35    ```
36   * CMake:
37
38    Read doc/cmake_readme.md for overall setup for cmake (If not done before)
39    after that execute below instruction in this directory.
40
41- Build and run unit tests
42
43    ```sh
44    $ make -f Makefile.cmake mod_test
45     ```
46
47    These unit tests are also included in the test target, alongside framework tests.
48
49    ```sh
50    $ make -f Makefile.cmake test
51     ```
52---
53# Description
54
55### Components
56```
57unit_test
58        ├── cfg.yml
59        ├── CMakeLists.txt
60        ├── unity_mocks
61        │   ├── arch_helpers.h
62        │   └── mocks
63        ├── utils
64        │   ├── generate_coverage_report.py
65        │   └── zip_coverage_report.sh
66        ├── gm.rb
67        └── user_guide.md
68contrib
69        └── cmock
70
71
72- cfg.yml: Default configuration for generating Mocks.
73- CMakeLists.txt:
74  Toplevel CMakeLists file for building unit test framework and test cases.
75- unity_mocks: This contains generated mocks for framework component.
76- utils: This script files will generate code coverage reports in the form of xml & html.
77- gm.rb: A ruby script to generate mocks for passed header file.
78- user_guide.md: This file.
79- cmock: CMock and unity source code.
80```
81# Writing Unit Tests
82
83Documents available at [Unity](http://www.throwtheswitch.org/unity) explains
84the main concepts. However as an example, below description explains how a
85simple test is implemented for source code ```module/scmi/test/mod_scmi_unit_test.c```
86
87Below is the simple implementation of an unit test for the function
88```scmi_base_discover_sub_vendor_handler``` available
89in the ```module/scmi/test/mod_scmi_unit_test.c```
90
91```
92#include "unity.h"
93
94void setUp (void) {} /* Is run before every test, put unit init calls here. */
95void tearDown (void) {} /* Is run after every test, put unit clean-up calls here. */
96
97void test_TheFirst(void)
98{
99    TEST_IGNORE_MESSAGE("Hello world!"); /* Ignore this test but print a message. */
100}
101void scmi_test01(void)
102{
103        int status;
104        status = FWK_SUCCESS;
105        TEST_ASSERT_EQUAL(status, FWK_SUCCESS);
106}
107
108void scmi_test02(void)
109{
110        int status;
111        status = FWK_SUCCESS;
112        TEST_ASSERT_EQUAL(status, FWK_SUCCESS);
113
114}
115
116int scmi_test_main(void)
117{
118    UNITY_BEGIN();
119    RUN_TEST(test_TheFirst);
120    RUN_TEST(scmi_test01);
121    RUN_TEST(scmi_test02);
122    return UNITY_END();
123
124}
125
126#if !defined(TEST_ON_TARGET)
127int main(void)
128{
129    return scmi_test_main();
130}
131#endif
132
133```
134Sadly above code is not complete but gives enough idea about how Unity
135can be used. However from SCP-Firmware point of view there are various problems
136that requires some work before above code can be useful. Please see below:
137
138```
139#include "unity.h"
140
141#include <mod_scmi.h>
142#include <internal/mod_scmi.h>
143#include <fwk_module_idx.h>
144#include <fwk_element.h>
145
146#include UNIT_TEST_SRC
147
148void setUp(void)
149{
150}
151
152void tearDown(void)
153{
154}
155
156void test_function_scmi_base_discover_sub_vendor_handler(void)
157{
158    fwk_id_t ex = FWK_ID_ELEMENT_INIT(
159        FAKE_MODULE_ID,
160        FAKE_SERVICE_IDX_OSPM);
161
162    TEST_ASSERT_EQUAL(
163        FWK_SUCCESS,
164        scmi_base_discover_sub_vendor_handler(ex, NULL)
165        );
166}
167
168int main(void) {
169    UNITY_BEGIN();
170    RUN_TEST(test_function_scmi_base_discover_sub_vendor_handler);
171    return UNITY_END();
172}
173
174```
175As you can see above, few extra lines been added in the code.
176The important line here, is the inclusion of module source code
177in the unit test source file.
178
179```
180#include UNIT_TEST_SRC // equivalent to #include <module/scmi/mod_scmi.c>
181```
182
183This is necessary because we would like to bring various global objects
184and ```static``` members and methods in unit tests scope which otherwise
185would not be available for the function call
186```scmi_base_discover_sub_vendor_handler```. Also note, use of
187```TEST_ASSERT_EQUAL``` which mark test *FAIL* or *PASS* based on return value
188of the function under test.
189
190Sadly, above code is still incomplete because the function under test
191```scmi_base_discover_sub_vendor_handler``` calls various other calls
192which are not available for this unit test code. One way to make
193these function available is to use
194[CMock](http://www.throwtheswitch.org/cmock) and generate the mocked
195version of the functions on which ```scmi_base_discover_sub_vendor_handler```
196is dependent.
197
198For example to generate mocked version of the functions available
199in the ```framework/include/fwk_module.h``` Use below command
200
201```sh
202${SCP_ROOT}/unit_test/gm.rb  -m ${SCP_ROOT}/unit_test/unity_mocks/mocks/
203-f fwk_module.h
204```
205and for ```internal``` version of this file
206```framework/include/internal/fwk_module.h```
207
208```sh
209${SCP_ROOT}/unit_test/gm.rb -m ${SCP_ROOT}/unit_test/unity_mocks/mocks/
210-f internal/fwk_module.h -dinternal -s_internal
211```
212
213It's recommended to export the path to gm.rb to PATH for ease of use, but
214a relative reference to gm.rb from within the current directory would work
215fine.
216
217Apart from above mocked code, we need to setup few global objects as well
218the complete code can be found in:
219
220```
221└── module
222    └── scmi
223        ├── CMakeLists.txt
224        ├── include
225        │   ├── internal
226        │   │   ├── mod_scmi.h
227        │   │   ├── scmi_base.h
228        │   │   └── scmi.h
229        │   ├── mod_scmi.h
230        │   ├── mod_scmi_header.h
231        │   └── mod_scmi_std.h
232        ├── Module.cmake
233        ├── src
234        │   ├── Makefile
235        │   ├── mod_scmi.c
236        │   └── mod_scmi_base.c
237        └── test
238            ├── fwk_module_idx.h
239            ├── mocks
240            │   ├── Mockmod_scmi_extra.c
241            │   ├── Mockmod_scmi_extra.h
242            │   └── .clang-format
243            ├── mod_scmi_extra.h
244            ├── mod_scmi_unit_test.c
245            ├── CMakeLists.txt
246            └── fwk_module_idx.h
247
248```
249
250One of the important point to note above is the presence of
251```mod_scmi_extra.h``` and its generated mock version
252```Mockmod_scmi_extra.h```. This is needed because CMock can
253not mock functions which are declared as function pointers in the
254interface headers. So this work around helps to generate the mock
255functions for these declarations.
256
257When generating a new mock, a .clang-format file is placed in the
258same directory so that style checks don't evaluate the autogenerated
259code contained within. This will also exclude any other files contained
260in the directory, so mocks should be placed in seperated mock directories
261from hand-written code.
262
263## Adding test for new modules
264
265The ```scmi``` and ```scmi_clock``` test directories are provided
266as a reference for new modules. The following process is intended
267as a general guide, and will vary on a case-by-case basis.
268
2691. Duplicate existing reference test directories as a starting point.
270
2712. Modify *.cmake file for our specific test case:
272    a. Change TEST_MODULE to name of module
273    b. Add directories of other modules used in test to OTHER_MODULE_INC
274    c. Replace framework sources with mocks using replace_with_mock
275    d. Anything else
276
2773. Populate config data, typically using a platform's config as starting point
278
2794. Modify ```fwk_module_idx.h``` manual mock
280
2815. Mock any APIs, by mocking the ```mod_{}_extra.h``` file
282
2836. Write SetUp() to initialise ctx structures, bind apis etc.
284
2857. Write test functions
286
2878. Modify ```unit_test/CMakeLists.txt``` to append test under
288    ```#Append new unit tests below here```
289
2909. The test should be ready to run under ```make -f Makefile.cmake mod_test```
291
292    **Note:** The common framework specific mocks should be generated under
293```unit_test/unity_mocks/mocks``` and module specific mock should be
294generated under module specific unit test directory e.g.
295```module/new_module/test/mocks```
296
297## fwk_core mock workaround
298
299CMock generates the fwk_run_main_loop mock as a returning function,
300because ```noreturn``` is stripped out via cfg.yml. Compilation therefore
301complains when this function terminates. To work around this, an infinite
302for loop needs to be appended to the function manual when updating
303fwk_core's mock.
304
305## Unit testing style guidelines
306
307For the addition and changes for Unit Testing, it is preferable to follow
308the guidelines below:
309- The format for the functions' names being tested should be as follow:
310  ```utest_``` followed by the function being tested and a pass or fail
311  expectation, for example: ```utest_function_being_tested_init_success```.
312- One test scenario for each test case. For example, if testing one case for a
313  function, have a test function for that case only.
314- Name the test functions according to the test being performed.
315- When an addition or change is made to a module that did not have unit testing
316  before, it is recommended that the enclosing functions affected by the change
317  should then be added for unit testing.
318
319### Executing Tests on target hardware or FVP models
320
321All above describes how to build and execute unit tests on host machine.
322However, if needed, tests can be executed on hardware/models as well.
323
324Note that to build and execute tests on target, a few more steps are required:
325
3261. Since Unity/CMock requires stdio for output the log messages
327a bare minimum scp_ramfw is needed for the target under test as example
328see product/juno/scp_ut
329
3302. The target firmware must include module/ut to execute the test required
331
3323. Follow TEST_ON_TARGET and TEST_ON_HOST in test/module/scmi/CMakeLists.txt
333to understand setup needed building test that can be executed on target
334
3354. Platform must provide definition for plat_execute_all_tests which
336is called by module/ut. See example product/juno/scp_ut/tests_entry.c
337