1# Testing
2
3The test harness which runs on our bots (called "runtests") picks up all
4executables in the "/boot/test" and "/system/test" directories and runs them.
5If you provide a command-line argument, such as `runtests -S -m widget_test`,
6runtests will only run the single test requested -- in this case, `widget_test`.
7
8"runtests" takes command-line arguments to toggle classes of tests to execute.
9
10These classes are the following:
11
12* **Small**: Isolated tests for functions and classes. These must be totally
13  synchronous and single-threaded. These tests should be parallelizable; there
14  shouldn't be any shared resources between them.
15* **Medium**: Single-process integration tests. Ideally these are also synchronous
16  and single-threaded but they might run through a large chunk of code in each
17  test case, or they might use disk, making them a bit slower.
18* **Large**: Slow, multi-process, or particularly incomprehensible single-process
19  integration tests. These tests are often too slow / flaky to run in a CQ, and
20  we should try to limit how many we have.
21* **Performance**: Tests which are expected to pass, but which are measured
22  using other metrics (thresholds, statistical techniques) to identify
23  regressions.
24
25Since runtests doesn't really know what "class" is executing when it launches a
26test, it encodes this information in the environment variable
27`RUNTESTS_TEST_CLASS`, which is detailed in [the unittest
28header][unittest-header] , and lets the executable itself decide what to run /
29not run. This environment variable is a bitmask indicating which tests to run.
30
31For example, if a a test executable is run with "small" and "medium" tests,
32it will be executed ONCE with `RUNTESTS_TEST_CLASS` set to 00000003 (the
33hex bitwise OR of "TEST_SMALL" and "TEST_MEDIUM" -- though this information
34should be parsed using the [unittest header][unittest-header], as it may be
35updated in the future).
36
37## Zircon Tests (ulib/test, and/or using ulib/unittest)
38
39The following macros can be used to filter tests into these categories:
40```
41RUN_TEST_SMALL(widget_tiny_test)
42RUN_TEST_MEDIUM(widget_test)
43RUN_TEST_LARGE(widget_big_test)
44RUN_TEST_PERFORMANCE(widget_benchmark)
45```
46
47The legacy `RUN_TEST(widget_test)` is aliased to mean the same thing as
48`RUN_TEST_SMALL`.
49
50## Zircon Kernel Tests (kernel/lib/unittest)
51
52For tests compiled into the kernel itself you can call them from the shell with
53`k ut`. `k ut all` will run all tests or you can use `k ut $TEST_NAME` to run a
54specific test.
55
56The output from these tests will only be shown on the serial console.
57
58## Fuchsia Tests (not using ulib/unittest)
59
60The environment variable `RUNTESTS_TEST_CLASS` will still be available to all
61executables launched by runtests. The [unittest header][unittest-header] can be
62used to parse different categories of tests which the runtests harness attempted
63to run.
64
65## Runtests CLI
66
67By default, runtests will run both small and medium tests.
68
69To determine how to run a custom set of test categories, run `runtests -h`,
70which includes usage information.
71
72[unittest-header]: ../system/ulib/unittest/include/unittest/unittest.h "Unittest Header"
73