1Running external test suites with OpenSSL
2=========================================
3
4It is possible to integrate external test suites into OpenSSL's `make test`.
5This capability is considered a developer option and does not work on all
6platforms.
7
8Python PYCA/Cryptography test suite
9===================================
10
11This python test suite runs cryptographic tests with a local OpenSSL build as
12the implementation.
13
14First checkout the `PYCA/Cryptography` module into `./pyca-cryptography` using:
15
16    $ git submodule update --init
17
18Then configure/build OpenSSL compatible with the python module:
19
20    $ ./config enable-external-tests
21    $ make
22
23The tests will run in a python virtual environment which requires virtualenv
24to be installed.
25
26    $ make test VERBOSE=1 TESTS=test_external_pyca
27
28Test failures and suppressions
29------------------------------
30
31Some tests target older (<=1.0.2) versions so will not run. Other tests target
32other crypto implementations so are not relevant. Currently no tests fail.
33
34krb5 test suite
35===============
36
37Much like the PYCA/Cryptography test suite, this builds and runs the krb5
38tests against the local OpenSSL build.
39
40You will need a git checkout of krb5 at the top level:
41
42    $ git clone https://github.com/krb5/krb5
43
44krb5's master has to pass this same CI, but a known-good version is
45krb5-1.15.1-final if you want to be sure.
46
47    $ cd krb5
48    $ git checkout krb5-1.15.1-final
49    $ cd ..
50
51OpenSSL must be built with external tests enabled:
52
53    $ ./config enable-external-tests
54    $ make
55
56krb5's tests will then be run as part of the rest of the suite, or can be
57explicitly run (with more debugging):
58
59    $ VERBOSE=1 make TESTS=test_external_krb5 test
60
61Test-failures suppressions
62--------------------------
63
64krb5 will automatically adapt its test suite to account for the configuration
65of your system.  Certain tests may require more installed packages to run.  No
66tests are expected to fail.
67
68GOST engine test suite
69======================
70
71Much like the PYCA/Cryptography test suite, this builds and runs the GOST engine
72tests against the local OpenSSL build.
73
74You will need a git checkout of gost-engine at the top level:
75
76    $ git submodule update --init
77
78Then configure/build OpenSSL enabling external tests:
79
80    $ ./config enable-external-tests
81    $ make
82
83GOST engine requires CMake for the build process.
84
85GOST engine tests will then be run as part of the rest of the suite, or can be
86explicitly run (with more debugging):
87
88    $ make test VERBOSE=1 TESTS=test_external_gost_engine
89
90OQSprovider test suite
91======================
92
93Much like the PYCA/Cryptography test suite, this builds and runs the OQS
94(OpenQuantumSafe -- www.openquantumsafe.org) provider tests against the
95local OpenSSL build.
96
97You will need a git checkout of oqsprovider at the top level:
98
99    $ git submodule update --init
100
101Then configure/build OpenSSL enabling external tests:
102
103    $ ./config enable-external-tests
104    $ make
105
106oqsprovider requires CMake for the build process.
107
108OQSprovider tests will then be run as part of the rest of the suite, or can be
109explicitly run (with more debugging):
110
111    $ make test VERBOSE=1 TESTS=test_external_oqsprovider
112
113The names of all supported quantum-safe algorithms are available at
114<https://github.com/open-quantum-safe/oqs-provider#algorithms>.
115
116Please note specific limitations of oqsprovider operations dependent on specific
117openssl versions as documented at
118<https://github.com/open-quantum-safe/oqs-provider#note-on-openssl-versions>.
119
120pkcs11-provider test suite
121==========================
122
123This builds and runs pkcs11-provider tests against the local OpenSSL build.
124
125You will need a git checkout of pkcs11-provider at the top level:
126
127    $ git submodule update --init
128
129Then configure/build OpenSSL enabling external tests:
130
131    $ ./config enable-external-tests
132    $ make
133
134pkcs11-provider requires meson for the build process. Moreover, it requires
135kryoptic, opensc and expect to run the tests.
136
137Tests will then be run as part of the rest of the suite, or can be
138explicitly run (with more debugging):
139
140    $ make test VERBOSE=1 TESTS=test_external_pkcs11_provider
141
142Test failures and suppressions
143------------------------------
144
145There are tests for different software tokens - softhsm, nss-softokn and kryoptic.
146Kryoptic tests will not run at this point. Currently no test fails.
147
148Updating test suites
149====================
150
151To update the commit for any of the above test suites:
152
153- Make sure the submodules are cloned locally:
154
155    $ git submodule update --init --recursive
156
157- Enter subdirectory and pull from the repository (use a specific branch/tag if required):
158
159    $ cd `<submodule-dir>`
160    $ git pull origin master
161
162- Go to root directory, there should be a new git status:
163
164    $ cd ../
165    $ git status
166      ...
167      #       modified:   `<submodule-dir>` (new commits)
168      ...
169
170- Add/commit/push the update
171
172    $ git add `<submodule-dir>`
173    $ git commit -m `"Updated <submodule> to latest commit"`
174    $ git push
175