This commit also:
- rename the reference component as component_test_full_block_cipher_legacy_dispatch()
- add a common configuration function, named common_block_cipher_dispatch() that
is used from both accelerated and reference components
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
This test case is incompatible with sanitizers (e.g. ASan), and thus
skipped. If the driver component uses a sanitizer but the reference
component doesn't, we have a PASS vs SKIP mismatch. Since this test case is
unrelated to drivers, we don't mind ignoring it.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The previous check also warned when on tests that were already skipped
in the reference config, which are not really a problem. The purpose of
this "uselessly ignored" check is to make sure that the ignore list
(together with the config common to driver and reference in all.sh)
always correct reflects what works or doesn't in driver-only builds. For
this it's enough to warn when a test is ignored but passing.
The previous, stricter check, was causing issues like:
Error: uselessly ignored: test_suite_pkcs12;PBE Encrypt, pad = 8 (PKCS7 padding disabled)
Error: uselessly ignored: test_suite_pkcs12;PBE Decrypt, (Invalid padding & PKCS7 padding disabled)
Error: uselessly ignored: test_suite_pkcs5;PBES2 Decrypt (Invalid padding & PKCS7 padding disabled)
These are skipped in the reference config because is has PKCS7 padding
enabled, and that's OK.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
This may come in handy when ignoring patterns in test_suite_cipher for
example which is split in several .data files where we'll want to ignore
the same patterns.
Currently none of the entries had a '.' in the test suite name, so this
doesn't change anything for existing entries.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Change from iterating on all available tests to iterating on tests with
an outcome: initially we were iterating on available tests but
immediately ignoring those that were always skipped. That last part
played poorly with the new error (we want to know if the test ignored
due to a pattern was skipped in the reference component), but when
removing it, we were left with iterating on all available tests then
skipping those that don't have outcomes entries: this is equivalent to
iterating on tests with an outcome entry, which is more readable.
Also, give an error if the outcome file contains no passing test from
the reference component: probably means we're using the wrong outcome
file.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
This also adds a new task in analyze_outcomes.py for checking
the accelaration coverage against the reference element.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
- the script now only terminates in case of hard faults
- each task is assigned a log
- this log tracks messages, warning and errors
- when task completes, errors and warnings are listed and
messages are appended to the main log
- on exit the main log is printed and the proper return value
is returned
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Restore guards from the previous release, instead of the new, more
permissive guards.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
By default, we disable ASN1_[PARSE/WRITE]_C in common_tfm_config.
In fact, this is what happens for accelerated p256m driver, which
means all asn1[parse/write] tests are skipped in driver_accel test.
However, those two macros are automatically enabled for built-in
ECDSA via PSA, which means all asn1[parse/write] tests are passed
in tfm_config test.
This commit simply ignores the whole asn1[parse/write] test suite
when analyzing between driver and reference.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
{} are valid characters in test description, but they're not escaped
properly in python string format(). To resolve the bug of KeyError
when it tries to log test description which contains {}, we replace
{XXX} format with {{XXX}} in order to escape {} in python string
format() properly.
In addition, the calls to Results.log() are also handled to avoid
similar potential problems.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
* Don't break string literals in the allow list definition
* Comment each test that belongs to the allow list is there.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
* Turn the warnings produced when finding non-executed tests that
are not in the allow list into errors.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
Introduce the --require-full-coverage in analyze_outcomes.py so that
when analyze_outcomes.py --require-full-coverage is called, those
tests that are not executed and are not in the allowed list issue an
error instead of a warning.
Note that it is useful to run analyze_outcomes.py on incomplete test
results, so this error mode needs to remain optional in the long
term.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
The allow list explicits which test cases are allowed to not be
executed when testing. This may be, for example, because a feature
is yet to be developed but the test for that feature is already in
our code base.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
This includes also:
- auto enabling ECP_LIGHT when MBEDTLS_PK_PARSE_EC_COMPRESSED is
defined
- replacing ECP_LIGHT guards with PK_PARSE_EC_COMPRESSED in pkparse
- disabling PK_PARSE_EC_COMPRESSED in tests with accelarated EC curves
(it get disabled also in the reference components because we want
to achieve test parity)
- remove skipped checks in analyze_outcomes.py
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
The exception in analyze_outcomes.py follows previous commit in which
a test in test_suite_pkparse was set with the ECP_C guard for a different
parsing of the private key between the legacy and PSA implementations.
The wrong guard in test_suite_ecp.function instead was erroneously
added in a past commit and it was setting a non-existing symbol of
mbedTLS so those tests were basically never executed.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Only some test cases are skipped for which ECP_C is mandatory,
but the other ones are included.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Actually this adds both the accelerated test as well as the
reference. Both of them are used to evaluate the driver's
coverage with analyze_outcomes.py script.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Allow the script to also execute the tests needed for the following
analysis. It doesn't affect the previous usage of this script:
- if the output file is already present, then only the analysis
is performed
- if the outfile does not exists, then tests are also executed
before doing the analysis
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Split the part the varies between driver/built-in builds to a separate
file for convenience. Fix analyze_outcomes.py to be able to exclude
specific data files and not just a whole family at once.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
In the python script I didn't use the word TODO because pylint doesn't
like that, but morally it's the same.
I removed the comment about "do we need a subset of compat.sh?" because
it turns out that `ssl-opt.sh` is already exercising all the key
exchanges:
% sed -n 's/.*force_ciphersuite=TLS-\([^ ]*\)-WITH.*/\1/p' tests/ssl-opt.sh | sort -u
DHE-PSK
DHE-RSA
ECDH-ECDSA
ECDHE-ECDSA
ECDHE-PSK
ECDHE-RSA
ECJPAKE
PSK
RSA
RSA-PSK
(the only omission is ECDH-RSA which is not of interest here and does
not actually differ from ECDH-ECDSA). So, we don't need a subset of
compat.sh because we're already getting enough testing from ssl-opt.sh
(not to mention test_suite_ssl).
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
The part "driver: skipped/failed, reference: passed" didn't add any
information, but used up space on the screen and made the output
slightly harder to parse.
OTOH, now that we have multiple analyze_vs_reference tasks, we
should print out which one we're doing, so that that output makes sense
in case of a failure on the CI (which runs all tasks).
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
For now, ignore test suites that don't have parity even is they should.
The purpose is just to prepare the infrastructure and map the work.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Use a dictionary to specify optional parameters for each task.
If the task is not specified then all tasks are executed.
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
Check that every available test case in the test suites and ssl-opt.sh
has been executed at least once.
For the time being, only report a warning, because our coverage is
incomplete. Once we've updated all.sh to have full coverage, this
warning should become an error.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is a new script designed to analyze test outcomes collected
during a whole CI run.
This commit introduces the script, the code to read the outcome file,
and a very simple framework to report errors. It does not perform any
actual analysis yet.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
