coffee/etl
1
0
Fork 0
mirror of https://github.com/ETLCPP/etl.git synced 2026-04-30 19:09:10 +08:00
Code Issues Packages Projects Releases Wiki Activity
etl/docs/testing.md
Roland Reichwein 866c8a315e
Extensions for testing (#1380) 
* Extensions for testing

Generalize run-tests.sh

Test all C++ versions at once

Fix combination of big endian and -Wsign-conversion

Failed on s390x (as reference for big endian)

Add github workflow for s390x

Add armhf container files

Devcontainers for i386 and riscv

Add github workflows for armhf, i386 and riscv64

Add run-tests.sh for foreign architectures

Document testing in doc/testing.md

Adjustments from clang-format run

Fix .devcontainer/s390x/Dockerfile for linebreak syntax

Fix exit code of run-test.sh

Previously, "exit $?" was used, actually the return value of
FailedCompilation and FailedTest which are always 0.

Now just using 1.

In run-tests.sh at ctest, use -V for printing number of tests unconditionally

While ctest suppresses individual test list by default, it didn't even
print the number of tests anymore, as run_tests.sh does because
it suppresses it output completely.

Now, by default print number of tests, and in verbose mode, print test list
in addition.

* Support powerpc as foreign architecture

* Add SFINAE constraints to etl::begin/end and reverse iterator free functions

The unconstrained etl::begin(), etl::end(), etl::cbegin(), etl::cend(),
etl::rbegin(), etl::rend(), etl::crbegin(), and etl::crend() templates
in the no-STL code path were matching iterator types during ADL, causing
a hard error with GCC 15's std::ranges::begin. When std::ranges performed
ADL on an etl::*::iterator, it found etl::begin() as a candidate; since
the iterator type has a nested iterator typedef, the return type TContainer::iterator
was valid, but calling .begin() on the iterator failed.

Fix: add etl::void_t<decltype(...)> SFINAE guards to each template,
ensuring they only participate in overload resolution when TContainer
actually has the corresponding member function (.begin(), .end(), etc.).

* - Fix red unit tests on 32 bits big-endian platform.

* Document powerpc architecture for testing

* Use Dockerfiles in cross testing github workflows

Synchronizes environment setup for github workflows to what is
defined in the development Dockerfiles. So they don't need to
be maintained separately.

---------

Co-authored-by: John Wellbelove <john.wellbelove@etlcpp.com>
Co-authored-by: Sergei Shirokov <sergej.shirokov@gmail.com>
Co-authored-by: John Wellbelove <jwellbelove@users.noreply.github.com>
2026-04-15 10:47:52 +02:00

413 lines
14 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Testing ETL
This document describes how to build and run the ETL test suite locally,
inside Dev Containers, and in CI.
## Table of Contents
1. [Prerequisites](#prerequisites)
2. [Running Tests Locally (`test/run-tests.sh`)](#running-tests-locally)
3. [Syntax Checks (`test/run-syntax-checks.sh`)](#syntax-checks)
4. [Cross-Architecture Testing (`.devcontainer/run-tests.sh`)](#cross-architecture-testing)
5. [Dev Containers for Native Compilers](#dev-containers-for-native-compilers)
6. [CMake Options Reference](#cmake-options-reference)
7. [CI Checks (GitHub Actions)](#ci-checks-github-actions)
8. [Appveyor (Windows / MSVC)](#appveyor-windows--msvc)
9. [Code Coverage](#code-coverage)
10. [Generator Tests (`scripts/generator_test.py`)](#generator-tests)
---
## Prerequisites
* **CMake** ≥ 3.10
* **GCC** and/or **Clang** (any version supported by the project)
* **Make** or **Ninja** (build backend)
* **Docker** (only needed for cross-architecture testing via `.devcontainer/run-tests.sh`)
* **QEMU user-mode** (installed automatically inside the cross-arch Docker images)
The project is header-only, so there is no library to compile – the build
step compiles the test binary `etl_tests` which links against a bundled copy
of **UnitTest++**.
---
## Running Tests Locally
The main entry point for local testing is **`test/run-tests.sh`**. It
iterates over a matrix of compiler / configuration combinations, creates a
temporary `build-make` directory for each one, runs CMake + Make + CTest,
and reports coloured pass/fail output (also appended to `log.txt`).
### Synopsis
```bash
cd test
./run-tests.sh <C++ Standard> [Optimisation] [Threads] [Sanitizer] [Compiler] [Verbose]
```
| Argument | Values | Default |
|---|---|---|
| C++ Standard | `11`, `14`, `17`, `20`, `23`, or `all` | *(required)* |
| Optimisation | `0`, `1`, `2`, `3` | `0` |
| Threads | any positive integer | `4` |
| Sanitizer | `s` (enable) / `n` (disable) | `n` (disabled) |
| Compiler | `gcc`, `clang` | all compilers |
| Verbose | `v` (enable) / `n` (disable) | `n` (disabled) |
### Examples
```bash
# Run all C++17 tests with GCC only, optimisation -O0, 8 threads
./run-tests.sh 17 0 8 n gcc
# Run every standard with both compilers, sanitizers enabled, verbose
./run-tests.sh all 0 4 s "" v
```
### What the script does
For every selected C++ standard the script loops over a built-in list of
configurations (STL / No STL / Force C++03 / Non-virtual messages / …) for
each selected compiler. For every combination it:
1. Creates a fresh `build-make` directory inside the configuration's source
subdirectory.
2. Invokes `cmake` with the appropriate `-D` flags (see
[CMake Options Reference](#cmake-options-reference)).
3. Builds via `cmake --build .` (parallel level controlled by
`CMAKE_BUILD_PARALLEL_LEVEL`).
4. Runs `ctest -V`.
5. Reports success or failure and removes the build directory.
The script exits immediately on the first compilation or test failure.
### Test configurations exercised
| Compiler | Configuration |
|---|---|
| GCC | STL |
| GCC | STL – Non-virtual messages |
| GCC | STL – Force C++03 |
| GCC | No STL |
| GCC | No STL – Force C++03 |
| GCC | No STL – Builtin mem functions |
| Clang | STL |
| Clang | STL – Force C++03 |
| Clang | No STL |
| Clang | No STL – Force C++03 |
| Clang | No STL – Builtin mem functions |
| GCC / Clang | Initializer list test |
| GCC / Clang | Error macros – log_errors, exceptions, log_errors_and_exceptions, assert_function |
---
## Syntax Checks
The script **`test/run-syntax-checks.sh`** performs compilation-only syntax
checks across multiple C++ standards and configurations. Unlike
`run-tests.sh`, it **does not run the test binary** – it only verifies that
the code compiles successfully. This is useful for quickly validating that
header changes do not introduce compilation errors across the supported
standard/configuration matrix.
### Synopsis
```bash
cd test
./run-syntax-checks.sh <C++ Standard> [Threads] [Compiler]
```
| Argument | Values | Default |
|---|---|---|
| C++ Standard | `03`, `11`, `14`, `17`, `20`, `23`, or `a` (all) | *(required)* |
| Threads | any positive integer | `4` |
| Compiler | `gcc`, `clang` | all compilers |
### Examples
```bash
# Check C++17 syntax with GCC only, using 8 threads
./run-syntax-checks.sh 17 8 gcc
# Check all standards with both compilers
./run-syntax-checks.sh a
```
### What the script does
The script operates from the `test/syntax_check` directory and iterates over
the selected C++ standard(s). For each standard and compiler combination it:
1. Creates fresh build directories (`bgcc` / `bclang`).
2. Invokes `cmake` with the appropriate `-D` flags for the configuration.
3. Builds via `cmake --build`.
4. Reports compilation success or failure (logged to `log.txt`).
The script exits immediately on the first compilation failure.
### Configurations checked per standard
For each C++ standard the following configurations are compiled:
| Compiler | Configuration |
|---|---|
| GCC | STL |
| GCC | No STL |
| GCC | STL – Built-in traits |
| GCC | No STL – Built-in traits |
| Clang | STL |
| Clang | No STL |
| Clang | STL – Built-in traits |
| Clang | No STL – Built-in traits |
---
## Cross-Architecture Testing
**`.devcontainer/run-tests.sh`** builds and runs the test suite for
non-x86_64 architectures using Docker and QEMU user-mode emulation. It is
designed to be run **from the project root**.
### Supported architectures
| Argument | Target | Endianness | QEMU binary |
|---|---|---|---|
| `armhf` | ARM hard-float (32-bit) | Little | `qemu-arm-static` |
| `i386` | x86 32-bit | Little | `qemu-i386-static` |
| `powerpc` | PowerPC 32-bit | Big | `qemu-ppc` |
| `riscv64` | RISC-V 64-bit | Little | `qemu-riscv64-static` |
| `s390x` | IBM Z (64-bit) | Big | `qemu-s390x-static` |
### Synopsis
```bash
# From the project root
.devcontainer/run-tests.sh <architecture>
```
### How it works
The script has two phases controlled by a second (internal) argument:
1. **Outside the container** (no second argument):
* Builds a Docker image from `.devcontainer/<arch>/Dockerfile`.
* Starts a container, bind-mounting the project at `/workspaces/etl`.
* Re-invokes itself *inside* the container with the `inside_container`
flag.
2. **Inside the container** (`inside_container`):
* Creates `build-<arch>` and runs CMake with the appropriate cross-
compilation toolchain file
(`.devcontainer/<arch>/toolchain-<arch>.cmake`).
* Builds with `cmake --build .` using all available cores.
* Runs the test suite via `ctest --output-on-failure`.
The toolchain files set `CMAKE_CROSSCOMPILING_EMULATOR` so that CTest can
run the binary transparently through QEMU.
### Example
```bash
# Build & run the armhf test suite
.devcontainer/run-tests.sh armhf
```
The cross-arch containers build with the following fixed settings:
* C++23, No STL, sanitizer off, optimisation -O0.
---
## Dev Containers for Native Compilers
The `.devcontainer/` directory also provides Dev Container definitions for a
wide range of **native** (x86_64) compiler versions. These are intended for
use with **VS Code Dev Containers** or **GitHub Codespaces**.
| Directory | Compiler |
|---|---|
| `gcc09` – `gcc15` | GCC 9 through 15 |
| `clang7` – `clang21` | Clang 7 through 21 |
Each subdirectory contains a `devcontainer.json` that references the shared
`Dockerfile` (`.devcontainer/Dockerfile`) and passes the appropriate base
Docker image via the `BASE_IMAGE_NAME` build argument (e.g. `gcc:15`).
The default Dev Container (`.devcontainer/devcontainer.json`) uses the
Microsoft C++ dev-container base image.
To use one of these containers:
1. Open the repository in VS Code.
2. **Ctrl+Shift+P → Dev Containers: Reopen in Container** and select the
desired configuration (e.g. *Gcc 15*).
3. Use `test/run-tests.sh` inside the container as described above.
---
## CMake Options Reference
When invoking CMake for the test suite (source directory is `test/`), the
following `-D` options control the build:
| Option | Type | Description |
|---|---|---|
| `BUILD_TESTS` | `BOOL` | Must be `ON` to compile the test binary. |
| `NO_STL` | `BOOL` | Build without the C++ Standard Library. |
| `ETL_CXX_STANDARD` | `STRING` | C++ standard: `11`, `14`, `17`, `20`, `23`. |
| `ETL_OPTIMISATION` | `STRING` | Compiler optimisation flag, e.g. `-O0`. |
| `ETL_ENABLE_SANITIZER` | `BOOL` | Enable address / undefined-behaviour sanitizers. |
| `ETL_USE_TYPE_TRAITS_BUILTINS` | `BOOL` | Use compiler built-in type traits. |
| `ETL_USER_DEFINED_TYPE_TRAITS` | `BOOL` | Use user-defined type traits. |
| `ETL_FORCE_TEST_CPP03_IMPLEMENTATION` | `BOOL` | Force the C++03 code paths even on newer standards. |
| `ETL_MESSAGES_ARE_NOT_VIRTUAL` | `BOOL` | Use non-virtual message types. |
| `ETL_USE_BUILTIN_MEM_FUNCTIONS` | `BOOL` | Use built-in memory functions in No-STL mode. |
| `CMAKE_TOOLCHAIN_FILE` | `PATH` | Toolchain file for cross-compilation. |
### Minimal manual build example
```bash
cd test
mkdir build && cd build
cmake -DBUILD_TESTS=ON -DNO_STL=OFF -DETL_CXX_STANDARD=20 ..
cmake --build . -j$(nproc)
ctest -V
```
---
## CI Checks (GitHub Actions)
Every push or pull request to `master`, `development`, or `pull-request/*`
branches triggers a comprehensive set of GitHub Actions workflows defined in
`.github/workflows/`.
### Workflow matrix
| Workflow file | Compiler | Standard | Notes |
|---|---|---|---|
| `gcc-c++11.yml` | GCC | C++11 | STL, No STL, Force C++03 |
| `gcc-c++14.yml` | GCC | C++14 | STL, No STL, Force C++03 |
| `gcc-c++17.yml` | GCC | C++17 | STL, No STL, Force C++03 |
| `gcc-c++20.yml` | GCC | C++20 | STL, No STL, Force C++03 |
| `gcc-c++23.yml` | GCC | C++23 | STL, No STL, Force C++03 |
| `clang-c++11.yml` | Clang | C++11 | STL, No STL, Force C++03 |
| `clang-c++14.yml` | Clang | C++14 | STL, No STL, Force C++03 |
| `clang-c++17.yml` | Clang | C++17 | STL, No STL, Force C++03 |
| `clang-c++20.yml` | Clang | C++20 | STL, No STL, Force C++03 |
| `clang-c++23.yml` | Clang | C++23 | STL, No STL, Force C++03 |
| `gcc-syntax-checks.yml` | GCC | C++03 – C++23 | Compilation-only syntax checks (no tests run) |
| `clang-syntax-checks.yml` | Clang | C++03 – C++23 | Compilation-only syntax checks (no tests run) |
| `msvc.yml` | MSVC 2022 | C++17 | Windows, STL & No STL |
| `gcc-c++23-armhf.yml` | GCC cross | C++23 | armhf via QEMU |
| `gcc-c++23-i386.yml` | GCC cross | C++23 | i386 via QEMU |
| `gcc-c++23-powerpc.yml` | GCC cross | C++23 | powerpc via QEMU |
| `gcc-c++23-riscv64.yml` | GCC cross | C++23 | RISC-V 64 via QEMU |
| `gcc-c++23-s390x.yml` | GCC cross | C++23 | s390x via QEMU |
| `coverage.yml` | GCC | — | Generates lcov coverage report, deploys to GitHub Pages |
| `generator.yml` | — | — | Runs the code generator |
| `platformio-update.yml` | — | — | PlatformIO registry update |
### Typical CI job structure
Each compiler/standard workflow follows the same pattern:
1. **Checkout** – `actions/checkout@v4`.
2. **Build** – set `CC`/`CXX`, call `cmake` with the appropriate `-D` flags,
then `make -j`.
3. **Run tests** – execute `./test/etl_tests -v` (or `ctest -V` for cross-
arch jobs).
The cross-architecture CI jobs additionally install a cross-compiler
toolchain and QEMU inside a `debian:trixie` container, use the matching
toolchain file from `.devcontainer/<arch>/`, and run tests via CTest (which
delegates to QEMU through `CMAKE_CROSSCOMPILING_EMULATOR`).
### Branches tested
* `master`
* `development`
* `pull-request/*`
All workflows run on both `push` and `pull_request` events (types: opened,
synchronize, reopened).
---
## Appveyor (Windows / MSVC)
The `appveyor.yml` at the repository root provides additional Windows CI
using **Visual Studio 2022**. It builds the `master` branch only.
Configurations tested:
* Debug MSVC C++14
* Debug MSVC C++14 – No STL
* Debug MSVC C++17
* Debug MSVC C++17 – No STL
* Debug MSVC C++20
* Debug MSVC C++20 – No STL
The build uses the VS 2022 solution file at `test/vs2022/etl.vcxproj`.
---
## Code Coverage
The `coverage.yml` GitHub Actions workflow generates an **lcov** coverage
report:
1. Runs `test/run-coverage.sh` which builds and tests with GCC coverage
flags.
2. Uploads the HTML report as a build artifact (retained for 30 days).
3. On pushes to `master`, deploys the report to **GitHub Pages**.
To generate coverage locally:
```bash
cd test
./run-coverage.sh
# Open test/build-coverage/coverage/index.html
```
---
## Generator Tests
The script **`scripts/generator_test.py`** verifies that the code generators
in `include/etl/generators/` produce output matching the checked-in header
files in `include/etl/private/`.
ETL uses [Cog](https://nedbatchelder.com/code/cog/) to generate certain
repetitive header files (e.g. `delegate.h`, `message_packet.h`). The
generator templates live in `include/etl/generators/*_generator.h` and the
generated output is committed to `include/etl/private/*.h`. This test
ensures the two stay in sync.
### Prerequisites
* **Python 3**
* **cogapp** – install via `pip install cogapp`
### Synopsis
```bash
python3 scripts/generator_test.py
```
### What the script does
1. Iterates over every `*_generator.h` file in `include/etl/generators/`.
2. Runs Cog on each generator, outputting to `build/generator_tmp/`.
3. Compares each generated file against the corresponding file in
`include/etl/private/`.
4. Reports success if all files match, or failure if any differ.
The script exits with code `0` on success and `1` on failure.
### CI integration
The `generator.yml` GitHub Actions workflow runs this script automatically
on pushes and pull requests to verify generator consistency.
Reference in New Issue View Git Blame Copy Permalink