etl/docs/source/testing.md

title	weight
Testing the ETL	6

This document describes how to build and run the ETL test suite locally, inside Dev Containers, and in CI.

Table of Contents

1. Prerequisites
2. Running Tests Locally (test/run-tests.sh)
3. Syntax Checks (test/run-syntax-checks.sh)
4. Cross-Architecture Testing (.devcontainer/run-tests.sh)
5. Dev Containers for Native Compilers
6. CMake Options Reference
7. CI Checks (GitHub Actions)
8. Appveyor (Windows / MSVC)
9. Code Coverage
10. Generator Tests (scripts/generator_test.py)

---

Prerequisites

• CMake ≥ 3.10
• GCC and/or Clang (any version supported by the project)
• Make or Ninja (build backend)
• Docker (only needed for cross-architecture testing via .devcontainer/run-tests.sh)
• QEMU user-mode (installed automatically inside the cross-arch Docker images)

The project is header-only, so there is no library to compile – the build step compiles the test binary etl_tests which links against a bundled copy of UnitTest++.

---

Running Tests Locally

The main entry point for local testing is test/run-tests.sh. It iterates over a matrix of compiler / configuration combinations, creates a temporary build-make directory for each one, runs CMake + Make + CTest, and reports coloured pass/fail output (also appended to log.txt).

Synopsis

cd test
./run-tests.sh <C++ Standard> [Optimisation] [Threads] [Sanitizer] [Compiler] [Verbose]

Argument	Values	Default
C++ Standard	11, 14, 17, 20, 23, or all	(required)
Optimisation	0, 1, 2, 3	0
Threads	any positive integer	4
Sanitizer	s (enable) / n (disable)	n (disabled)
Compiler	gcc, clang	all compilers
Verbose	v (enable) / n (disable)	n (disabled)

Examples

# Run all C++17 tests with GCC only, optimisation -O0, 8 threads
./run-tests.sh 17 0 8 n gcc

# Run every standard with both compilers, sanitizers enabled, verbose
./run-tests.sh all 0 4 s "" v

What the script does

For every selected C++ standard the script loops over a built-in list of configurations (STL / No STL / Force C++03 / Non-virtual messages / …) for each selected compiler. For every combination it:

1. Creates a fresh build-make directory inside the configuration's source subdirectory.
2. Invokes cmake with the appropriate -D flags (see CMake Options Reference).
3. Builds via cmake --build . (parallel level controlled by CMAKE_BUILD_PARALLEL_LEVEL).
4. Runs ctest -V.
5. Reports success or failure and removes the build directory.

The script exits immediately on the first compilation or test failure.

Test configurations exercised

Compiler	Configuration
GCC	STL
GCC	STL – Non-virtual messages
GCC	STL – Force C++03
GCC	No STL
GCC	No STL – Force C++03
GCC	No STL – Builtin mem functions
Clang	STL
Clang	STL – Force C++03
Clang	No STL
Clang	No STL – Force C++03
Clang	No STL – Builtin mem functions
GCC / Clang	Initializer list test
GCC / Clang	Error macros – log_errors, exceptions, log_errors_and_exceptions, assert_function

---

Syntax Checks

The script test/run-syntax-checks.sh performs compilation-only syntax checks across multiple C++ standards and configurations. Unlike run-tests.sh, it does not run the test binary – it only verifies that the code compiles successfully. This is useful for quickly validating that header changes do not introduce compilation errors across the supported standard/configuration matrix.

Synopsis

cd test
./run-syntax-checks.sh <C++ Standard> [Threads] [Compiler]

Argument	Values	Default
C++ Standard	03, 11, 14, 17, 20, 23, or a (all)	(required)
Threads	any positive integer	4
Compiler	gcc, clang	all compilers

Examples

# Check C++17 syntax with GCC only, using 8 threads
./run-syntax-checks.sh 17 8 gcc

# Check all standards with both compilers
./run-syntax-checks.sh a

What the script does

The script operates from the test/syntax_check directory and iterates over the selected C++ standard(s). For each standard and compiler combination it:

1. Creates fresh build directories (bgcc / bclang).
2. Invokes cmake with the appropriate -D flags for the configuration.
3. Builds via cmake --build.
4. Reports compilation success or failure (logged to log.txt).

The script exits immediately on the first compilation failure.

Configurations checked per standard

For each C++ standard the following configurations are compiled:

Compiler	Configuration
GCC	STL
GCC	No STL
GCC	STL – Built-in traits
GCC	No STL – Built-in traits
Clang	STL
Clang	No STL
Clang	STL – Built-in traits
Clang	No STL – Built-in traits

---

Cross-Architecture Testing

.devcontainer/run-tests.sh builds and runs the test suite for non-x86_64 architectures using Docker and QEMU user-mode emulation. It is designed to be run from the project root.

Supported architectures

Argument	Target	Endianness	QEMU binary
armhf	ARM hard-float (32-bit)	Little	qemu-arm-static
i386	x86 32-bit	Little	qemu-i386-static
powerpc	PowerPC 32-bit	Big	qemu-ppc
riscv64	RISC-V 64-bit	Little	qemu-riscv64-static
s390x	IBM Z (64-bit)	Big	qemu-s390x-static

Synopsis

# From the project root
.devcontainer/run-tests.sh <architecture>

How it works

The script has two phases controlled by a second (internal) argument:

1. Outside the container (no second argument):

   • Builds a Docker image from .devcontainer/<arch>/Dockerfile.
   • Starts a container, bind-mounting the project at /workspaces/etl.
   • Re-invokes itself inside the container with the inside_container flag.

2. Inside the container (inside_container):

   • Creates build-<arch> and runs CMake with the appropriate cross- compilation toolchain file (.devcontainer/<arch>/toolchain-<arch>.cmake).
   • Builds with cmake --build . using all available cores.
   • Runs the test suite via ctest --output-on-failure.

The toolchain files set CMAKE_CROSSCOMPILING_EMULATOR so that CTest can run the binary transparently through QEMU.

Example

# Build & run the armhf test suite
.devcontainer/run-tests.sh armhf

The cross-arch containers build with the following fixed settings:

• C++23, No STL, sanitizer off, optimisation -O0.

---

Dev Containers for Native Compilers

The .devcontainer/ directory also provides Dev Container definitions for a wide range of native (x86_64) compiler versions. These are intended for use with VS Code Dev Containers or GitHub Codespaces.

Directory	Compiler
gcc09 – gcc15	GCC 9 through 15
clang7 – clang21	Clang 7 through 21

Each subdirectory contains a devcontainer.json that references the shared Dockerfile (.devcontainer/Dockerfile) and passes the appropriate base Docker image via the BASE_IMAGE_NAME build argument (e.g. gcc:15).

The default Dev Container (.devcontainer/devcontainer.json) uses the Microsoft C++ dev-container base image.

To use one of these containers:

1. Open the repository in VS Code.
2. Ctrl+Shift+P → Dev Containers: Reopen in Container and select the desired configuration (e.g. Gcc 15).
3. Use test/run-tests.sh inside the container as described above.

---

CMake Options Reference

When invoking CMake for the test suite (source directory is test/), the following -D options control the build:

Option	Type	Description
BUILD_TESTS	BOOL	Must be ON to compile the test binary.
NO_STL	BOOL	Build without the C++ Standard Library.
ETL_CXX_STANDARD	STRING	C++ standard: 11, 14, 17, 20, 23.
ETL_OPTIMISATION	STRING	Compiler optimisation flag, e.g. -O0.
ETL_ENABLE_SANITIZER	BOOL	Enable address / undefined-behaviour sanitizers.
ETL_USE_TYPE_TRAITS_BUILTINS	BOOL	Use compiler built-in type traits.
ETL_USER_DEFINED_TYPE_TRAITS	BOOL	Use user-defined type traits.
ETL_FORCE_TEST_CPP03_IMPLEMENTATION	BOOL	Force the C++03 code paths even on newer standards.
ETL_MESSAGES_ARE_NOT_VIRTUAL	BOOL	Use non-virtual message types.
ETL_USE_BUILTIN_MEM_FUNCTIONS	BOOL	Use built-in memory functions in No-STL mode.
CMAKE_TOOLCHAIN_FILE	PATH	Toolchain file for cross-compilation.

Minimal manual build example

cd test
mkdir build && cd build
cmake -DBUILD_TESTS=ON -DNO_STL=OFF -DETL_CXX_STANDARD=20 ..
cmake --build . -j$(nproc)
ctest -V

---

CI Checks (GitHub Actions)

Every push or pull request to master, development, or pull-request/* branches triggers a comprehensive set of GitHub Actions workflows defined in .github/workflows/.

Workflow matrix

Workflow file	Compiler	Standard	Notes
gcc-c++11.yml	GCC	C++11	STL, No STL, Force C++03
gcc-c++14.yml	GCC	C++14	STL, No STL, Force C++03
gcc-c++17.yml	GCC	C++17	STL, No STL, Force C++03
gcc-c++20.yml	GCC	C++20	STL, No STL, Force C++03
gcc-c++23.yml	GCC	C++23	STL, No STL, Force C++03
clang-c++11.yml	Clang	C++11	STL, No STL, Force C++03
clang-c++14.yml	Clang	C++14	STL, No STL, Force C++03
clang-c++17.yml	Clang	C++17	STL, No STL, Force C++03
clang-c++20.yml	Clang	C++20	STL, No STL, Force C++03
clang-c++23.yml	Clang	C++23	STL, No STL, Force C++03
gcc-syntax-checks.yml	GCC	C++03 – C++23	Compilation-only syntax checks (no tests run)
clang-syntax-checks.yml	Clang	C++03 – C++23	Compilation-only syntax checks (no tests run)
msvc.yml	MSVC 2022	C++17	Windows, STL & No STL
gcc-c++23-armhf.yml	GCC cross	C++23	armhf via QEMU
gcc-c++23-i386.yml	GCC cross	C++23	i386 via QEMU
gcc-c++23-powerpc.yml	GCC cross	C++23	powerpc via QEMU
gcc-c++23-riscv64.yml	GCC cross	C++23	RISC-V 64 via QEMU
gcc-c++23-s390x.yml	GCC cross	C++23	s390x via QEMU
coverage.yml	GCC	—	Generates lcov coverage report, deploys to GitHub Pages
generator.yml	—	—	Runs the code generator
platformio-update.yml	—	—	PlatformIO registry update

Typical CI job structure

Each compiler/standard workflow follows the same pattern:

1. Checkout – actions/checkout@v4.
2. Build – set CC/CXX, call cmake with the appropriate -D flags, then make -j.
3. Run tests – execute ./test/etl_tests -v (or ctest -V for cross- arch jobs).

The cross-architecture CI jobs additionally install a cross-compiler toolchain and QEMU inside a debian:trixie container, use the matching toolchain file from .devcontainer/<arch>/, and run tests via CTest (which delegates to QEMU through CMAKE_CROSSCOMPILING_EMULATOR).

Branches tested

• master
• development
• pull-request/*

All workflows run on both push and pull_request events (types: opened, synchronize, reopened).

---

Appveyor (Windows / MSVC)

The appveyor.yml at the repository root provides additional Windows CI using Visual Studio 2022. It builds the master branch only.

Configurations tested:

• Debug MSVC C++14
• Debug MSVC C++14 – No STL
• Debug MSVC C++17
• Debug MSVC C++17 – No STL
• Debug MSVC C++20
• Debug MSVC C++20 – No STL

The build uses the VS 2022 solution file at test/vs2022/etl.vcxproj.

---

Code Coverage

The coverage.yml GitHub Actions workflow generates an lcov coverage report:

1. Runs test/run-coverage.sh which builds and tests with GCC coverage flags.
2. Uploads the HTML report as a build artifact (retained for 30 days).
3. On pushes to master, deploys the report to GitHub Pages.

To generate coverage locally:

cd test
./run-coverage.sh
# Open test/build-coverage/coverage/index.html

---

Generator Tests

The script scripts/generator_test.py verifies that the code generators in include/etl/generators/ produce output matching the checked-in header files in include/etl/private/.

ETL uses Cog to generate certain repetitive header files (e.g. delegate.h, message_packet.h). The generator templates live in include/etl/generators/*_generator.h and the generated output is committed to include/etl/private/*.h. This test ensures the two stay in sync.

Prerequisites

• Python 3
• cogapp – install via pip install cogapp

Synopsis

python3 scripts/generator_test.py

What the script does

1. Iterates over every *_generator.h file in include/etl/generators/.
2. Runs Cog on each generator, outputting to build/generator_tmp/.
3. Compares each generated file against the corresponding file in include/etl/private/.
4. Reports success if all files match, or failure if any differ.

The script exits with code 0 on success and 1 on failure.

CI integration

The generator.yml GitHub Actions workflow runs this script automatically on pushes and pull requests to verify generator consistency.