etl/docs/docker.md

Docker for Development

Overview

The ETL repository ships a set of Docker-based development environments under .devcontainer/. They give every contributor an identical, reproducible toolchain regardless of host operating system. Three flavours are provided:

Flavour	Path	Purpose
Default	.devcontainer/	Day-to-day development (Microsoft C++ dev-container base image)
Compiler-specific	.devcontainer/gcc09/ … .devcontainer/gcc15/, .devcontainer/clang7/ … .devcontainer/clang21/	Test against a specific GCC or Clang version
s390x big-endian	.devcontainer/s390x/	Cross-compile and run tests on an s390x target via QEMU

All containers include CMake, Make, Git, Python 3, cogapp (the code generator used by ETL), clang-format 18, and treefmt.

Prerequisites

• Docker (or Docker Desktop) – any recent version that supports docker build and docker run.
• VS Code with the Dev Containers extension (optional, but recommended for the smoothest experience).
• Git – to clone the repository.

Quick Start

Using the helper script

The fastest way to get a shell inside the default container from the project root:

./scripts/run-docker.sh

This script performs two steps:

1. Builds the image from .devcontainer/Dockerfile and tags it etl.
2. Runs an interactive container that bind-mounts the repository at /home/vscode/etl so that edits made inside the container are visible on the host (and vice versa).

You are dropped into a Bash shell as the vscode user with the working directory set to the repository root.

Using VS Code Dev Containers

1. Open the repository folder in VS Code.
2. When prompted, click Reopen in Container – or run the command Dev Containers: Reopen in Container from the command palette.
3. VS Code reads .devcontainer/devcontainer.json, builds the image, and attaches to the running container automatically.

To open a specific compiler variant instead, run Dev Containers: Open Folder in Container… and pick the sub-folder (e.g. .devcontainer/gcc14/), or use the command palette action Dev Containers: Open Named Container Configuration… and select the desired name (e.g. "Gcc 14", "Clang 18").

Default Development Container

The file .devcontainer/Dockerfile is a multi-purpose image used by the default configuration and by every compiler-specific variant (they simply override the BASE_IMAGE_NAME build argument).

Base image

mcr.microsoft.com/devcontainers/cpp:2

The exact digest is pinned in devcontainer.json so that builds are reproducible even if the upstream tag is updated.

Installed tools

The Dockerfile installs the following on top of the base image:

Tool	Purpose
python3, pip	Runtime for helper scripts
python3-cogapp / cogapp	ETL code generator
git	Version control
wget	Downloading additional tooling
cmake, make	Build system
clang-format (v18)	Source formatting (see source-formatting.md)
treefmt (v2.4.1)	Single-command formatting wrapper

Reproducible builds with Debian snapshots

The default configuration sets the build argument DEBIAN_SNAPSHOT=20260223T000000Z. When this value is not "none", the Dockerfile rewrites the APT sources to point at snapshot.debian.org/archive/debian/<timestamp>, ensuring that every contributor installs identical package versions. Compiler-specific variants that are based on upstream gcc:* or silkeh/clang:* images set DEBIAN_SNAPSHOT=none because those images manage their own package sources.

Compiler-Specific Containers

Each sub-folder under .devcontainer/ contains a devcontainer.json that reuses the same Dockerfile (../Dockerfile) but overrides the BASE_IMAGE_NAME build argument to select a different compiler.

GCC variants

Folder	Base image	Name
gcc09/	gcc:9	Gcc 09
gcc10/	gcc:10	Gcc 10
gcc11/	gcc:11	Gcc 11
gcc12/	gcc:12	Gcc 12
gcc13/	gcc:13	Gcc 13
gcc14/	gcc:14	Gcc 14
gcc15/	gcc:15	Gcc 15

Clang variants

Folder	Base image	Name
clang7/	silkeh/clang:7	Clang 7
clang8/	silkeh/clang:8	Clang 8
clang9/	silkeh/clang:9	Clang 9
clang10/	silkeh/clang:10	Clang 10
clang11/	silkeh/clang:11	Clang 11
clang12/	silkeh/clang:12	Clang 12
clang13/	silkeh/clang:13	Clang 13
clang14/	silkeh/clang:14	Clang 14
clang15/	silkeh/clang:15	Clang 15
clang16/	silkeh/clang:16	Clang 16
clang17/	silkeh/clang:17	Clang 17
clang18/	silkeh/clang:18	Clang 18
clang19/	silkeh/clang:19	Clang 19
clang20/	silkeh/clang:20	Clang 20
clang21/	silkeh/clang:21	Clang 21

All compiler-specific variants set DEBIAN_SNAPSHOT to "none" because they rely on the upstream image's own package sources.

s390x Big-Endian Cross-Compilation

The s390x container lives in .devcontainer/s390x/ and has its own Dockerfile (it does not reuse the default one). It is based on debian:trixie and installs:

• QEMU user-mode emulation (qemu-user-static, qemu-user, binfmt-support)
• s390x cross-compilation toolchain (gcc-s390x-linux-gnu, g++-s390x-linux-gnu)
• CMake, Make, Ninja, Git, wget

Container setup

Open .devcontainer/s390x/ as a Dev Container in VS Code, or build manually:

docker build -t etl-s390x .devcontainer/s390x
docker run -it --rm -v .:/workspaces/etl -w /workspaces/etl etl-s390x

CMake toolchain

A CMake toolchain file is provided at .devcontainer/s390x/toolchain-s390x.cmake. It sets:

• CMAKE_SYSTEM_PROCESSOR to s390x
• Cross-compilers s390x-linux-gnu-gcc / g++
• CMAKE_CROSSCOMPILING_EMULATOR to /usr/bin/qemu-s390x-static

The VS Code Dev Container configuration already passes this toolchain file via cmake.configureArgs, so CMake Tools picks it up automatically.

Running tests under QEMU

Because the toolchain file sets CMAKE_CROSSCOMPILING_EMULATOR, CTest automatically invokes qemu-s390x-static when running test binaries. No extra flags are needed:

cmake -S test -B build-s390x \
  -DCMAKE_TOOLCHAIN_FILE=.devcontainer/s390x/toolchain-s390x.cmake \
  -DBUILD_TESTS=ON -DNO_STL=OFF -DETL_CXX_STANDARD=17 -G Ninja
cmake --build build-s390x
ctest --test-dir build-s390x

Building and Running Tests

Once inside any container (default, compiler-specific, or s390x) you can build and run the ETL test suite.

Quick CMake workflow

# Configure – build tests with C++17
cmake -S test -B build -DBUILD_TESTS=ON -DETL_CXX_STANDARD=17

# Build
cmake --build build -j $(nproc)

# Run tests
ctest --test-dir build

Change DETL_CXX_STANDARD to 11, 14, 17, 20, or 23 as needed. Add -DNO_STL=ON to build without the standard library.

Using the run-tests script

The repository also provides a convenience script in test/:

cd test
./run-tests.sh <standard> [optimisation] [threads] [sanitizer] [compiler] [verbose]

Argument	Values	Default
C++ standard	11, 14, 17, 20, 23	(required)
Optimisation	0, 1, 2, 3	0
Threads	any integer	4
Sanitizer	s (enable) / n (disable)	n
Compiler	gcc / clang	all
Verbose	v (enable) / n (disable)	n

Example – run C++17 tests at -O2 with 8 threads using GCC:

./run-tests.sh 17 2 8 n gcc n

Formatting Inside the Container

The default container ships with clang-format 18 and treefmt. See source-formatting.md for the full formatting guide.

Quick reference:

# Format all tracked C/C++ files with treefmt
treefmt

# Or use clang-format directly via the wrapper
./scripts/clang-format-wrapper -i include/etl/*.h

The wrapper script scripts/clang-format-wrapper resolves the correct clang-format binary (prefers clang-format-18, falls back to clang-format after checking the major version).

Customisation

To add extra packages or tools to the default container, edit .devcontainer/Dockerfile. The image follows a straightforward apt-get install pattern, so adding a new package is as simple as appending it to the existing apt-get line.

To create a new compiler variant:

1. Create a folder under .devcontainer/ (e.g. .devcontainer/gcc16/).
2. Add a devcontainer.json that references "../Dockerfile" and sets BASE_IMAGE_NAME to the desired image (e.g. gcc:16).
3. Set DEBIAN_SNAPSHOT to "none" for upstream compiler images.

Example:

{
  "name": "Gcc 16",
  "build": {
    "dockerfile": "../Dockerfile",
    "args": {
      "BASE_IMAGE_NAME": "gcc:16",
      "DEBIAN_SNAPSHOT": "none"
    },
    "context": "../context"
  }
}

Troubleshooting

Symptom	Cause / Fix
apt-get fails with "Release file … is not valid yet"	The Debian snapshot timestamp is in the future relative to the build host clock. Either update DEBIAN_SNAPSHOT in devcontainer.json or set it to "none".
clang-format reports the wrong version	The wrapper expects version 18. Make sure the image installs clang-format (or clang-format-18) and that the binary is on PATH.
Permission errors on mounted files	The run-docker.sh script runs as user vscode. Ensure your host UID matches, or adjust the --user flag.
s390x tests crash immediately	Verify that qemu-user-static is installed and that binfmt-support is active. On some hosts you may need to register binfmt handlers with docker run --privileged --rm tonistiigi/binfmt --install all.
Build is very slow the first time	Docker is downloading and building the image from scratch. Subsequent builds use the layer cache and are much faster.