etl/docs/generators.md

Code Generation for Pre-C++11 Support

ETL supports C++03 (also referred to as C++98) environments where variadic templates, constexpr, and other modern features are unavailable. To provide equivalent functionality, certain headers are generated using Cog, a Python-based code generation tool that embeds Python snippets inside source files.

This document explains how the code generation system works and how to regenerate the headers if you modify a generator template.

---

Overview

Directory	Contents
include/etl/generators/	Generator templates (*_generator.h) and batch scripts
include/etl/private/	Generated output (*_cpp03.h) committed to the repository
scripts/generator_test.py	CI script that verifies generators match committed files

The generator templates contain embedded Python code (delimited by [[[cog and ]]]) that produces the repetitive C++03 boilerplate. Cog processes these templates and writes the expanded output to include/etl/private/.

---

Generated Headers

The following C++03 compatibility headers are generated:

Generator	Output	Purpose
fsm_fwd_decl_cpp03_generator.h	fsm_fwd_decl_cpp03.h	FSM forward declarations
fsm_friend_decl_cpp03_generator.h	fsm_friend_decl_cpp03.h	FSM friend declarations
fsm_cpp03_generator.h	fsm_cpp03.h	Finite state machine implementation
message_router_cpp03_generator.h	message_router_cpp03.h	Message router
message_packet_cpp03_generator.h	message_packet_cpp03.h	Message packet
largest_type_cpp03_generator.h	largest_type_cpp03.h	Largest type metafunction
largest_alignment_cpp03_generator.h	largest_alignment_cpp03.h	Largest alignment metafunction
largest_cpp03_generator.h	largest_cpp03.h	Largest type/size utilities
smallest_cpp03_generator.h	smallest_cpp03.h	Smallest type/size utilities
type_traits_cpp03_generator.h	type_traits_cpp03.h	Type traits (is_one_of, etc.)
type_lookup_cpp03_generator.h	type_lookup_cpp03.h	Type lookup metafunction
type_select_cpp03_generator.h	type_select_cpp03.h	Type selection metafunction
variant_pool_cpp03_generator.h	variant_pool_cpp03.h	Variant pool

---

Generator Parameters

Cog variables control how many template parameter overloads are generated:

Variable	Default	Used by
Handlers	16	FSM and message router generators
NTypes	16	Type utility generators (largest, smallest, lookup, select, variant pool)
IsOneOf	16	Type traits generator (is_one_of)

These defaults produce overloads supporting up to 16 types or handlers, which is sufficient for most embedded applications while keeping compile times reasonable.

---

Prerequisites

• Python 3

• cogapp – install via:

  pip install cogapp
  

---

Regenerating Headers

Using the batch scripts (Windows)

Each generator has a corresponding .bat file in include/etl/generators/:

cd include/etl/generators
generate.bat          # Regenerate all headers
generate_fsm.bat      # Regenerate FSM headers only
generate_smallest.bat # Regenerate smallest_cpp03.h only
# etc.

Manual invocation

Run Cog directly from the include/etl/generators/ directory:

cd include/etl/generators

# Example: regenerate smallest_cpp03.h
python3 -m cogapp -d -e -o../private/smallest_cpp03.h -DNTypes=16 smallest_cpp03_generator.h

# Example: regenerate fsm_cpp03.h
python3 -m cogapp -d -e -o../private/fsm_cpp03.h -DHandlers=16 fsm_cpp03_generator.h

Cog options used:

Option	Meaning
-d	Delete the generator markers from output
-e	Warn if the input file has no generator markers
-o<file>	Write output to the specified file
-D<var>=<value>	Define a Cog variable

Regenerating all headers

The generate.bat script regenerates every header:

cd include/etl/generators
./generate.bat   # Windows
# or run the commands manually on Linux/macOS

On Linux/macOS you can run the commands from generate.bat directly in your shell (they are standard python3 -m cogapp invocations).

---

Verifying Generators

After modifying a generator template, verify the output matches the committed file:

python3 scripts/generator_test.py

This script:

1. Runs Cog on every *_generator.h file.
2. Compares each output against the corresponding file in include/etl/private/.
3. Reports success or failure.

The generator.yml GitHub Actions workflow runs this automatically on every push and pull request.

---

How Generators Work

A generator template contains standard C++ code interspersed with Cog directives. For example, from smallest_cpp03_generator.h:

/*[[[cog
import cog
cog.outl("template <typename T1, ")
for n in range(2, int(NTypes)):
    cog.out("typename T%s = void, " % n)
cog.outl("typename T%s = void>" % int(NTypes))
]]]*/
// Generated code appears here after running Cog
/*[[[end]]]*/

When Cog processes this file with -DNTypes=16, the Python code executes and outputs the expanded template parameter list supporting 16 types.

---

Adding a New Generator

1. Create include/etl/generators/<name>_cpp03_generator.h with Cog directives.
2. Add a corresponding entry to generate.bat.
3. Run generate.bat (or the equivalent Cog command) to produce include/etl/private/<name>_cpp03.h.
4. Commit both the generator and the generated output.
5. Verify with python3 scripts/generator_test.py.

---

Troubleshooting

Problem	Solution
ModuleNotFoundError: No module named 'cogapp'	Install Cog: pip install cogapp
Generator output differs from committed file	Regenerate and commit the updated output
Need more than 16 types/handlers	Change -DNTypes= or -DHandlers= and regenerate