Keep the digit-separator feature off the default hot path

PR #369's "performance-neutral" claim did not hold: the default
(no-separator) parse was ~70% larger in the hot frame (parse_number_string
inlined into from_chars: 833 -> 1414 x86/arm64 instructions), because

  1. parse_number_string is force-inlined (always_inline), and the runtime
     dispatch inlined BOTH the has_separator==true and ==false instantiations
     into the default caller -- dragging the entire separator-aware scanner
     into the hot frame; and
  2. the runtime separator branch survived in from_chars() even though a
     default-constructed parse_options provably has no separator, because the
     thin from_chars_caller forwarder was not inlined, so the compile-time
     '\0' was lost across the call boundary and the branch (plus two calls)
     could not be folded away.

Fixes:
  * FASTFLOAT_NOINLINE + a cold parse_number_string_with_separator trampoline
    so the separator instantiation stays strictly out of line; the default
    caller only ever materializes the has_separator==false code.
  * Force-inline from_chars_caller::call (all three specializations) so the
    separator branch constant-folds away on the common from_chars() path.

Net: the default hot frame returns to 844 instructions (vs 833 on main; the
+11 is the unrelated parse_number_string refactor and is unchanged by this
commit). The separator path is unaffected and all tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
재욱 2026-06-21 11:24:32 +09:00
parent 1d35d26b48
commit 8428fd7c49
2 changed files with 26 additions and 11 deletions

View File

@ -217,6 +217,12 @@ using parse_options = parse_options_t<char>;
#define fastfloat_really_inline inline __attribute__((always_inline))
#endif
#ifdef FASTFLOAT_VISUAL_STUDIO
#define FASTFLOAT_NOINLINE __declspec(noinline)
#else
#define FASTFLOAT_NOINLINE __attribute__((noinline, cold))
#endif
// Branch-probability hint marking the rare slow-path branches as cold, so the
// optimizer keeps the out-of-line slow-path re-parse off the hot path (and does
// not duplicate the force-inlined hot scanner into the caller, which bloated

View File

@ -139,7 +139,7 @@ fastfloat_really_inline bool rounds_to_nearest() noexcept {
template <typename T> struct from_chars_caller {
template <typename UC>
FASTFLOAT_CONSTEXPR20 static from_chars_result_t<UC>
fastfloat_really_inline FASTFLOAT_CONSTEXPR20 static from_chars_result_t<UC>
call(UC const *first, UC const *last, T &value,
parse_options_t<UC> options) noexcept {
return from_chars_advanced(first, last, value, options);
@ -149,7 +149,7 @@ template <typename T> struct from_chars_caller {
#ifdef __STDCPP_FLOAT32_T__
template <> struct from_chars_caller<std::float32_t> {
template <typename UC>
FASTFLOAT_CONSTEXPR20 static from_chars_result_t<UC>
fastfloat_really_inline FASTFLOAT_CONSTEXPR20 static from_chars_result_t<UC>
call(UC const *first, UC const *last, std::float32_t &value,
parse_options_t<UC> options) noexcept {
// if std::float32_t is defined, and we are in C++23 mode; macro set for
@ -166,7 +166,7 @@ template <> struct from_chars_caller<std::float32_t> {
#ifdef __STDCPP_FLOAT64_T__
template <> struct from_chars_caller<std::float64_t> {
template <typename UC>
FASTFLOAT_CONSTEXPR20 static from_chars_result_t<UC>
fastfloat_really_inline FASTFLOAT_CONSTEXPR20 static from_chars_result_t<UC>
call(UC const *first, UC const *last, std::float64_t &value,
parse_options_t<UC> options) noexcept {
// if std::float64_t is defined, and we are in C++23 mode; macro set for
@ -289,23 +289,32 @@ from_chars_advanced(parsed_number_string_t<UC> &pns, T &value) noexcept {
return answer;
}
template <bool bjf, typename UC>
FASTFLOAT_NOINLINE FASTFLOAT_CONSTEXPR20 parsed_number_string_t<UC>
parse_number_string_with_separator(UC const *first, UC const *last,
parse_options_t<UC> options,
bool store_spans) noexcept {
return parse_number_string<bjf, true, UC>(first, last, options, store_spans);
}
// Runtime -> compile-time dispatch over both boolean knobs of
// parse_number_string. basic_json_fmt was already dispatched this way; the
// digit separator is selected here too so that the separator-aware code paths
// stay confined to the (cold) has_separator==true instantiation. Callers that
// never set a separator -- the overwhelming majority -- run the
// stay confined to the (cold, out-of-line) has_separator==true instantiation.
// Callers that never set a separator -- the overwhelming majority -- run the
// has_separator==false instantiation, which is byte-for-byte the original
// separator-free parser.
// separator-free parser; the separator check is a single predictable branch
// into cold code.
template <typename UC>
fastfloat_really_inline FASTFLOAT_CONSTEXPR20 parsed_number_string_t<UC>
parse_number_string_options(UC const *first, UC const *last,
parse_options_t<UC> options, bool bjf,
bool store_spans) noexcept {
if (options.digit_separator != UC('\0')) {
return bjf ? parse_number_string<true, true, UC>(first, last, options,
store_spans)
: parse_number_string<false, true, UC>(first, last, options,
store_spans);
if fastfloat_unlikely (options.digit_separator != UC('\0')) {
return bjf ? parse_number_string_with_separator<true, UC>(
first, last, options, store_spans)
: parse_number_string_with_separator<false, UC>(
first, last, options, store_spans);
}
return bjf ? parse_number_string<true, false, UC>(first, last, options,
store_spans)