TOCTOU race between lexer construction and locale changes causes float truncation

[jgreitemann]

Description

There appears to be a time-of-check-to-time-of-use (TOCTOU) race condition in float parsing that causes silent data loss when another thread changes the locale during JSON deserialization.

The lexer queries localeconv()->decimal_point at construction time, replacing . in the token buffer with that character. Later, strtod parses the buffer using the current locale. If another thread changes the locale between these two points, the locale's current decimal point won't match what's in the buffer, causing strtod to stop parsing at the wrong character.

Check: lexer constructor (lexer.hpp:127) queries get_decimal_point() which calls localeconv().

Use: scan_number_done (lexer.hpp:1293) calls strtod() which respects the current locale.

Any setlocale() call from another thread between construction and parsing can cause misalignment.

Additionally, changing locale in parallel with locale-dependent functions like strtod() is undefined behavior. Since JSON itself does not allow for decimal separators other than ., it is quite surprising that json::parse() exhibits the same safety issue and should be document if it cannot be avoided.

Reproduction steps

1. Compile main.cpp (included below) with -std=c++20. Alternatively, clone https://github.com/jgreitemann/json-locale-bug-repro and run the commands below.
2. Run the binary (it switches locale between C and de_DE on the main thread while a worker thread parses JSON)
3. Observe truncation or assertion failure

# Debug builds trigger an assertion
make debug && ./build/repro-debug

# Release builds silently truncate
make release && ./build/repro

Expected vs. actual results

Expected behavior

99.123456 parsed correctly, with 10M+ iterations completing without error.

Actual behavior (release)

Fractional part is silently truncated: 99.123456 becomes 99.0.

I've also observed the reproduction to crash with SIGSEGV occasionally. I suspect that's another symptom of the undefined behavior due to using strtod while the locale is being set.

Sample output (release)

Locale: C
TRUNCATION at iteration 68: expected 99.123456000, got 99.000000000

or

Locale: C
fish: Job 1, './build/repro' terminated by signal SIGSEGV (Address boundary error)

Actual behavior (debug)

JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size()) at lexer.hpp:1296 fires since the token buffer is not fully consumed.

Sample output (debug)

Locale: C
Assertion failed: (endptr == token_buffer.data() + token_buffer.size()), function scan_number, file lexer.hpp, line 1296.

Minimal code example

// main.cpp
#include <cstdio>
#include <cstdlib>
#include <thread>

#include <nlohmann/json.hpp>

using json = nlohmann::json;

const char *kJson = R"({"val": 99.123456})";
static constexpr double kExpected = 99.123456;
static constexpr int kMaxSwitches = 10'000'000;

void worker(std::stop_token stop) {
  for (int i = 0; !stop.stop_requested(); ++i) {
    json j = json::parse(kJson);
    double val = j.value("val", 0.0);
    if (val != kExpected) {
      fprintf(stderr, "TRUNCATION at iteration %d: expected %.9f, got %.9f\n",
              i, kExpected, val);
      std::exit(1);
    }
  }
}

int main() {
  printf("Locale: %s\n", setlocale(LC_NUMERIC, nullptr));

  {
    int switches = 0;
    bool german = false;

    std::jthread w(worker);

    for (; switches < kMaxSwitches; ++switches) {
      setlocale(LC_NUMERIC, german ? "de_DE.UTF-8" : "C");
      german = !german;
    }
  }

  printf("OK: %d iterations without truncation\n", kMaxSwitches);
}

Compiler and operating system

• Apple clang 21.0.0 / macOS 26.5
• GCC 14.1.0 (glibc 2.31.0) / Linux 7.0.0
• VS 2022, MSVC 19.44.35227 / Windows 11

Library version

v3.11.3 (commit d10879b, latest develop as of this report)

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[nlohmann]

Good catch! Not sure if we can fix this without making parsing slower for everybody not messing with locales.

[jgreitemann]

One option to mitigate the issue would be to pin the thread-local locale using uselocale (POSIX) / _configthreadlocale (Windows) and change it back afterwards. I'm assuming that's what you have in mind as a "slower" option? For reference, I've confirmed that this works and it's probably the mitigation I'll end up going with in production (where I can take the liberty to only do this once on the JSON-parsing worker thread, so the perf impact is amortized).

Another option could be to use the extended locale support (xlocale(3)) function strtod_l afforded by glibc, FreeBSD, Darwin, ... (but crucially not musl AFAICT) and its Windows counterpart _strtod_l. These work by passing the locale explicitly without changing global state. The locale needs to be allocated by newlocale/_create_locale, though, but this can probably be done once in a static? I would hope that this doesn't perform worse compared to strtod but haven't tested it yet.

If the extended locale support is not available, would you prefer to fall back to the current implementation and document the safety requirements w.r.t. locale for other platforms or go with the uselocale solution? The latter approach would avoid the UB/TOCTOU for all users and (hopefully) potentially only regress on perf for users on platforms without extended locale support.

Lastly, std::from_chars<double> in C++17 is locale-independent and generally performs better than strtod. It could be used as an alternative implementation when C++17 support is detected. This can be combined with the above approaches to further reduce the share of users potentially affected by UB/TOCTOU or perf regressions.

Let me know which approaches you think are viable, then I can prepare a patch and perform some benchmarks.

[gregmarr]

Additionally, changing locale in parallel with locale-dependent functions like strtod() is undefined behavior. Since JSON itself does not allow for decimal separators other than ., it is quite surprising that json::parse() exhibits the same safety issue and should be document if it cannot be avoided.

So the simplest method would be to explicitly document that this library uses locale-specific functions in parse(), so changing the locale in parallel with a parse() call can cause undefined behavior. That's not great, so I think pursuing other methods is worthwhile.

At least there is only one line where it calls the overload set of functions strtof that chooses std::strtof, std::strtod or std::strtold based on the output type, so at least that changes would be localized. You'd need to remove the replacement of the decimal point, but that would be a good thing IMHO.

I think falling back to the _l family is good when they're available. From what I've been able to see, the _l performance is the same as non-_l for single threaded apps, and better for multi-threaded apps, presumably because non-_l gets the current locale and calls _l, or at least the same helper function that _l calls.

I am not the maintainer, so @nlohmann has the final say, but I think this would be a good preference order:

• std::from_chars
• strtod_l
• strtod with the replacement of the decimal point

This would need to be detected in a way that allows determining whether or not the decimal point replacement is needed. It could be that the decimal point replacement code remains unconditionally, and it's just always populated with '.' when from_chars or strtod_l is used.

[jgreitemann]

At least there is only one line where it calls the overload set of functions strtof that chooses std::strtof, std::strtod or std::strtold based on the output type, so at least that changes would be localized.

The integer parsing functions std::strtoll and std::strtoull are also used which are also locale-dependent to cover things like thousands separators. Aside from the uses by the lexer, std::strtoull is also used to parse JSON pointers.

Since integers don't have a decimal point and JSON doesn't allow for thousands separators (and the lexer's state machine precludes them), there is no TOCTOU issue for integers. However, changing locale from another thread while these functions are called is still UB, so the fix should probably also cover those functions as well.

[gregmarr]

However, changing locale from another thread while these functions are called is still UB, so the fix should probably also cover those functions as well.

I am personally less worried about that, but if it's a simple fix, then it may be worth it.

[nlohmann]

I think it's a good idea to use std::from_chars where it's available. Also strtod_l. The question is how reliable we can detect these.

[jgreitemann]

std::from_chars could potentially be detected using the feature-test macro __cpp_lib_to_chars (Godbolt). Feature testing is itself a C++20 feature but by the looks of it, it also works when C++20-ready compilers are put in C++17 mode, but not in C++14 mode. However, this would leave a few false negatives, e.g. GCC 10 doesn't know about feature testing so it comes back "Not supported", even in C++17 mode. I think that would be fine since it's an optimization anyway and I suspect the demographic that uses C++17 but doesn't update their toolchain is (hopefully) quite small? Should we also include a macro to forcefully enable/disable the from_chars path?

strtod_l/_strtol could be detected by SFINAE since there are no additional includes necessary.

I'll cook up a PR for this on the weekend to see how this would work out.

[gregmarr]

Sounds good. I think the hardest part is going to be setting a flag that says whether or not to get the decimal point from the locale depending on whether or not strtod_l is available.

[jgreitemann]

Just a quick heads-up: I have it mostly working now but I want to polish a few things before opening a (draft) PR. I'm a bit pressed for time, so it could be a couple of weeks. I'd expect there will be an ensuing fight with CI until all platforms/compilers are happy.

The std::from_chars feature detection turned out to be a bit of a pain because libc++ added floating point support for std::from_chars only recently and with the notable omission of the long double overload.

The SFINAE-based detection of extended locale support I got working. One surprise was that musl does in fact define strtof_l and friends for glibc compatibility but ignores the locale argument and merely delegates to strtof. Fortunately, if I understood correctly, musl doesn't let you set different LC_NUMERIC locales, so that's not an issue. There doesn't seem to be a dedicated musl CI workflow; I wonder if it would be worth adding that?

I can also confirm that integer parsing (via strtol et al.) indeed suffers from UB when switching locale concurrently. Where my original reproduction mostly triggered the assert and only sometimes crashed, a modified version with only an integer field in the JSON object "reliably" reproduces SIGSEGV on Linux and macOS (since there's no TOCTOU race to trip over first). On Windows, the global locale is apparently protected by a mutex, so no crash there, but perf is noticeably degraded. Therefore, my aim would be to expand the fix to also cover integer parsing once I got it working for the floats.