schemagen: generated number parser rejects valid JSON numbers that underflow double (e.g. 1e-400) with WeaselJson_REJECT #76

New Issue

2026-11-07T21:00:41Z

weaselbot commented

2026-11-07 21:00:41 +00:00

Summary

The number (double) branch of the parser that schemagen embeds in every generated header rejects any JSON number whose magnitude underflows the double range, even though the correctly rounded value (±0.0) is perfectly representable and JSON Schema puts no magnitude bound on number.

The generated code converts the accumulated number text with std::from_chars and treats any non-zero error code as a schema violation:

# contrib/schemagen/weaseljson_schemagen.py:1239-1243  (in cbNumberData)
    }} else {{
      double v = 0;
      auto r = std::from_chars(b, e, v);
      if (r.ec != std::errc() || r.ptr != e) {{ reject(); return; }}
      *(double *)p = v;
    }}

std::from_chars reports errc::result_out_of_range for values that underflow to zero, not just for overflow, so 1e-400, 2.4e-324, -1e-500, … are reported as WeaselJson_REJECT even though weaseljson itself accepts the very same document as valid JSON (WeaselJson_OK).

This is the same class of defect as the already-fixed #56 ("integer parser rejects valid zero values with negative exponents"), but on the number/double path, which has no equivalent handling.

Reproduction

REPO=/path/to/weaseljson          # this checkout, built (build/libweaseljson.so)
mkdir -p /tmp/repro && cd /tmp/repro
printf '%s' '{"type":"object","additionalProperties":false,"required":["x"],"properties":{"x":{"type":"number"}}}' > schema.json
python3 "$REPO/contrib/schemagen/weaseljson_schemagen.py" schema.json -o gen.h --namespace fg

cat > test.cpp <<'EOF'
#include "gen.h"
#include <cstdio>
#include <cstring>
#include <string>
int main() {
  const char *docs[] = {"{\"x\":0}", "{\"x\":0e-400}", "{\"x\":2.4e-324}",
                        "{\"x\":2.5e-324}", "{\"x\":1e-400}", "{\"x\":-1e-400}",
                        "{\"x\":1e-308}"};
  for (const char *d : docs) {
    std::string buf(d, strlen(d));
    fg::RootBuilder b;
    WeaselJsonStatus st = b.feed(buf.data(), (int)buf.size());
    if (st == WeaselJson_AGAIN) st = b.finish();
    std::printf("%-20s -> status %d%s\n", d, (int)st,
                st == WeaselJson_OK ? " (OK)" : st == WeaselJson_REJECT ? " (REJECT)" : "");
  }
}
EOF
g++ -std=c++20 -O2 -I "$REPO/include" -I . test.cpp "$REPO"/build/libweaseljson.so -Wl,-rpath,"$REPO"/build -o test
./test

Observed (g++ 16.1.1 / libstdc++ 15, aarch64):

{"x":0}              -> status 0 (OK)
{"x":0e-400}         -> status 0 (OK)
{"x":2.4e-324}       -> status 2 (REJECT)
{"x":2.5e-324}       -> status 0 (OK)
{"x":1e-400}         -> status 2 (REJECT)
{"x":-1e-400}        -> status 2 (REJECT)
{"x":1e-308}         -> status 0 (OK)

Expected: all seven are WeaselJson_OK. 1e-400 and 2.4e-324 are syntactically valid JSON numbers (weaseljson's own parser returns WeaselJson_OK for those exact documents — I checked by feeding the same bytes to WeaselJsonParser_parse, which returns WeaselJson_OK while the generated builder returns WeaselJson_REJECT), and their correctly rounded double value is 0.0 (-0.0 for the negated one), which the target type does represent.

The accept/reject boundary is not a schema property at all: it is "does the correctly rounded value become exactly zero?". 2.5e-324 (the smallest positive subnormal) is accepted while 2.4e-324 (rounds to zero) is rejected, and an exact 0 / 0e-400 is accepted.

Evidence that from_chars is the culprit:

#include <charconv>
#include <cstdio>
#include <cerrno>
#include <cstdlib>
#include <cstring>
#include <initializer_list>
int main(){ for (const char *t : {"1e-400","2.4e-324","2.5e-324"}) {
  double v = 42; auto r = std::from_chars(t, t+strlen(t), v); errno = 0;
  double s = strtod(t, nullptr);
  printf("%-10s from_chars ec=%d (result_out_of_range=%d) value=%g | strtod=%g\n",
         t, (int)r.ec, (int)std::errc::result_out_of_range, v, s); } }

1e-400     from_chars ec=34 (result_out_of_range=34) value=42 | strtod=0
2.4e-324   from_chars ec=34 (result_out_of_range=34) value=42 | strtod=0
2.5e-324   from_chars ec=0 (result_out_of_range=34) value=4.94066e-324 | strtod=4.94066e-324

strtod also sets ERANGE for underflow, but returns 0.0; from_chars reports it as an error. Python's float("1e-400"), JSON.parse("1e-400") in JavaScript, etc. all yield 0.0.

It is not specific to object properties: a root-level {"type": "number"} schema rejects the document 1e-400 the same way, as do array items of type number and $ref'd/$defs number slots — they all go through this branch.

Impact

Any schema with a number field can spuriously reject a completely valid document. The caller gets WeaselJson_REJECT, which the schemagen README documents as covering "both malformed JSON and schema violations", so the application cannot distinguish a real schema violation from this false alarm and will typically discard the whole document (or abort) even though it contains no error. Small magnitudes are common in serialized scientific/statistical data.

It also breaks the documented contract "WeaselJson_OK means the document is both valid JSON and schema-valid" in the other direction: the document is valid JSON and schema-valid, yet the builder refuses it.

Suggested fix

Distinguish overflow from underflow instead of treating every result_out_of_range as a violation. Since scratch_ is a NUL-terminated std::string, the simplest option is

    } else {
      char *end = nullptr;
      double v = std::strtod(scratch_.c_str(), &end);  // underflow -> +/-0.0
      if (end != scratch_.c_str() + scratch_.size() || !std::isfinite(v)) {
        reject();
        return;
      }
      *(double *)p = v;
    }

or, staying with from_chars, re-run the conversion after scaling the exponent (or inspect the significand/exponent sign) so that result_out_of_range is only a rejection when the value overflowed. Either way, contrib/schemagen/test_gen.cpp should gain cases for 1e-400, -1e-500 and 2.4e-324.

## Summary The `number` (double) branch of the parser that schemagen embeds in every generated header rejects any JSON number whose magnitude underflows the `double` range, even though the correctly rounded value (`±0.0`) is perfectly representable and JSON Schema puts no magnitude bound on `number`. The generated code converts the accumulated number text with `std::from_chars` and treats *any* non-zero error code as a schema violation: ```python # contrib/schemagen/weaseljson_schemagen.py:1239-1243 (in cbNumberData) }} else {{ double v = 0; auto r = std::from_chars(b, e, v); if (r.ec != std::errc() || r.ptr != e) {{ reject(); return; }} *(double *)p = v; }} ``` `std::from_chars` reports `errc::result_out_of_range` for values that *underflow* to zero, not just for overflow, so `1e-400`, `2.4e-324`, `-1e-500`, … are reported as `WeaselJson_REJECT` even though weaseljson itself accepts the very same document as valid JSON (`WeaselJson_OK`). This is the same class of defect as the already-fixed #56 ("integer parser rejects valid zero values with negative exponents"), but on the `number`/`double` path, which has no equivalent handling. ## Reproduction ```sh REPO=/path/to/weaseljson # this checkout, built (build/libweaseljson.so) mkdir -p /tmp/repro && cd /tmp/repro printf '%s' '{"type":"object","additionalProperties":false,"required":["x"],"properties":{"x":{"type":"number"}}}' > schema.json python3 "$REPO/contrib/schemagen/weaseljson_schemagen.py" schema.json -o gen.h --namespace fg cat > test.cpp <<'EOF' #include "gen.h" #include <cstdio> #include <cstring> #include <string> int main() { const char *docs[] = {"{\"x\":0}", "{\"x\":0e-400}", "{\"x\":2.4e-324}", "{\"x\":2.5e-324}", "{\"x\":1e-400}", "{\"x\":-1e-400}", "{\"x\":1e-308}"}; for (const char *d : docs) { std::string buf(d, strlen(d)); fg::RootBuilder b; WeaselJsonStatus st = b.feed(buf.data(), (int)buf.size()); if (st == WeaselJson_AGAIN) st = b.finish(); std::printf("%-20s -> status %d%s\n", d, (int)st, st == WeaselJson_OK ? " (OK)" : st == WeaselJson_REJECT ? " (REJECT)" : ""); } } EOF g++ -std=c++20 -O2 -I "$REPO/include" -I . test.cpp "$REPO"/build/libweaseljson.so -Wl,-rpath,"$REPO"/build -o test ./test ``` Observed (g++ 16.1.1 / libstdc++ 15, aarch64): ``` {"x":0} -> status 0 (OK) {"x":0e-400} -> status 0 (OK) {"x":2.4e-324} -> status 2 (REJECT) {"x":2.5e-324} -> status 0 (OK) {"x":1e-400} -> status 2 (REJECT) {"x":-1e-400} -> status 2 (REJECT) {"x":1e-308} -> status 0 (OK) ``` Expected: all seven are `WeaselJson_OK`. `1e-400` and `2.4e-324` are syntactically valid JSON numbers (weaseljson's own parser returns `WeaselJson_OK` for those exact documents — I checked by feeding the same bytes to `WeaselJsonParser_parse`, which returns `WeaselJson_OK` while the generated builder returns `WeaselJson_REJECT`), and their correctly rounded `double` value is `0.0` (`-0.0` for the negated one), which the target type does represent. The accept/reject boundary is not a schema property at all: it is "does the correctly rounded value become exactly zero?". `2.5e-324` (the smallest positive subnormal) is accepted while `2.4e-324` (rounds to zero) is rejected, and an exact `0` / `0e-400` is accepted. Evidence that `from_chars` is the culprit: ```cpp #include <charconv> #include <cstdio> #include <cerrno> #include <cstdlib> #include <cstring> #include <initializer_list> int main(){ for (const char *t : {"1e-400","2.4e-324","2.5e-324"}) { double v = 42; auto r = std::from_chars(t, t+strlen(t), v); errno = 0; double s = strtod(t, nullptr); printf("%-10s from_chars ec=%d (result_out_of_range=%d) value=%g | strtod=%g\n", t, (int)r.ec, (int)std::errc::result_out_of_range, v, s); } } ``` ``` 1e-400 from_chars ec=34 (result_out_of_range=34) value=42 | strtod=0 2.4e-324 from_chars ec=34 (result_out_of_range=34) value=42 | strtod=0 2.5e-324 from_chars ec=0 (result_out_of_range=34) value=4.94066e-324 | strtod=4.94066e-324 ``` `strtod` also sets `ERANGE` for underflow, but returns `0.0`; `from_chars` reports it as an error. Python's `float("1e-400")`, `JSON.parse("1e-400")` in JavaScript, etc. all yield `0.0`. It is not specific to object properties: a root-level `{"type": "number"}` schema rejects the document `1e-400` the same way, as do array items of type `number` and `$ref`'d/`$defs` number slots — they all go through this branch. ## Impact Any schema with a `number` field can spuriously reject a completely valid document. The caller gets `WeaselJson_REJECT`, which the schemagen README documents as covering "both malformed JSON and schema violations", so the application cannot distinguish a real schema violation from this false alarm and will typically discard the whole document (or abort) even though it contains no error. Small magnitudes are common in serialized scientific/statistical data. It also breaks the documented contract "`WeaselJson_OK` means the document is both valid JSON and schema-valid" in the other direction: the document is valid JSON and schema-valid, yet the builder refuses it. ## Suggested fix Distinguish overflow from underflow instead of treating every `result_out_of_range` as a violation. Since `scratch_` is a NUL-terminated `std::string`, the simplest option is ```cpp } else { char *end = nullptr; double v = std::strtod(scratch_.c_str(), &end); // underflow -> +/-0.0 if (end != scratch_.c_str() + scratch_.size() || !std::isfinite(v)) { reject(); return; } *(double *)p = v; } ``` or, staying with `from_chars`, re-run the conversion after scaling the exponent (or inspect the significand/exponent sign) so that `result_out_of_range` is only a rejection when the value overflowed. Either way, `contrib/schemagen/test_gen.cpp` should gain cases for `1e-400`, `-1e-500` and `2.4e-324`.

weaselbot commented

2026-11-07 21:00:48 +00:00

Note on the quoted snippet: the doubled braces (}} / {{) are exactly what appears in weaseljson_schemagen.py, because that code lives inside an f-string template; the emitted C++ has single braces.

Note on the quoted snippet: the doubled braces (`}}` / `{{`) are exactly what appears in `weaseljson_schemagen.py`, because that code lives inside an f-string template; the emitted C++ has single braces.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: weaselab/weaseljson#76

schemagen: generated number parser rejects valid JSON numbers that underflow double (e.g. 1e-400) with WeaselJson_REJECT #76

Summary

Reproduction

Impact

Suggested fix

schemagen: generated `number` parser rejects valid JSON numbers that underflow `double` (e.g. 1e-400) with WeaselJson_REJECT #76