_escape_cpp_string() in contrib/schemagen/weaseljson_schemagen.py escapes U+0000 as the universal character name \u0000 (line 48). In a C++ narrow string literal that produces an embedded NUL byte, but the generated comparisons in matchKey and the enum matcher compare std::string_view/std::string against const char *, i.e. NUL-terminated (strlen) values. A property key or enum value containing U+0000 therefore degenerates to its NUL-prefix during matching. The generated parser then:
routes values for a shorter key to the field of a U+0000-containing key (silently stores data in the wrong member / accepts schema-invalid documents), and
rejects documents whose key or enum value contains U+0000 (schema-valid input rejected), and
stores the wrong enum class index when another enum value equals the NUL-truncated prefix.
Root cause
contrib/schemagen/weaseljson_schemagen.py:
_escape_cpp_string() (lines 21-55): cp < 0x20 branch at lines 47-48 emits f"\\u{cp:04X}", so chr(0) becomes the literal escape \u0000, which evaluates to an embedded NUL byte in a narrow string literal.
_matchkey() lines 830-835 emit if (key == "<escaped>") return i; — key is a std::string_view, compared against a const char *; the string_view conversion uses traits::length(str) (strlen), so "a\u0000b" compares as "a".
_enum_name_arrays() lines 913-920 emit static constexpr const char *X_names[] = { "a\u0000b", ... }, and the generated cbStringData compares scratch_ == si.enames[i] (std::string vs const char *, also strlen-based).
Note this is distinct from the fixed issue #35 (tokenization of control characters): the escaping here compiles fine, it just produces a byte sequence that strlen-based comparison truncates.
Reproduction 1 — property key containing U+0000
Schema (schema.json, note a\u0000b is the key a + NUL + b):
python3 contrib/schemagen/weaseljson_schemagen.py schema.json -o gen.h --namespace gen
Generated gen.h (matchKey, line ~411):
if(key=="a\u0000b")return0;// literal contains an embedded NUL; compares as "a"
if(key=="a")return1;
Harness (compiled with clang++ -std=c++20 -I <include> gen-test.cpp src/lib.cpp):
#include"gen.h"#include<cstdio>#include<string>usingnamespacegen;intmain(){RootBuilderb;std::stringdoc=R"({"a":5})";// key "a" -> std::string field; 5 is a number
WeaselJsonStatuss=b.feed(doc.data(),(int)doc.size());if(s==WeaselJson_AGAIN)s=b.finish();printf("status=%s",s==WeaselJson_OK?"OK":"REJECT");if(s==WeaselJson_OK){Rootr=b.take();printf(" a_b=%s a=%s",r.a_b.has_value()?std::to_string(*r.a_b).c_str():"absent",r.a.has_value()?"set":"absent");}printf("\n");RootBuilderb2;std::stringdoc2=std::string("{\"a\0b\":5}",9);// raw NUL byte key = schema's first property
s=b2.feed(doc2.data(),(int)doc2.size());if(s==WeaselJson_AGAIN)s=b2.finish();printf("status=%s\n",s==WeaselJson_OK?"OK":"REJECT");}
Observed output:
status=OK a_b=5 a=absent <- `{"a":5}` accepted, 5 stored in the field for key "a�b"
status=REJECT <- key "a"+NUL+"b" with integer 5 is schema-valid; rejected
Expected: {"a":5} must be REJECT (key a selects the std::string field; 5 is a number) and {"a<NUL>b":5} must be OK.
// gen.h: enum class X : int { a_b, a }; X_names[] = { "ab", "a" };
doc{"x":"a"}->OK,butx=0(enumeratorfor"ab");schemasays"a"isindex1doc{"x":"a<NUL>b"}->REJECT,though"ab"isthefirstenumvalue(shouldbeOK,x=0)
(The direction depends on declaration order: with ["a", "a\u0000b"], the value "a<NUL>b" is rejected while "a" matches correctly; with ["a\u0000b", "a"], "a" silently stores the wrong enumerator.)
Impact
For any schema whose property keys or string-enum values contain U+0000 ("\u0000" is a valid JSON escape), the generated builder accepts schema-invalid documents with data written into the wrong member, or rejects schema-valid documents, or stores the wrong enum value — all silently (no error at generation time, no crash). The same mis-matching applies to nested objects (every object kind emits the same matchKey pattern).
Suggested fix directions
Compare with explicit lengths, e.g. emit key == std::string_view("a\u0000b", 3) and store enum names as {ptr, len} pairs used with scratch_ == std::string_view(...); or
escape U+0000 in a way that avoids an embedded NUL in the literal, or
reject U+0000 in property keys / enum values at generation time and document it as unsupported (like the other unsupported constructs).
## Summary
`_escape_cpp_string()` in `contrib/schemagen/weaseljson_schemagen.py` escapes U+0000 as the universal character name `\u0000` (line 48). In a C++ narrow string literal that produces an **embedded NUL byte**, but the generated comparisons in `matchKey` and the enum matcher compare `std::string_view`/`std::string` against `const char *`, i.e. **NUL-terminated** (strlen) values. A property key or enum value containing U+0000 therefore degenerates to its NUL-prefix during matching. The generated parser then:
1. routes values for a shorter key to the field of a U+0000-containing key (silently stores data in the wrong member / accepts schema-invalid documents), and
2. rejects documents whose key or enum value contains U+0000 (schema-valid input rejected), and
3. stores the wrong `enum class` index when another enum value equals the NUL-truncated prefix.
## Root cause
`contrib/schemagen/weaseljson_schemagen.py`:
- `_escape_cpp_string()` (lines 21-55): `cp < 0x20` branch at lines 47-48 emits `f"\\u{cp:04X}"`, so `chr(0)` becomes the literal escape `\u0000`, which evaluates to an embedded NUL byte in a narrow string literal.
- `_matchkey()` lines 830-835 emit `if (key == "<escaped>") return i;` — `key` is a `std::string_view`, compared against a `const char *`; the string_view conversion uses `traits::length(str)` (strlen), so `"a\u0000b"` compares as `"a"`.
- `_enum_name_arrays()` lines 913-920 emit `static constexpr const char *X_names[] = { "a\u0000b", ... }`, and the generated `cbStringData` compares `scratch_ == si.enames[i]` (`std::string` vs `const char *`, also strlen-based).
Note this is distinct from the fixed issue #35 (tokenization of control characters): the escaping here *compiles fine*, it just produces a byte sequence that strlen-based comparison truncates.
## Reproduction 1 — property key containing U+0000
Schema (`schema.json`, note `a\u0000b` is the key `a` + NUL + `b`):
```json
{"type": "object", "additionalProperties": false,
"properties": {"a\u0000b": {"type": "integer"}, "a": {"type": "string"}}}
```
```
python3 contrib/schemagen/weaseljson_schemagen.py schema.json -o gen.h --namespace gen
```
Generated `gen.h` (matchKey, line ~411):
```cpp
if (key == "a\u0000b") return 0; // literal contains an embedded NUL; compares as "a"
if (key == "a") return 1;
```
Harness (compiled with `clang++ -std=c++20 -I <include> gen-test.cpp src/lib.cpp`):
```cpp
#include "gen.h"
#include <cstdio>
#include <string>
using namespace gen;
int main() {
RootBuilder b;
std::string doc = R"({"a":5})"; // key "a" -> std::string field; 5 is a number
WeaselJsonStatus s = b.feed(doc.data(), (int)doc.size());
if (s == WeaselJson_AGAIN) s = b.finish();
printf("status=%s", s == WeaselJson_OK ? "OK" : "REJECT");
if (s == WeaselJson_OK) { Root r = b.take();
printf(" a_b=%s a=%s", r.a_b.has_value() ? std::to_string(*r.a_b).c_str() : "absent",
r.a.has_value() ? "set" : "absent"); }
printf("\n");
RootBuilder b2;
std::string doc2 = std::string("{\"a\0b\":5}", 9); // raw NUL byte key = schema's first property
s = b2.feed(doc2.data(), (int)doc2.size());
if (s == WeaselJson_AGAIN) s = b2.finish();
printf("status=%s\n", s == WeaselJson_OK ? "OK" : "REJECT");
}
```
Observed output:
```
status=OK a_b=5 a=absent <- `{"a":5}` accepted, 5 stored in the field for key "a�b"
status=REJECT <- key "a"+NUL+"b" with integer 5 is schema-valid; rejected
```
Expected: `{"a":5}` must be **REJECT** (key `a` selects the `std::string` field; 5 is a number) and `{"a<NUL>b":5}` must be **OK**.
## Reproduction 2 — enum value containing U+0000
Schema: `{"type":"object","additionalProperties":false,"properties":{"x":{"enum":["a\u0000b","a"]}},"required":["x"]}`
```cpp
// gen.h: enum class X : int { a_b, a }; X_names[] = { "a�b", "a" };
doc {"x":"a"} -> OK, but x = 0 (enumerator for "a�b"); schema says "a" is index 1
doc {"x":"a<NUL>b"} -> REJECT, though "a�b" is the first enum value (should be OK, x = 0)
```
(The direction depends on declaration order: with `["a", "a\u0000b"]`, the value `"a<NUL>b"` is rejected while `"a"` matches correctly; with `["a\u0000b", "a"]`, `"a"` silently stores the wrong enumerator.)
## Impact
For any schema whose property keys or string-enum values contain U+0000 (`"\u0000"` is a valid JSON escape), the generated builder accepts schema-invalid documents with data written into the wrong member, or rejects schema-valid documents, or stores the wrong enum value — all silently (no error at generation time, no crash). The same mis-matching applies to nested objects (every object kind emits the same `matchKey` pattern).
## Suggested fix directions
- Compare with explicit lengths, e.g. emit `key == std::string_view("a\u0000b", 3)` and store enum names as `{ptr, len}` pairs used with `scratch_ == std::string_view(...)`; or
- escape U+0000 in a way that avoids an embedded NUL in the literal, or
- reject U+0000 in property keys / enum values at generation time and document it as unsupported (like the other unsupported constructs).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
_escape_cpp_string()incontrib/schemagen/weaseljson_schemagen.pyescapes U+0000 as the universal character name\u0000(line 48). In a C++ narrow string literal that produces an embedded NUL byte, but the generated comparisons inmatchKeyand the enum matcher comparestd::string_view/std::stringagainstconst char *, i.e. NUL-terminated (strlen) values. A property key or enum value containing U+0000 therefore degenerates to its NUL-prefix during matching. The generated parser then:enum classindex when another enum value equals the NUL-truncated prefix.Root cause
contrib/schemagen/weaseljson_schemagen.py:_escape_cpp_string()(lines 21-55):cp < 0x20branch at lines 47-48 emitsf"\\u{cp:04X}", sochr(0)becomes the literal escape\u0000, which evaluates to an embedded NUL byte in a narrow string literal._matchkey()lines 830-835 emitif (key == "<escaped>") return i;—keyis astd::string_view, compared against aconst char *; the string_view conversion usestraits::length(str)(strlen), so"a\u0000b"compares as"a"._enum_name_arrays()lines 913-920 emitstatic constexpr const char *X_names[] = { "a\u0000b", ... }, and the generatedcbStringDatacomparesscratch_ == si.enames[i](std::stringvsconst char *, also strlen-based).Note this is distinct from the fixed issue #35 (tokenization of control characters): the escaping here compiles fine, it just produces a byte sequence that strlen-based comparison truncates.
Reproduction 1 — property key containing U+0000
Schema (
schema.json, notea\u0000bis the keya+ NUL +b):Generated
gen.h(matchKey, line ~411):Harness (compiled with
clang++ -std=c++20 -I <include> gen-test.cpp src/lib.cpp):Observed output:
Expected:
{"a":5}must be REJECT (keyaselects thestd::stringfield; 5 is a number) and{"a<NUL>b":5}must be OK.Reproduction 2 — enum value containing U+0000
Schema:
{"type":"object","additionalProperties":false,"properties":{"x":{"enum":["a\u0000b","a"]}},"required":["x"]}(The direction depends on declaration order: with
["a", "a\u0000b"], the value"a<NUL>b"is rejected while"a"matches correctly; with["a\u0000b", "a"],"a"silently stores the wrong enumerator.)Impact
For any schema whose property keys or string-enum values contain U+0000 (
"\u0000"is a valid JSON escape), the generated builder accepts schema-invalid documents with data written into the wrong member, or rejects schema-valid documents, or stores the wrong enum value — all silently (no error at generation time, no crash). The same mis-matching applies to nested objects (every object kind emits the samematchKeypattern).Suggested fix directions
key == std::string_view("a\u0000b", 3)and store enum names as{ptr, len}pairs used withscratch_ == std::string_view(...); or