WeaselJson_OVERFLOW depends on chunk framing: partially-parsed literals and \u escapes push tokens onto the nesting stack, so the same document is accepted whole but OVERFLOWs when streamed
#71
WeaselJson_OVERFLOW is documented as "json is too deeply nested" (include/weaseljson.h:39), and stackSize is documented as a nesting-depth budget ("Increasing stack size increases memory usage but also increases the depth of nested json accepted", include/weaseljson.h:52-55).
In reality the pushdown stack in src/parser3.h also holds tokenizer progress tokens. When a true/false/null literal or a \uXXXX escape does not fit entirely in the current chunk, the parser pushes the not-yet-matched tokens onto the same fixed-size stack (T_R T_U N_TRUE, T_A T_L T_S N_FALSE, T_U T_L N_NULL, N_STRING_FOLLOWING_ESCAPE, four T_HEX, or six symbols for a surrogate pair). The whole-buffer path consumes those tokens inline and pushes nothing.
Consequence: the minimum stackSize needed for a document depends on how the bytes are chunked. The same document, same parser, same stackSize is accepted when fed whole and returns WeaselJson_OVERFLOW when streamed in smaller pieces. WeaselJson_OVERFLOW is a terminal state (see #52), so the caller cannot recover from the spurious failure, and cannot size the stack reliably because the requirement depends on the (unpredictable) segmentation of the transport.
Reproduction
repro.cpp (links against the library, public API only):
#include<cstdio>#include<cstring>#include"weaseljson.h"staticvoidev(void*){}staticvoiddt(void*,constchar*,int,int){}staticWeaselJsonStatusrun(constchar*doc,intstackSize,intstride){WeaselJsonCallbacksc{};c.on_begin_object=ev;c.on_end_object=ev;c.on_string_data=dt;c.on_key_data=dt;c.on_begin_array=ev;c.on_end_array=ev;c.on_number_data=dt;c.on_true_literal=ev;c.on_false_literal=ev;c.on_null_literal=ev;WeaselJsonParser*p=WeaselJsonParser_create(stackSize,&c,nullptr,0);if(!p)returnWeaselJson_NULL;charbuf[64];intlen=(int)std::strlen(doc);std::memcpy(buf,doc,(size_t)len+1);WeaselJsonStatuss=WeaselJson_AGAIN;for(inti=0;i<len&&s==WeaselJson_AGAIN;i+=stride){intn=stride<len-i?stride:len-i;s=WeaselJsonParser_parse(p,buf+i,n);}if(s==WeaselJson_AGAIN)s=WeaselJsonParser_parse(p,nullptr,0);WeaselJsonParser_destroy(p);returns;}intmain(){constchar*docs[]={"1","true","false","null","[true]","[true,false]","\"\\u00e9abcdefgh\"","[\"\\u00e9abcdefgh\"]"};constchar*name[]={"OK","AGAIN","REJECT","OVERFLOW","NULL"};for(autodoc:docs){for(intstack=3;stack<=8;++stack){autow=run(doc,stack,1<<20);// whole buffer
autob=run(doc,stack,1);// one byte at a time
std::printf("%-20s stackSize=%d whole=%-9s 1-byte-chunks=%-9s%s\n",doc,stack,name[w],name[b],w!=b?" <-- FRAMING DEPENDENT":"");}std::printf("\n");}}
Minimum stackSize that succeeds, measured by bisection:
document
whole buffer
1 byte per parse()
true
3
5
null
3
5
false
3
6
[true]
4
6
[true,false]
4
7
"\u00e9abcdefgh"
3
7
["\u00e9abcdefgh"]
4
8
["\ud801\udc37"]
4
10
(3 is the smallest stackSize for which WeaselJsonParser_create() succeeds at all.)
The failure happens as soon as the offending byte arrives, not at EOF. Feeding ["\u00e9abcdefgh"] one byte at a time with stackSize = 4 gives the per-byte statuses AGAIN AGAIN OVERFLOW (it dies on the \), and ["\ud801\udc37"] with stackSize = 6 gives AGAIN AGAIN AGAIN OVERFLOW (dies on the u), while the identical bytes fed in one shot at the same stackSize end in WeaselJson_OK.
Note that none of these documents is "too deeply nested" — true is a scalar with no nesting at all — yet WeaselJson_OVERFLOW is returned.
Root cause
Literal and escape tokens are pushed onto the same fixed pushdown stack that models nesting (src/parser3.h, HEAD 730a4cd):
n_value(), src/parser3.h:371-379 (true), :387-395 (false), :403-411 (null): the fast path (bufEnd - buf >= 3 / >= 4) validates the literal with memcmp and pushes nothing; the slow path pushes 3 or 4 symbols, e.g.
n_string2(), src/parser3.h:611-612: when fewer than 11 bytes remain in the chunk it takes the slow escape path and pushes N_STRING_FOLLOWING_ESCAPE.
n_string_following_escape(), src/parser3.h:721: pushes four T_HEX symbols ({T_HEX, T_HEX, T_HEX, T_HEX2}).
t_hex2(), src/parser3.h:803-804: for a UTF-16 surrogate pair it pushes six more ({T_BACKSLASH, T_U2, T_HEX, T_HEX, T_HEX, T_HEX3}).
push() (src/parser3.h:118-124) returns WeaselJson_OVERFLOW as soon as those transient symbols do not fit, so the effective nesting budget is reduced by up to 6 slots — but only for the chunk boundaries that happen to split a literal or an escape, which is why the outcome depends on framing.
Expected vs actual
Expected: for a given stackSize the accept/reject/overflow decision is a function of the input bytes only, and WeaselJson_OVERFLOW means the document's nesting exceeds the stack (this is what the header comment and the overflow state is sticky test in src/test.cpp:205 assume).
Actual: the decision also depends on where the parse() boundaries fall; a document that fits is reported as "too deeply nested" purely because of segmentation, and every later parse() keeps returning WeaselJson_OVERFLOW (terminal), so the caller has to discard the partially parsed result and restart with an arbitrarily larger stack.
Impact
This is a streaming parser, so callers hand it whatever the transport delivers (one byte at a time, 4 KiB socket reads, mmap'ed pages). With a stackSize chosen for the application's maximum nesting depth, a document whose true/false/null/\uXXXX happens to straddle a chunk boundary is rejected with a bogus, sticky WeaselJson_OVERFLOW, while the very same document parses when the reads happen to align differently. That is a nondeterministic failure which does not reproduce when the input is slurped whole — which is how most existing tests, testStreaming() in src/fuzz.cpp and the fuzz_corpus regression run feed it (fixed strides 1..15 over the corpus, and the corpus happens to stay far from the interesting stack sizes). The RootBuilder generated by contrib/schemagen exposes exactly this knob (RootBuilder(int stackSize = 1024)), and its feed() surfaces the bogus WeaselJson_OVERFLOW to applications that size the stack for their schema's depth.
Possible fixes
Either is fine; the point is that the stack requirement must not depend on chunking:
Keep the partial-literal/escape progress in Parser3 state (a small counter/state field, like the existing numDfa/utf8Codepoint state) instead of pushing T_* symbols onto the pushdown stack, or
Keep the current encoding but have WeaselJsonParser_create() reserve a fixed number of slots for the transient tokens (so the documented depth budget for stackSize is exact and framing-independent), rejecting too-small stackSize values at creation time.
## Summary
`WeaselJson_OVERFLOW` is documented as *"json is too deeply nested"* (`include/weaseljson.h:39`), and `stackSize` is documented as a nesting-depth budget (*"Increasing stack size increases memory usage but also increases the depth of nested json accepted"*, `include/weaseljson.h:52-55`).
In reality the pushdown stack in `src/parser3.h` also holds **tokenizer progress** tokens. When a `true`/`false`/`null` literal or a `\uXXXX` escape does not fit entirely in the current chunk, the parser pushes the not-yet-matched tokens onto the *same* fixed-size stack (`T_R T_U N_TRUE`, `T_A T_L T_S N_FALSE`, `T_U T_L N_NULL`, `N_STRING_FOLLOWING_ESCAPE`, four `T_HEX`, or six symbols for a surrogate pair). The whole-buffer path consumes those tokens inline and pushes nothing.
Consequence: **the minimum `stackSize` needed for a document depends on how the bytes are chunked.** The same document, same parser, same `stackSize` is accepted when fed whole and returns `WeaselJson_OVERFLOW` when streamed in smaller pieces. `WeaselJson_OVERFLOW` is a terminal state (see #52), so the caller cannot recover from the spurious failure, and cannot size the stack reliably because the requirement depends on the (unpredictable) segmentation of the transport.
## Reproduction
`repro.cpp` (links against the library, public API only):
```cpp
#include <cstdio>
#include <cstring>
#include "weaseljson.h"
static void ev(void *) {}
static void dt(void *, const char *, int, int) {}
static WeaselJsonStatus run(const char *doc, int stackSize, int stride) {
WeaselJsonCallbacks c{};
c.on_begin_object = ev; c.on_end_object = ev; c.on_string_data = dt; c.on_key_data = dt;
c.on_begin_array = ev; c.on_end_array = ev; c.on_number_data = dt;
c.on_true_literal = ev; c.on_false_literal = ev; c.on_null_literal = ev;
WeaselJsonParser *p = WeaselJsonParser_create(stackSize, &c, nullptr, 0);
if (!p) return WeaselJson_NULL;
char buf[64];
int len = (int)std::strlen(doc);
std::memcpy(buf, doc, (size_t)len + 1);
WeaselJsonStatus s = WeaselJson_AGAIN;
for (int i = 0; i < len && s == WeaselJson_AGAIN; i += stride) {
int n = stride < len - i ? stride : len - i;
s = WeaselJsonParser_parse(p, buf + i, n);
}
if (s == WeaselJson_AGAIN) s = WeaselJsonParser_parse(p, nullptr, 0);
WeaselJsonParser_destroy(p);
return s;
}
int main() {
const char *docs[] = {"1", "true", "false", "null", "[true]", "[true,false]",
"\"\\u00e9abcdefgh\"", "[\"\\u00e9abcdefgh\"]"};
const char *name[] = {"OK", "AGAIN", "REJECT", "OVERFLOW", "NULL"};
for (auto doc : docs) {
for (int stack = 3; stack <= 8; ++stack) {
auto w = run(doc, stack, 1 << 20); // whole buffer
auto b = run(doc, stack, 1); // one byte at a time
std::printf("%-20s stackSize=%d whole=%-9s 1-byte-chunks=%-9s%s\n", doc, stack,
name[w], name[b], w != b ? " <-- FRAMING DEPENDENT" : "");
}
std::printf("\n");
}
}
```
Observed output (gcc 16 / aarch64, Release; clang 22 behaves the same):
```
1 stackSize=3 whole=OK 1-byte-chunks=OK
true stackSize=3 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
true stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
true stackSize=5 whole=OK 1-byte-chunks=OK
false stackSize=3 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
false stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
false stackSize=5 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
false stackSize=6 whole=OK 1-byte-chunks=OK
null stackSize=3 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
null stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
null stackSize=5 whole=OK 1-byte-chunks=OK
[true] stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
[true] stackSize=5 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
[true] stackSize=6 whole=OK 1-byte-chunks=OK
[true,false] stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
[true,false] stackSize=5 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
[true,false] stackSize=6 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
[true,false] stackSize=7 whole=OK 1-byte-chunks=OK
"\u00e9abcdefgh" stackSize=3 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
"\u00e9abcdefgh" stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
"\u00e9abcdefgh" stackSize=5 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
"\u00e9abcdefgh" stackSize=6 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
"\u00e9abcdefgh" stackSize=7 whole=OK 1-byte-chunks=OK
["\u00e9abcdefgh"] stackSize=4 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
["\u00e9abcdefgh"] stackSize=5 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
["\u00e9abcdefgh"] stackSize=6 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
["\u00e9abcdefgh"] stackSize=7 whole=OK 1-byte-chunks=OVERFLOW <-- FRAMING DEPENDENT
["\u00e9abcdefgh"] stackSize=8 whole=OK 1-byte-chunks=OK
```
Minimum `stackSize` that succeeds, measured by bisection:
| document | whole buffer | 1 byte per `parse()` |
|-------------------------|--------------|----------------------|
| `true` | 3 | 5 |
| `null` | 3 | 5 |
| `false` | 3 | 6 |
| `[true]` | 4 | 6 |
| `[true,false]` | 4 | 7 |
| `"\u00e9abcdefgh"` | 3 | 7 |
| `["\u00e9abcdefgh"]` | 4 | 8 |
| `["\ud801\udc37"]` | 4 | 10 |
(`3` is the smallest `stackSize` for which `WeaselJsonParser_create()` succeeds at all.)
The failure happens as soon as the offending byte arrives, not at EOF. Feeding `["\u00e9abcdefgh"]` one byte at a time with `stackSize = 4` gives the per-byte statuses `AGAIN AGAIN OVERFLOW` (it dies on the `\`), and `["\ud801\udc37"]` with `stackSize = 6` gives `AGAIN AGAIN AGAIN OVERFLOW` (dies on the `u`), while the identical bytes fed in one shot at the same `stackSize` end in `WeaselJson_OK`.
Note that none of these documents is "too deeply nested" — `true` is a scalar with no nesting at all — yet `WeaselJson_OVERFLOW` is returned.
## Root cause
Literal and escape tokens are pushed onto the same fixed pushdown stack that models nesting (`src/parser3.h`, HEAD 730a4cd):
* `n_value()`, `src/parser3.h:371-379` (`true`), `:387-395` (`false`), `:403-411` (`null`): the fast path (`bufEnd - buf >= 3` / `>= 4`) validates the literal with `memcmp` and pushes nothing; the slow path pushes 3 or 4 symbols, e.g.
```cpp
} else {
if (auto s = self->push({T_A, T_L, T_S, N_FALSE})) { // src/parser3.h:395
return s;
}
}
```
* `n_string2()`, `src/parser3.h:611-612`: when fewer than 11 bytes remain in the chunk it takes the slow escape path and pushes `N_STRING_FOLLOWING_ESCAPE`.
* `n_string_following_escape()`, `src/parser3.h:721`: pushes four `T_HEX` symbols (`{T_HEX, T_HEX, T_HEX, T_HEX2}`).
* `t_hex2()`, `src/parser3.h:803-804`: for a UTF-16 surrogate pair it pushes six more (`{T_BACKSLASH, T_U2, T_HEX, T_HEX, T_HEX, T_HEX3}`).
`push()` (`src/parser3.h:118-124`) returns `WeaselJson_OVERFLOW` as soon as those transient symbols do not fit, so the effective nesting budget is reduced by up to 6 slots — but only for the chunk boundaries that happen to split a literal or an escape, which is why the outcome depends on framing.
## Expected vs actual
* Expected: for a given `stackSize` the accept/reject/overflow decision is a function of the input bytes only, and `WeaselJson_OVERFLOW` means the document's nesting exceeds the stack (this is what the header comment and the `overflow state is sticky` test in `src/test.cpp:205` assume).
* Actual: the decision also depends on where the `parse()` boundaries fall; a document that fits is reported as "too deeply nested" purely because of segmentation, and every later `parse()` keeps returning `WeaselJson_OVERFLOW` (terminal), so the caller has to discard the partially parsed result and restart with an arbitrarily larger stack.
## Impact
This is a streaming parser, so callers hand it whatever the transport delivers (one byte at a time, 4 KiB socket reads, mmap'ed pages). With a `stackSize` chosen for the application's maximum nesting depth, a document whose `true`/`false`/`null`/`\uXXXX` happens to straddle a chunk boundary is rejected with a bogus, sticky `WeaselJson_OVERFLOW`, while the very same document parses when the reads happen to align differently. That is a nondeterministic failure which does not reproduce when the input is slurped whole — which is how most existing tests, `testStreaming()` in `src/fuzz.cpp` and the `fuzz_corpus` regression run feed it (fixed strides 1..15 over the corpus, and the corpus happens to stay far from the interesting stack sizes). The `RootBuilder` generated by `contrib/schemagen` exposes exactly this knob (`RootBuilder(int stackSize = 1024)`), and its `feed()` surfaces the bogus `WeaselJson_OVERFLOW` to applications that size the stack for their schema's depth.
## Possible fixes
Either is fine; the point is that the stack requirement must not depend on chunking:
1. Keep the partial-literal/escape progress in `Parser3` state (a small counter/state field, like the existing `numDfa`/`utf8Codepoint` state) instead of pushing `T_*` symbols onto the pushdown stack, or
2. Keep the current encoding but have `WeaselJsonParser_create()` reserve a fixed number of slots for the transient tokens (so the documented depth budget for `stackSize` is exact and framing-independent), rejecting too-small `stackSize` values at creation time.
Do fix 2 described above but reserve extra for transient tokens in addition to the supplied stackSize. Find an upper bound on the max number of transient tokens and document it. Augment our fuzz tests so that it would have caught this. The property we want to test is "whether or not a document parses is independent of the streaming pattern"
Do fix 2 described above but reserve extra for transient tokens in addition to the supplied stackSize. Find an upper bound on the max number of transient tokens and document it. Augment our fuzz tests so that it would have caught this. The property we want to test is "whether or not a document parses is independent of the streaming pattern"
weaselbot
was assigned by andrew2026-10-27 19:54:08 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
WeaselJson_OVERFLOWis documented as "json is too deeply nested" (include/weaseljson.h:39), andstackSizeis documented as a nesting-depth budget ("Increasing stack size increases memory usage but also increases the depth of nested json accepted",include/weaseljson.h:52-55).In reality the pushdown stack in
src/parser3.halso holds tokenizer progress tokens. When atrue/false/nullliteral or a\uXXXXescape does not fit entirely in the current chunk, the parser pushes the not-yet-matched tokens onto the same fixed-size stack (T_R T_U N_TRUE,T_A T_L T_S N_FALSE,T_U T_L N_NULL,N_STRING_FOLLOWING_ESCAPE, fourT_HEX, or six symbols for a surrogate pair). The whole-buffer path consumes those tokens inline and pushes nothing.Consequence: the minimum
stackSizeneeded for a document depends on how the bytes are chunked. The same document, same parser, samestackSizeis accepted when fed whole and returnsWeaselJson_OVERFLOWwhen streamed in smaller pieces.WeaselJson_OVERFLOWis a terminal state (see #52), so the caller cannot recover from the spurious failure, and cannot size the stack reliably because the requirement depends on the (unpredictable) segmentation of the transport.Reproduction
repro.cpp(links against the library, public API only):Observed output (gcc 16 / aarch64, Release; clang 22 behaves the same):
Minimum
stackSizethat succeeds, measured by bisection:parse()truenullfalse[true][true,false]"\u00e9abcdefgh"["\u00e9abcdefgh"]["\ud801\udc37"](
3is the smalleststackSizefor whichWeaselJsonParser_create()succeeds at all.)The failure happens as soon as the offending byte arrives, not at EOF. Feeding
["\u00e9abcdefgh"]one byte at a time withstackSize = 4gives the per-byte statusesAGAIN AGAIN OVERFLOW(it dies on the\), and["\ud801\udc37"]withstackSize = 6givesAGAIN AGAIN AGAIN OVERFLOW(dies on theu), while the identical bytes fed in one shot at the samestackSizeend inWeaselJson_OK.Note that none of these documents is "too deeply nested" —
trueis a scalar with no nesting at all — yetWeaselJson_OVERFLOWis returned.Root cause
Literal and escape tokens are pushed onto the same fixed pushdown stack that models nesting (
src/parser3.h, HEAD730a4cd):n_value(),src/parser3.h:371-379(true),:387-395(false),:403-411(null): the fast path (bufEnd - buf >= 3/>= 4) validates the literal withmemcmpand pushes nothing; the slow path pushes 3 or 4 symbols, e.g.n_string2(),src/parser3.h:611-612: when fewer than 11 bytes remain in the chunk it takes the slow escape path and pushesN_STRING_FOLLOWING_ESCAPE.n_string_following_escape(),src/parser3.h:721: pushes fourT_HEXsymbols ({T_HEX, T_HEX, T_HEX, T_HEX2}).t_hex2(),src/parser3.h:803-804: for a UTF-16 surrogate pair it pushes six more ({T_BACKSLASH, T_U2, T_HEX, T_HEX, T_HEX, T_HEX3}).push()(src/parser3.h:118-124) returnsWeaselJson_OVERFLOWas soon as those transient symbols do not fit, so the effective nesting budget is reduced by up to 6 slots — but only for the chunk boundaries that happen to split a literal or an escape, which is why the outcome depends on framing.Expected vs actual
stackSizethe accept/reject/overflow decision is a function of the input bytes only, andWeaselJson_OVERFLOWmeans the document's nesting exceeds the stack (this is what the header comment and theoverflow state is stickytest insrc/test.cpp:205assume).parse()boundaries fall; a document that fits is reported as "too deeply nested" purely because of segmentation, and every laterparse()keeps returningWeaselJson_OVERFLOW(terminal), so the caller has to discard the partially parsed result and restart with an arbitrarily larger stack.Impact
This is a streaming parser, so callers hand it whatever the transport delivers (one byte at a time, 4 KiB socket reads, mmap'ed pages). With a
stackSizechosen for the application's maximum nesting depth, a document whosetrue/false/null/\uXXXXhappens to straddle a chunk boundary is rejected with a bogus, stickyWeaselJson_OVERFLOW, while the very same document parses when the reads happen to align differently. That is a nondeterministic failure which does not reproduce when the input is slurped whole — which is how most existing tests,testStreaming()insrc/fuzz.cppand thefuzz_corpusregression run feed it (fixed strides 1..15 over the corpus, and the corpus happens to stay far from the interesting stack sizes). TheRootBuildergenerated bycontrib/schemagenexposes exactly this knob (RootBuilder(int stackSize = 1024)), and itsfeed()surfaces the bogusWeaselJson_OVERFLOWto applications that size the stack for their schema's depth.Possible fixes
Either is fine; the point is that the stack requirement must not depend on chunking:
Parser3state (a small counter/state field, like the existingnumDfa/utf8Codepointstate) instead of pushingT_*symbols onto the pushdown stack, orWeaselJsonParser_create()reserve a fixed number of slots for the transient tokens (so the documented depth budget forstackSizeis exact and framing-independent), rejecting too-smallstackSizevalues at creation time.Do fix 2 described above but reserve extra for transient tokens in addition to the supplied stackSize. Find an upper bound on the max number of transient tokens and document it. Augment our fuzz tests so that it would have caught this. The property we want to test is "whether or not a document parses is independent of the streaming pattern"