parser3: keep the stackSize needed by a document independent of chunking (reserve transient token slots) #72

Merged

andrew merged 3 commits from weaselbot/weaseljson:weaselbot/issue-71 into main 2026-10-30 13:05:59 +00:00

Conversation 1 Commits 3 Files Changed 6

+218 -65

3 Commits

Author	SHA1	Message	Date
weaselbot	bce971fbd2	ci: don't hardcode ~/.local/bin when running pre-commit ... The pre-commit job failed with /root/.local/bin/pre-commit: No such file or directory even though `pipx install pre-commit` succeeded: `pipx install` writes the app to PIPX_BIN_DIR, and the runner image no longer uses ~/.local/bin for it. Run the binary from PATH when pipx put it there, and fall back to `pipx run pre-commit` otherwise, so the hook still runs regardless of where pipx exposes its apps.	2026-10-27 18:53:07 -04:00
weaselbot	507be6f0a2	Address review: keep the transient-token reserve out of the public header ... The create() documentation described how the reserve is implemented (six extra slots for literals and \uXXXX escapes that straddle a chunk boundary). That is an implementation detail; the header should only promise the behaviour callers can rely on, i.e. that the accept/reject/ overflow decision depends on the document's bytes and on stackSize, not on how they are split across parse() calls. The derivation of the upper bound lives next to kTransientSymbols in src/parser3.h, where it can be kept in sync with the pushes. Also make the chunking test cover every split point, not just fixed strides: each document is split in two at every possible offset, so every literal, escape and surrogate pair in the fixture straddles a chunk boundary. Verified that this fails on the pre-fix parser ("true", stackSize 3, cut 1: whole buffer accepts, two chunks overflow).	2026-10-27 17:31:49 -04:00
weaselbot	ac619af24d	parser3: reserve stack slots for chunk-split literals and escapes ... WeaselJson_OVERFLOW is documented as "json is too deeply nested", and stackSize as a nesting-depth budget. In practice the pushdown stack also held tokenizer progress for a `true`/`false`/`null` literal or a \uXXXX escape that straddled the end of a chunk: the whole-buffer path consumes those characters inline and pushes nothing, while the slow path pushed up to six symbols (a surrogate pair) onto the same stack. So the stackSize a document needed depended on where the parse() boundaries fell: e.g. "true" was accepted at stackSize 3 when fed whole but returned the sticky WeaselJson_OVERFLOW when fed one byte at a time. Reserve slots for those symbols in addition to the supplied stackSize, as a fixed upper bound rather than by changing the encoding: the largest single push of transient symbols is t_hex2()'s six symbols for a surrogate pair, and no transient symbol is ever pushed while another one is on the stack, so kTransientSymbols = 6 is enough. Nesting pushes are still bounded by stackSize, so OVERFLOW now means exactly what the header says and the accept/reject/overflow decision is a function of the input bytes and stackSize only. Document the reserve in weaseljson.h and cover the property with tests: a doctest sweeping stackSize 3..16 over documents whose literals, escapes and surrogate pairs land on every chunk boundary, and a fuzz harness that now sweeps small stack sizes and requires the status of every chunking (stride 1..15 plus every single split point) to match the whole-buffer status exactly.	2026-10-27 16:20:00 -04:00