Address review: keep the transient-token reserve out of the public header
CI / build (-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, clang-arm64, ubuntu-latest-arm64, true) (pull_request) Successful in 1m12s
CI / build (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc-arm64, ubuntu-latest-arm64, false) (pull_request) Successful in 53s
CI / pre-commit (pull_request) Failing after 47s
CI / build (-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, clang-amd64, ubuntu-latest-amd64, true) (pull_request) Successful in 1m44s
CI / build (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc-amd64, ubuntu-latest-amd64, false) (pull_request) Successful in 1m42s

The create() documentation described how the reserve is implemented (six
extra slots for literals and \uXXXX escapes that straddle a chunk
boundary). That is an implementation detail; the header should only
promise the behaviour callers can rely on, i.e. that the accept/reject/
overflow decision depends on the document's bytes and on stackSize, not
on how they are split across parse() calls. The derivation of the upper
bound lives next to kTransientSymbols in src/parser3.h, where it can be
kept in sync with the pushes.

Also make the chunking test cover every split point, not just fixed
strides: each document is split in two at every possible offset, so every
literal, escape and surrogate pair in the fixture straddles a chunk
boundary. Verified that this fails on the pre-fix parser ("true",
stackSize 3, cut 1: whole buffer accepts, two chunks overflow).
This commit is contained in:
weaselbot committed 2026-10-27 17:31:49 -04:00
1 parent ac619af24d
commit 507be6f0a2
3 files changed
+58 -32

No files matched your search

+4 -8
View File
@@ -50,14 +50,10 @@ enum WeaselJsonFlags {
};
/** Create a parser. Increasing stack size increases memory usage but also
* increases the depth of nested json accepted. `stackSize` is an exact budget:
* whether a document is accepted, rejected or reported as too deeply nested
* depends only on its bytes and on `stackSize`, not on how the bytes are split
* across WeaselJsonParser_parse calls. To get that guarantee, the parser
* allocates 6 slots beyond `stackSize` for tokenizer progress: literals
* ("true", "false", "null") and \uXXXX escapes that straddle the end of a chunk
* record their unmatched characters there instead of spending the nesting
* budget. `callbacks` and `userdata` must outlive the returned parser. Returns
* increases the depth of nested json accepted. Whether a document is accepted,
* rejected or reported as too deeply nested depends only on its bytes and on
* `stackSize`, not on how the bytes are split across WeaselJsonParser_parse
* calls. `callbacks` and `userdata` must outlive the returned parser. Returns
* null if there's insufficient available memory, or if `stackSize` is negative
* or too small to hold a minimal document. */
WeaselJsonParser *WeaselJsonParser_create(int stackSize,