parser3: reserve stack slots for chunk-split literals and escapes
CI / build (-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, clang-arm64, ubuntu-latest-arm64, true) (pull_request) Successful in 1m3s
CI / build (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc-arm64, ubuntu-latest-arm64, false) (pull_request) Successful in 54s
CI / pre-commit (pull_request) Failing after 49s
CI / build (-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, clang-amd64, ubuntu-latest-amd64, true) (pull_request) Successful in 1m46s
CI / build (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc-amd64, ubuntu-latest-amd64, false) (pull_request) Successful in 1m36s
CI / build (-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, clang-arm64, ubuntu-latest-arm64, true) (pull_request) Successful in 1m3s
CI / build (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc-arm64, ubuntu-latest-arm64, false) (pull_request) Successful in 54s
CI / pre-commit (pull_request) Failing after 49s
CI / build (-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, clang-amd64, ubuntu-latest-amd64, true) (pull_request) Successful in 1m46s
CI / build (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc-amd64, ubuntu-latest-amd64, false) (pull_request) Successful in 1m36s
WeaselJson_OVERFLOW is documented as "json is too deeply nested", and stackSize as a nesting-depth budget. In practice the pushdown stack also held tokenizer progress for a `true`/`false`/`null` literal or a \uXXXX escape that straddled the end of a chunk: the whole-buffer path consumes those characters inline and pushes nothing, while the slow path pushed up to six symbols (a surrogate pair) onto the same stack. So the stackSize a document needed depended on where the parse() boundaries fell: e.g. "true" was accepted at stackSize 3 when fed whole but returned the sticky WeaselJson_OVERFLOW when fed one byte at a time. Reserve slots for those symbols in addition to the supplied stackSize, as a fixed upper bound rather than by changing the encoding: the largest single push of transient symbols is t_hex2()'s six symbols for a surrogate pair, and no transient symbol is ever pushed while another one is on the stack, so kTransientSymbols = 6 is enough. Nesting pushes are still bounded by stackSize, so OVERFLOW now means exactly what the header says and the accept/reject/overflow decision is a function of the input bytes and stackSize only. Document the reserve in weaseljson.h and cover the property with tests: a doctest sweeping stackSize 3..16 over documents whose literals, escapes and surrogate pairs land on every chunk boundary, and a fuzz harness that now sweeps small stack sizes and requires the status of every chunking (stride 1..15 plus every single split point) to match the whole-buffer status exactly.
This commit is contained in:
1 parent
730a4cd0f6
commit
ac619af24d
5 files changed
+183
-64
No files matched your search
+11
-5
@@ -36,7 +36,7 @@ enum WeaselJsonStatus {
|
||||
WeaselJson_AGAIN,
|
||||
/** Invalid json */
|
||||
WeaselJson_REJECT,
|
||||
/** json is too deeply nested */
|
||||
/** json is too deeply nested for the parser's stack size */
|
||||
WeaselJson_OVERFLOW,
|
||||
/** Tried to call parse on a null parser */
|
||||
WeaselJson_NULL,
|
||||
@@ -50,10 +50,16 @@ enum WeaselJsonFlags {
|
||||
};
|
||||
|
||||
/** Create a parser. Increasing stack size increases memory usage but also
|
||||
* increases the depth of nested json accepted. `callbacks` and `userdata` must
|
||||
* outlive the returned parser. Returns null if there's insufficient available
|
||||
* memory, or if `stackSize` is negative or too small to hold a minimal
|
||||
* document. */
|
||||
* increases the depth of nested json accepted. `stackSize` is an exact budget:
|
||||
* whether a document is accepted, rejected or reported as too deeply nested
|
||||
* depends only on its bytes and on `stackSize`, not on how the bytes are split
|
||||
* across WeaselJsonParser_parse calls. To get that guarantee, the parser
|
||||
* allocates 6 slots beyond `stackSize` for tokenizer progress: literals
|
||||
* ("true", "false", "null") and \uXXXX escapes that straddle the end of a chunk
|
||||
* record their unmatched characters there instead of spending the nesting
|
||||
* budget. `callbacks` and `userdata` must outlive the returned parser. Returns
|
||||
* null if there's insufficient available memory, or if `stackSize` is negative
|
||||
* or too small to hold a minimal document. */
|
||||
WeaselJsonParser *WeaselJsonParser_create(int stackSize,
|
||||
const WeaselJsonCallbacks *callbacks,
|
||||
void *userdata, int flags);
|
||||
|
||||
Reference in new issue
Block a user