Status: this is an upstream LLVM/Clang codegen bug, not a preserve_none misuse in this repo.
preserve_none makes all general-purpose registers caller-saved. It does not require every call to be a tail call: a non-tail caller may call a preserve_none function as long as it saves any live state across the call. The failure here is the known AArch64 bug where the caller uses x19 as its frame base, a preserve_none callee clobbers x19, and the caller resumes using the clobbered value:
Confirmed by disassembly: both n_string2 and scan_string_impl are preserve_none and use mov x19, sp as their frame base; scan_string_impl overwrites x19 and never saves or restores it, so n_string2 resumes addressing its locals through the wrong frame. That is why emptying PRESERVE_NONE fixes it: the functions then save and restore x19 like ordinary code.
The reproducer, corpus failure, and PRESERVE_NONE-empty result below remain valid. The original root-cause text and "Suggested fix" section are corrected inline. Gating or removing PRESERVE_NONE is a workaround for this compiler bug, not the real fix.
Summary
PRESERVE_NONE (__attribute__((preserve_none))) is applied to functions that are not always reached through a tail call. n_string2() calls scan_string() with an ordinary call (src/parser3.h:649) and keeps buf/bufEnd live across it; because both functions are annotated preserve_none, the callee is allowed to clobber everything, and the caller's locals are corrupted.
In a plain build this happens to work, but with clang + AddressSanitizer it is reliably observable: the parser segfaults on the valid document "", and the repo's own fuzz_corpus ctest fails. A musttail-less build (the trampoline from #1) is affected too: there every continuation is called with an ordinary call, and the parser then reports WeaselJson_REJECT for "" instead of WeaselJson_AGAIN/WeaselJson_OK.
(conflict_set_shared_symbols / conflict_set_static_symbols also fail in that build, but only because ASan adds imported symbols; that is unrelated.)
With PRESERVE_NONE defined empty in src/preserve_none.h, the same -fsanitize=fuzzer,address build replays all 1840 corpus files without a crash.
Root cause (partly incorrect — see the status note above)
src/preserve_none.h defines PRESERVE_NONE as __attribute__((preserve_none)) whenever the compiler supports it. src/parser3.h:358-361 (and :346-355 on x86_64) annotate scan_string with it:
but n_string2 calls it with an ordinary call (src/parser3.h:644-651):
inlinePRESERVE_NONEContinuationStatusn_string2(Parser3*self,char*buf,char*bufEnd){...if(autos=scan_string(self,buf,bufEnd)){// line 649: not a tail call
returns;}switch(*buf){// line 652: SEGV
Instrumenting that build prints, for the repro above:
n_string2: before scan_string buf=0xfbff2a8d00a1 end=0xfbff2a8d00a2
n_string2: after scan_string buf=(nil) end=0x80000004ec884 rc=0
i.e. n_string2's own parameters were clobbered by the scan_string call, and the following switch (*buf) dereferences nullptr.
preserve_none means the callee preserves nothing (no callee-saved registers, no frame). It is not limited to tail calls: a non-tail caller may call a preserve_none function as long as it spills any live state across the call. The failure here is the upstream AArch64 bug in the status note (x19 used as a frame base and clobbered by a preserve_none callee). The two non-tail call sites are:
scan_string / scan_string_impl are PRESERVE_NONE but called with an ordinary call from n_string2 (and scan_string in turn makes a MUSTTAIL call, so the pair only works because of codegen luck).
When musttail is unavailable (MUSTTAIL empty), Parser3::parse's trampoline (src/parser3.h:1133-1146) calls everyPRESERVE_NONE continuation with an ordinary call and then relies on its own loop state. Simulating that build (musttail.h stubbed, preserve_none.h untouched) with clang -O0 -fsanitize=address makes the parser return WeaselJson_REJECT (2) for ""; emptying PRESERVE_NONE makes it return WeaselJson_AGAIN/WeaselJson_OK. This is a wrong answer on valid input, not just a crash.
Impact
Any AddressSanitizer build of the library (a normal thing to do while working on the parser, and the standard way to run src/fuzz.cpp) segfaults on valid JSON within a few inputs, and can silently misjudge valid documents in the trampoline configuration. Until this is fixed the project cannot be fuzz-tested with ASan, so real memory-safety regressions in parser3.h cannot be caught.
It contradicts the README's "Robust to crashes with untrusted input" for that (very common) build configuration.
preserve_none is a micro-optimization here — it only avoids saving callee-saved registers in the tail-call chain — so dropping it costs almost nothing.
Suggested workaround
Remove PRESERVE_NONE from scan_string/scan_string_impl (they are not exclusively tail-called), or make n_string2's call a real tail call.
Only define PRESERVE_NONE where it is sound, e.g. gate it on HAS_MUSTTAILand on the build not using a sanitizer:
With PRESERVE_NONE empty, the full corpus passes under -fsanitize=fuzzer,address and the tests above pass, so a targeted fix is easy to validate.
> **Status: this is an upstream LLVM/Clang codegen bug, not a `preserve_none` misuse in this repo.**
>
> `preserve_none` makes all general-purpose registers caller-saved. It does not require every call to be a tail call: a non-tail caller may call a `preserve_none` function as long as it saves any live state across the call. The failure here is the known AArch64 bug where the caller uses x19 as its frame base, a `preserve_none` callee clobbers x19, and the caller resumes using the clobbered value:
>
> - [llvm/llvm-project#70577](https://github.com/llvm/llvm-project/issues/70577)
> - [llvm/llvm-project#177519](https://github.com/llvm/llvm-project/issues/177519)
> - Same root cause: [weaselab/conflict-set#38](https://git.weaselab.dev/weaselab/conflict-set/issues/38)
>
> Confirmed by disassembly: both `n_string2` and `scan_string_impl` are `preserve_none` and use `mov x19, sp` as their frame base; `scan_string_impl` overwrites x19 and never saves or restores it, so `n_string2` resumes addressing its locals through the wrong frame. That is why emptying `PRESERVE_NONE` fixes it: the functions then save and restore x19 like ordinary code.
>
> The reproducer, corpus failure, and `PRESERVE_NONE`-empty result below remain valid. The original root-cause text and "Suggested fix" section are corrected inline. Gating or removing `PRESERVE_NONE` is a workaround for this compiler bug, not the real fix.
## Summary
`PRESERVE_NONE` (`__attribute__((preserve_none))`) is applied to functions that are **not** always reached through a tail call. `n_string2()` calls `scan_string()` with an ordinary call (`src/parser3.h:649`) and keeps `buf`/`bufEnd` live across it; because both functions are annotated `preserve_none`, the callee is allowed to clobber everything, and the caller's locals are corrupted.
In a plain build this happens to work, but with clang + AddressSanitizer it is reliably observable: the parser segfaults on the valid document `""`, and the repo's own `fuzz_corpus` ctest fails. A `musttail`-less build (the trampoline from #1) is affected too: there *every* continuation is called with an ordinary call, and the parser then reports `WeaselJson_REJECT` for `""` instead of `WeaselJson_AGAIN`/`WeaselJson_OK`.
Environment: clang 22.1.8 (aarch64, Fedora), cmake 3.x + ninja. `g++ 16.1.1` shows no symptom.
## Reproduction (public API only)
```cpp
// repro.cpp
#include "weaseljson.h"
#include <cstdio>
#include <string>
static void e(void *) {}
static void d(void *, const char *, int, int) {}
int main() {
WeaselJsonCallbacks c;
c.on_begin_object = e; c.on_end_object = e;
c.on_begin_array = e; c.on_end_array = e;
c.on_string_data = d; c.on_key_data = d; c.on_number_data = d;
c.on_true_literal = e; c.on_false_literal = e; c.on_null_literal = e;
auto *p = WeaselJsonParser_create(1024, &c, nullptr, 0);
std::string doc = "\"\""; // minimal valid document
printf("parse=%d\n", (int)WeaselJsonParser_parse(p, doc.data(), (int)doc.size()));
printf("eof=%d\n", (int)WeaselJsonParser_parse(p, nullptr, 0));
WeaselJsonParser_destroy(p);
}
```
```sh
clang++ -std=c++20 -O0 -g -fsanitize=address -I include repro.cpp src/lib.cpp -o repro && ./repro
```
Expected:
```
parse=1 # WeaselJson_AGAIN
eof=0 # WeaselJson_OK
```
Actual:
```
AddressSanitizer:DEADLYSIGNAL
ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000
#0 parser3::n_string2(parser3::Parser3*, char*, char*) src/parser3.h:652:11
#1 parser3::Parser3::parse(char*, int) src/parser3.h:1139:7
#2 WeaselJsonParser_parse src/lib.cpp:51:31
```
Sanity checks on the same document and source tree:
| build | result |
|---|---|
| `clang++ -O0` (no sanitizer) | `parse=1`, `eof=0` |
| `g++ -O0 -fsanitize=address` | `parse=1`, `eof=0` |
| `clang++ -O0 -fsanitize=address` | SEGV |
| `clang++ -O0 -fsanitize=address`, `preserve_none.h`'s `PRESERVE_NONE` defined empty | `parse=1`, `eof=0` |
## The same failure in the project's own test suite
```sh
cmake -S . -B build-asan -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_CXX_FLAGS='-fsanitize=address -O1' \
-DCMAKE_EXE_LINKER_FLAGS=-fsanitize=address \
-DCMAKE_SHARED_LINKER_FLAGS=-fsanitize=address
cmake --build build-asan
cd build-asan && ctest -R fuzz_corpus --output-on-failure
```
`fuzz_corpus` fails on the *first* corpus file (`corpus/0011ff84601edda95be93d80e939250a787fea01`), again in `parser3::n_string2`:
```
Running: .../corpus/0011ff84601edda95be93d80e939250a787fea01
AddressSanitizer:DEADLYSIGNAL
ERROR: AddressSanitizer: SEGV on unknown address 0x000000000060
#0 parser3::n_string2 ... parser3.h
#1 parser3::Parser3::parse ... parser3.h:1139
#2 WeaselJsonParser_parse ... lib.cpp:51
#3 toValue ... json_value.h:214
#4 testStringRoundTrip ... fuzz.cpp:168
```
(`conflict_set_shared_symbols` / `conflict_set_static_symbols` also fail in that build, but only because ASan adds imported symbols; that is unrelated.)
With `PRESERVE_NONE` defined empty in `src/preserve_none.h`, the same `-fsanitize=fuzzer,address` build replays all 1840 corpus files without a crash.
## Root cause (partly incorrect — see the status note above)
`src/preserve_none.h` defines `PRESERVE_NONE` as `__attribute__((preserve_none))` whenever the compiler supports it. `src/parser3.h:358-361` (and `:346-355` on x86_64) annotate `scan_string` with it:
```cpp
inline PRESERVE_NONE ContinuationStatus scan_string(Parser3 *self, char *&buf,
char *bufEnd) {
MUSTTAIL return scan_string_impl<simd<int8_t, 32>>(self, buf, bufEnd);
}
```
but `n_string2` calls it with an ordinary call (`src/parser3.h:644-651`):
```cpp
inline PRESERVE_NONE ContinuationStatus n_string2(Parser3 *self, char *buf,
char *bufEnd) {
...
if (auto s = scan_string(self, buf, bufEnd)) { // line 649: not a tail call
return s;
}
switch (*buf) { // line 652: SEGV
```
Instrumenting that build prints, for the repro above:
```
n_string2: before scan_string buf=0xfbff2a8d00a1 end=0xfbff2a8d00a2
n_string2: after scan_string buf=(nil) end=0x80000004ec884 rc=0
```
i.e. `n_string2`'s own parameters were clobbered by the `scan_string` call, and the following `switch (*buf)` dereferences `nullptr`.
`preserve_none` means the callee preserves nothing (no callee-saved registers, no frame). It is **not** limited to tail calls: a non-tail caller may call a `preserve_none` function as long as it spills any live state across the call. The failure here is the upstream AArch64 bug in the status note (x19 used as a frame base and clobbered by a `preserve_none` callee). The two non-tail call sites are:
1. `scan_string` / `scan_string_impl` are `PRESERVE_NONE` but called with an ordinary call from `n_string2` (and `scan_string` in turn makes a `MUSTTAIL` call, so the pair only works because of codegen luck).
2. When `musttail` is unavailable (`MUSTTAIL` empty), `Parser3::parse`'s trampoline (`src/parser3.h:1133-1146`) calls *every* `PRESERVE_NONE` continuation with an ordinary call and then relies on its own loop state. Simulating that build (`musttail.h` stubbed, `preserve_none.h` untouched) with clang `-O0 -fsanitize=address` makes the parser return `WeaselJson_REJECT` (2) for `""`; emptying `PRESERVE_NONE` makes it return `WeaselJson_AGAIN`/`WeaselJson_OK`. This is a wrong answer on valid input, not just a crash.
## Impact
* Any AddressSanitizer build of the library (a normal thing to do while working on the parser, and the standard way to run `src/fuzz.cpp`) segfaults on valid JSON within a few inputs, and can silently misjudge valid documents in the trampoline configuration. Until this is fixed the project cannot be fuzz-tested with ASan, so real memory-safety regressions in `parser3.h` cannot be caught.
* It contradicts the README's "Robust to crashes with untrusted input" for that (very common) build configuration.
* `preserve_none` is a micro-optimization here — it only avoids saving callee-saved registers in the tail-call chain — so dropping it costs almost nothing.
## Suggested workaround
* Remove `PRESERVE_NONE` from `scan_string`/`scan_string_impl` (they are not exclusively tail-called), or make `n_string2`'s call a real tail call.
* Only define `PRESERVE_NONE` where it is sound, e.g. gate it on `HAS_MUSTTAIL` *and* on the build not using a sanitizer:
```c
#if __has_attribute(preserve_none) && defined(HAS_MUSTTAIL) && \
!defined(__SANITIZE_ADDRESS__) && !__has_feature(address_sanitizer)
#define PRESERVE_NONE __attribute__((preserve_none))
#else
#define PRESERVE_NONE
#endif
```
With `PRESERVE_NONE` empty, the full corpus passes under `-fsanitize=fuzzer,address` and the tests above pass, so a targeted fix is easy to validate.
andrew
changed title from Parser segfaults on valid input when built with clang + AddressSanitizer: PRESERVE_NONE is applied to functions that are called non-tail to aarch64 + ASan: clang miscompiles preserve_none (x19 frame-base clobber); parser segfaults on valid input2026-11-10 19:54:49 +00:00
weaselbot
was assigned by andrew2026-11-10 19:57:21 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
PRESERVE_NONE(__attribute__((preserve_none))) is applied to functions that are not always reached through a tail call.n_string2()callsscan_string()with an ordinary call (src/parser3.h:649) and keepsbuf/bufEndlive across it; because both functions are annotatedpreserve_none, the callee is allowed to clobber everything, and the caller's locals are corrupted.In a plain build this happens to work, but with clang + AddressSanitizer it is reliably observable: the parser segfaults on the valid document
"", and the repo's ownfuzz_corpusctest fails. Amusttail-less build (the trampoline from #1) is affected too: there every continuation is called with an ordinary call, and the parser then reportsWeaselJson_REJECTfor""instead ofWeaselJson_AGAIN/WeaselJson_OK.Environment: clang 22.1.8 (aarch64, Fedora), cmake 3.x + ninja.
g++ 16.1.1shows no symptom.Reproduction (public API only)
Expected:
Actual:
Sanity checks on the same document and source tree:
clang++ -O0(no sanitizer)parse=1,eof=0g++ -O0 -fsanitize=addressparse=1,eof=0clang++ -O0 -fsanitize=addressclang++ -O0 -fsanitize=address,preserve_none.h'sPRESERVE_NONEdefined emptyparse=1,eof=0The same failure in the project's own test suite
fuzz_corpusfails on the first corpus file (corpus/0011ff84601edda95be93d80e939250a787fea01), again inparser3::n_string2:(
conflict_set_shared_symbols/conflict_set_static_symbolsalso fail in that build, but only because ASan adds imported symbols; that is unrelated.)With
PRESERVE_NONEdefined empty insrc/preserve_none.h, the same-fsanitize=fuzzer,addressbuild replays all 1840 corpus files without a crash.Root cause (partly incorrect — see the status note above)
src/preserve_none.hdefinesPRESERVE_NONEas__attribute__((preserve_none))whenever the compiler supports it.src/parser3.h:358-361(and:346-355on x86_64) annotatescan_stringwith it:but
n_string2calls it with an ordinary call (src/parser3.h:644-651):Instrumenting that build prints, for the repro above:
i.e.
n_string2's own parameters were clobbered by thescan_stringcall, and the followingswitch (*buf)dereferencesnullptr.preserve_nonemeans the callee preserves nothing (no callee-saved registers, no frame). It is not limited to tail calls: a non-tail caller may call apreserve_nonefunction as long as it spills any live state across the call. The failure here is the upstream AArch64 bug in the status note (x19 used as a frame base and clobbered by apreserve_nonecallee). The two non-tail call sites are:scan_string/scan_string_implarePRESERVE_NONEbut called with an ordinary call fromn_string2(andscan_stringin turn makes aMUSTTAILcall, so the pair only works because of codegen luck).musttailis unavailable (MUSTTAILempty),Parser3::parse's trampoline (src/parser3.h:1133-1146) calls everyPRESERVE_NONEcontinuation with an ordinary call and then relies on its own loop state. Simulating that build (musttail.hstubbed,preserve_none.huntouched) with clang-O0 -fsanitize=addressmakes the parser returnWeaselJson_REJECT(2) for""; emptyingPRESERVE_NONEmakes it returnWeaselJson_AGAIN/WeaselJson_OK. This is a wrong answer on valid input, not just a crash.Impact
src/fuzz.cpp) segfaults on valid JSON within a few inputs, and can silently misjudge valid documents in the trampoline configuration. Until this is fixed the project cannot be fuzz-tested with ASan, so real memory-safety regressions inparser3.hcannot be caught.preserve_noneis a micro-optimization here — it only avoids saving callee-saved registers in the tail-call chain — so dropping it costs almost nothing.Suggested workaround
PRESERVE_NONEfromscan_string/scan_string_impl(they are not exclusively tail-called), or maken_string2's call a real tail call.PRESERVE_NONEwhere it is sound, e.g. gate it onHAS_MUSTTAILand on the build not using a sanitizer:With
PRESERVE_NONEempty, the full corpus passes under-fsanitize=fuzzer,addressand the tests above pass, so a targeted fix is easy to validate.Parser segfaults on valid input when built with clang + AddressSanitizer: PRESERVE_NONE is applied to functions that are called non-tailto aarch64 + ASan: clang miscompiles preserve_none (x19 frame-base clobber); parser segfaults on valid input