247 lines
11 KiB
Markdown
247 lines
11 KiB
Markdown
# C++ translation of `Statement.closedForm`
|
||
|
||
This directory contains a C++20 translation of `Statement.closedForm` from
|
||
`Statement.lean`, an equivalent summarised implementation, a libFuzzer
|
||
differential harness, and a nanobench comparison.
|
||
|
||
The Lean build is untouched.
|
||
|
||
## Layout
|
||
|
||
| Path | Purpose |
|
||
| --- | --- |
|
||
| `src/model.h` | Data model: borrowed input keys (`std::string_view`), owned result keys (`std::string`), `WithTop Key` = `std::optional`, `ℕ∞` = `NatInf`. |
|
||
| `src/literal.cpp` | Line-by-line transliteration of `Statement.closedForm`, including its repeated scans. |
|
||
| `src/fast.cpp` | The same decision tree, but with a summary of the reported keys and lazily cached cardinalities. |
|
||
| `oracle/gen.lean` | Deterministic generator of golden vectors by evaluating the real Lean `closedForm`. |
|
||
| `tests/golden.cpp` | Checks the literal translation against the Lean golden vectors. |
|
||
| `tests/diff_corpus.cpp` | Compares literal and fast on the sorted subset of the golden corpus. |
|
||
| `tests/readable_gen.h` | Readable-by-construction instance generator (owns the selector keys and reported keys, with borrowed views into them). |
|
||
| `tests/extreme.cpp` | Regression test for offsets at the ends of the 32-bit input range. |
|
||
| `tests/exhaustive.cpp` | Deterministic exhaustive sweep of small readable instances (also the coverage driver). |
|
||
| `tests/fuzz_readable.cpp` | libFuzzer differential test: literal vs. fast on readable instances. |
|
||
| `tests/fuzz_raw.cpp` | libFuzzer safety test on arbitrary instances under ASan/UBSan. |
|
||
| `tests/bench.cpp` | nanobench literal-vs-fast benchmark. |
|
||
| `run-fuzz.sh` | Rebuilds and runs both fuzzers for a fixed time. |
|
||
| `run-coverage.sh` | Builds with Clang source-based coverage and requires 100% line and branch coverage of `src/fast.cpp`. |
|
||
| `third_party/nanobench.h` | Vendored nanobench (commit `7252da9d96bd0d71f508e9b2813ab1f57fdfaec1`). |
|
||
|
||
## Data model
|
||
|
||
Inputs borrow and results own. `RangeRead` selectors and the reported keys in
|
||
`Result` are `std::string_view`s (the reported keys via a `std::span`), so a
|
||
caller can point them straight at existing storage and the implementation reads
|
||
them in place. `ConflictRange` returns owned `std::string`s because the decision
|
||
procedure can synthesise endpoints (`keyAfter`, clamp bounds) that are not
|
||
present in the input.
|
||
|
||
Selector offsets are `std::int32_t`, matching the caller contract. The closed
|
||
form converts the public `RangeRead` to `detail::RangeRead`, whose offsets are
|
||
`Int = std::int64_t`, before deriving selectors (for example
|
||
`end.offset - residualT`). All intermediate `Int` arithmetic assumes 64-bit
|
||
signed integers do not overflow; with 32-bit input offsets and in-memory key
|
||
counts that is far from the limit. The corpus parsers read offsets into a wider
|
||
integer and reject values outside the 32-bit range before narrowing to
|
||
`Selector`.
|
||
|
||
## Scope of the two implementations
|
||
|
||
`literal.cpp` is the total transliteration: it is defined on arbitrary inputs,
|
||
and `golden_test` compares it with Lean over the whole generated corpus, most of
|
||
which is not readable. `fast.cpp` is the readable-only implementation:
|
||
`diff_corpus` filters the corpus to readable cases, and the fuzzers and the
|
||
exhaustive sweep generate readable instances. The Lean anchor therefore covers
|
||
`literal` on all inputs, while `fast` is anchored to `literal` on the readable
|
||
subset.
|
||
|
||
## Build
|
||
|
||
```sh
|
||
cd cpp
|
||
cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER=clang++-21
|
||
cmake --build build
|
||
```
|
||
|
||
## Golden vectors from Lean
|
||
|
||
```sh
|
||
cd ..
|
||
lake env lean --run cpp/oracle/gen.lean > cpp/oracle/golden.txt
|
||
./cpp/build/golden_test cpp/oracle/golden.txt
|
||
# golden: 40000 cases, 0 mismatches
|
||
```
|
||
|
||
The golden test compares **exact endpoints**, because the literal translation
|
||
reproduces Lean's interval encoding exactly (including empty intervals).
|
||
|
||
`diff_corpus` compares the literal and fast implementations on the generated
|
||
cases that satisfy the readable sorting invariant:
|
||
|
||
```sh
|
||
./cpp/build/diff_corpus cpp/oracle/golden.txt
|
||
```
|
||
|
||
`extreme_test` checks both implementations on offsets from `INT32_MIN` to
|
||
`INT32_MAX`, which the fuzzers do not generate:
|
||
|
||
```sh
|
||
./cpp/build/extreme_test
|
||
# extreme: 7574 cases, 0 mismatches
|
||
```
|
||
|
||
`exhaustive_test` enumerates a specific finite class rather than sampling it:
|
||
every sorted database of at most four stored keys drawn from a fixed universe of
|
||
boundary keys, every selector key (including `⊥` and `⊤`) with offsets in
|
||
`[-3, 5]`, both traversal directions, and every readable result shape (the
|
||
complete traversal, each nonempty truncated prefix, or the empty result when the
|
||
traversal is empty):
|
||
|
||
```sh
|
||
./cpp/build/exhaustive_test
|
||
# exhaustive: 13565772 cases, 0 mismatches
|
||
```
|
||
|
||
It runs in about a second and is also the coverage driver. The class is not all
|
||
readable inputs; the guarantee is that every instance in this family agrees. In
|
||
particular, branch coverage cannot exercise a predicate operand that the
|
||
implementation no longer contains: the coverage result below describes the code
|
||
as written, it is not a proof of equivalence with `literal.cpp`.
|
||
|
||
## Coverage
|
||
|
||
```sh
|
||
./cpp/run-coverage.sh
|
||
```
|
||
|
||
The script configures a Clang source-based-coverage build, runs
|
||
`exhaustive_test`, and fails unless `src/fast.cpp` has 100% line and branch
|
||
coverage. `exhaustive_test` alone covers every region, function, line, and
|
||
branch side of the summarised implementation:
|
||
|
||
```text
|
||
src/fast.cpp 570 regions (0 missed) 123 functions (0 missed) 684 lines (0 missed) 388 branches (0 missed)
|
||
```
|
||
|
||
That is 100.00% line, function, region, and branch coverage. Offset 5 is part of
|
||
the sweep specifically to exercise the last three short-circuit branch sides,
|
||
which the previous `[-3, 4]` window missed.
|
||
|
||
Because equivalence is only promised on readable inputs, `src/fast.cpp` deletes
|
||
both unreachable control-flow branches and predicate operands that an enclosing
|
||
dispatch invariant fixes. Examples include the complete-closed family, the
|
||
offset-one total/any wrappers, the `end.offset <= 1` shortcuts, the
|
||
`crossedCompleteEndLeOne` guard, and the repeated
|
||
`!c.r->reverse`/`!o.more`/offset tests inside subtrees that are only entered
|
||
after those tests succeeded. `literal.cpp` keeps the full Lean conditions so the
|
||
translation can still be audited line by line.
|
||
|
||
## Fuzzing
|
||
|
||
```sh
|
||
cd cpp
|
||
mkdir -p /tmp/cr-corpus/readable /tmp/cr-corpus/raw
|
||
./build/fuzz_readable -max_total_time=300 /tmp/cr-corpus/readable
|
||
./build/fuzz_raw -max_total_time=300 /tmp/cr-corpus/raw
|
||
```
|
||
|
||
or simply `./cpp/run-fuzz.sh` (optionally `./cpp/run-fuzz.sh build 600`).
|
||
The CMake fuzz targets compile the implementation sources directly so that the
|
||
decoder code, not just the harness, is covered by ASan/UBSan.
|
||
|
||
`fuzz_readable` builds a sorted, duplicate-free database, resolves both
|
||
selectors against it, and reports either the complete traversal or a nonempty
|
||
prefix. This makes every generated pair genuinely `Readable`. The differential
|
||
check treats all encodings of the empty interval as equivalent and otherwise
|
||
requires exact endpoints.
|
||
|
||
`fuzz_raw` only checks memory safety and undefined behaviour for arbitrary
|
||
(possibly unreadable) inputs.
|
||
|
||
One representative validation run on this machine:
|
||
|
||
| check | cases / runs | result |
|
||
| --- | ---: | --- |
|
||
| literal vs. Lean golden vectors | 40,000 | 0 mismatches (exact endpoints) |
|
||
| fast vs. literal on sorted corpus | 18,971 | 0 mismatches |
|
||
| extreme-offset regression (`extreme_test`) | 7,574 | 0 mismatches |
|
||
| exhaustive readable sweep (`exhaustive_test`) | 13,565,772 | 0 mismatches |
|
||
| readable differential fuzzing (60 s) | 1,182,424 | 0 mismatches |
|
||
| raw safety fuzzing (60 s) | 1,905,270 | no ASan/UBSan findings |
|
||
|
||
`extreme_test` covers offsets the fuzzers deliberately avoid (they draw offsets
|
||
from `[-256, 255]`). Both implementations decide the closed form's
|
||
`keyAfterN n k < upper` test from the key lengths (`keyAfterNLt`) rather than
|
||
materialising `n` zero bytes, so an `end.offset` near `INT32_MAX` stays
|
||
constant-time instead of attempting a 2 GiB allocation.
|
||
|
||
## Benchmark
|
||
|
||
```sh
|
||
./cpp/build/cr_bench
|
||
```
|
||
|
||
Representative results (Clang 21, default Release build: `-O3 -DNDEBUG`; medians
|
||
of nine runs pinned to one core; relative to the literal translation, lower is
|
||
better for `fast`).
|
||
CPU frequency scaling makes the absolute numbers noisy, but the shape is stable:
|
||
|
||
| case | literal | fast | fast/literal |
|
||
| --- | ---: | ---: | ---: |
|
||
| small complete (8 keys) | 41.6 ns | 16.2 ns | 0.39x |
|
||
| small truncated forward | 16.9 ns | 22.9 ns | 1.35x |
|
||
| crossed truncated forward | 56.0 ns | 46.6 ns | 0.83x |
|
||
| crossed truncated reverse | 130.1 ns | 82.7 ns | 0.64x |
|
||
| long complete forward (512 keys) | 2189 ns | 32.6 ns | 0.01x |
|
||
| long truncated forward | 15.0 ns | 19.6 ns | 1.31x |
|
||
| long truncated reverse | 9.5 ns | 12.8 ns | 1.35x |
|
||
|
||
The large win is on complete reads: the literal translation rescans the reported
|
||
keys many times (`residualT`, `reportedBelowBegin`, `allBelowEnd`, ...), while
|
||
the summary answers those questions in `O(log n)` or from a cached count.
|
||
Plain truncated cases are dominated by the shared decision tree, so the two
|
||
implementations are close; the literal path is slightly cheaper there because it
|
||
performs no summary setup. Crossed cases report 16 of a 33-key range, so the
|
||
summary wins there too.
|
||
|
||
Against the same implementation with owned input keys (the previous model), the
|
||
borrowed-key model changes the costs as follows (nine interleaved runs each,
|
||
median). The crossed cases are absent from this table: their benchmark inputs
|
||
were corrected to nonempty ranges after the owned-key measurements.
|
||
|
||
| case | literal (borrow vs own) | fast (borrow vs own) |
|
||
| --- | ---: | ---: |
|
||
| small complete | -64% | -17% |
|
||
| small truncated forward | -33% | -19% |
|
||
| long complete forward | -69% | -15% |
|
||
| long truncated forward | -29% | -6% |
|
||
| long truncated reverse | -16% | +16% |
|
||
|
||
The complete-read gains come from not copying the reported keys just to
|
||
normalise the result: a forward result is now the caller's span, and a reverse
|
||
result is a reversed view rather than an owned copy. The fast path also benefits
|
||
because it previously normalised complete results the same way. Truncated
|
||
reverse (`fast`) is the one small regression: it is a few nanoseconds and a few
|
||
dozen instructions, and the other paths move in the opposite direction, so it
|
||
reads as code-layout/code-generation noise rather than extra algorithmic work.
|
||
|
||
|
||
## Why the fast implementation is equivalent
|
||
|
||
The fast implementation replays the same decision tree on readable inputs. The
|
||
only changes are:
|
||
|
||
* `reportedBelowBegin` and `residualT` are summarised once. On readable inputs the
|
||
reported keys are sorted in traversal order, so both counts are binary
|
||
searches. The summaries depend only on the read and on the multiset of
|
||
reported keys, so they survive the reversed-view normalisation.
|
||
* Interval cardinalities (`intervalCard`) are cached lazily.
|
||
* Complete observations are not copied: a forward result borrows the caller's
|
||
span, and a reversed result is viewed backwards in place. (The previous
|
||
owned-key model copied and reversed the reported keys here.)
|
||
* `allBelowEnd` is `residualT == 0` and `anyAtOrAboveEnd` is `residualT != 0`.
|
||
|
||
No branch condition or returned endpoint is otherwise changed on readable
|
||
inputs; branches that `Readable` makes unreachable are omitted. Equivalence is
|
||
therefore only claimed for readable inputs, and `fuzz_raw` deliberately does not
|
||
compare outputs on arbitrary inputs.
|