weaselbot
d040ed63a4
CI / release (arm64, , ubuntu-latest-arm64) (pull_request) Successful in 2m18s
CI / pre-commit (pull_request) Successful in 1m58s
CI / test (-DCMAKE_BUILD_TYPE=Debug -DMSAN_TOOLCHAIN_PATH=/opt/msan, debug) (pull_request) Successful in 3m47s
CI / test (-DCMAKE_CXX_FLAGS=-DUSE_64_BIT=1, 64-bit-versions) (pull_request) Successful in 3m16s
CI / test (-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++, gcc) (pull_request) Successful in 3m12s
CI / test (-DUSE_SIMD_FALLBACK=ON, simd-fallback) (pull_request) Successful in 3m17s
CI / release (amd64, -DMSAN_TOOLCHAIN_PATH=/opt/msan, ubuntu-latest-amd64) (pull_request) Successful in 5m32s
CI / coverage (pull_request) Successful in 3m42s
The aarch64 NEON paths in getNodeIndex/getNodeIndexExists,
getChildGeq(Node16*), scan16, and checkMaxBetweenExclusiveImpl<Node16>
loaded the full 16-element Node16::index array (and the Node48
reverseIndex array via scan16) with NEON intrinsics, then masked the
result down to [0, numChildren). Only the in-use slots are initialized;
the unused bytes are indeterminate, so the wide loads were undefined
behavior in C++ (per [basic.indet]) even though the trailing lanes were
discarded. MSan reports this on x86-64; on aarch64 it is the same UB but
MSan's imprecise modeling doesn't flag it at -O0, so there is no red->green
test.
Mirror the existing x86-64 fix (commit 6fed133): implement the index
operations in file-level assembly, where loading and operating on
indeterminate values is well-defined. Add simd_aarch64.S with
find_eq_16, find_ge_16, and mask_in_range_16. AArch64 lacks pmovmskb, so
(like the prior NEON code) these return a 64-bit nibble mask rather than
a 16-bit bitmask; the C++ call sites keep their existing nibble-mask
arithmetic and only swap the inline NEON load/compare for the assembly
call. The childMaxVersion compares stay in C++ NEON intrinsics, matching
x86-64's compare16: those slots are always initialized to zero by the
allocator, so the wide loads are defined.
The assembly functions carry `bti c` landing pads and the same
aeabi_feature_and_bits attributes the compiler emits for
-mbranch-protection=standard, so the object stays BTI/PAC/GCS-compatible
(and warning-free under -z force-bti). CMakeLists.txt builds simd_aarch64.S
into the object library and the SIMD test/bench/fuzz targets on aarch64.
Closes #68
49 lines
2.0 KiB
C++
49 lines
2.0 KiB
C++
#pragma once
|
|
|
|
#include <cstdint>
|
|
|
|
#if defined(__x86_64__) && !defined(USE_SIMD_FALLBACK)
|
|
|
|
// SIMD operations on potentially-indeterminate Node16::index[16] bytes.
|
|
// Implemented in file-level assembly (simd_x86_64.S) because loading and
|
|
// operating on indeterminate values is UB in C++ but well-defined in
|
|
// assembly. The caller must mask the returned bitfield to
|
|
// [0, numChildren) before using it.
|
|
//
|
|
// Each function returns a 16-bit bitmask in the low 16 bits of a uint32_t
|
|
// (upper 16 bits are zero). Bit i is set iff the condition holds at index i.
|
|
|
|
extern "C" {
|
|
// Returns bit i set iff idx[i] == key
|
|
uint32_t find_eq_16(const uint8_t idx[16], uint8_t key);
|
|
// Returns bit i set iff idx[i] >= child
|
|
uint32_t find_ge_16(const uint8_t idx[16], uint8_t child);
|
|
// Returns bit i set iff begin <= idx[i] < end
|
|
uint32_t mask_in_range_16(const uint8_t idx[16], uint8_t begin, uint8_t end);
|
|
}
|
|
|
|
#elif defined(__aarch64__) && !defined(USE_SIMD_FALLBACK)
|
|
|
|
// SIMD operations on potentially-indeterminate Node16::index[16] bytes.
|
|
// Implemented in file-level assembly (simd_aarch64.S) because loading and
|
|
// operating on indeterminate values is UB in C++ but well-defined in
|
|
// assembly. The caller must mask the returned bitfield to [0, numChildren)
|
|
// before using it.
|
|
//
|
|
// AArch64 has no pmovmskb-equivalent, so (unlike x86-64) each function returns
|
|
// a 64-bit "nibble mask": nibble i (bits [4i, 4i+4)) is 0xf iff the condition
|
|
// holds at index i. Bit (4i + 3) is the high bit of byte i's result. Callers
|
|
// locate a set lane with std::countr_zero(bitfield) / 4 and mask the valid
|
|
// lanes with (uint64_t(1) << (numChildren * 4)) - 1.
|
|
|
|
extern "C" {
|
|
// Returns nibble i = 0xf iff idx[i] == key
|
|
uint64_t find_eq_16(const uint8_t idx[16], uint8_t key);
|
|
// Returns nibble i = 0xf iff idx[i] >= child
|
|
uint64_t find_ge_16(const uint8_t idx[16], uint8_t child);
|
|
// Returns nibble i = 0xf iff begin <= idx[i] < end
|
|
uint64_t mask_in_range_16(const uint8_t idx[16], uint8_t begin, uint8_t end);
|
|
}
|
|
|
|
#endif
|