Compare commits

..
43 Commits
Author SHA1 Message Date
weaselbot dee3a8f640 Return 0 instead of -1 from hash_table getBytes()
The hash_table implementation cannot accurately track memory usage and was
returning -1 from ConflictSet::getBytes() and ConflictSet_getBytes(). That
violates the API contract that getBytes() returns a non-negative byte count.

Change both entry points to return 0 and document in ConflictSet.h that
implementations which do not track memory usage may return 0.

Add a regression test in test_conflict_set.py that loads the hash_table
implementation and verifies getBytes() is non-negative.

Closes #62
2026-07-07 11:00:52 -04:00
andrew 732d19efa1 Merge pull request 'Fix RealDataBench subspan and version API contract violations' (#61) from weaselbot/conflict-set:weaselbot/issue-58 into main
Reviewed-on: weaselab/conflict-set#61
2026-06-29 18:49:58 +00:00
andrew 9449190d02 Merge pull request 'Fix undefined behavior on empty input in strinc() and prefixRange()' (#60) from weaselbot/conflict-set:weaselbot/issue-59 into main
Reviewed-on: weaselab/conflict-set#60
2026-06-29 18:26:04 +00:00
weaselbot 4fcdc5d7e9 Fix RealDataBench subspan and version API contract violations
- Use subspan count `line.size() - 2` instead of `line.size()` to avoid reading past the line bounds, and guard lines shorter than the two-byte prefix.
- Clamp `readVersion` and `setOldestVersion` arguments to `0` so the conflict set never receives negative, non-monotonic versions during the warmup phase.

Fixes #58.
2026-06-29 14:01:23 -04:00
weaselbot 63f9a139da Fix undefined behavior on empty input in strinc() and prefixRange()
strinc() in ConflictSet.cpp used std::string_view::size() (size_t) and subtracted 1 without first checking for an empty string. For the root node, getSearchPath() returns the empty string, so every debug correctness check underflowed size_t and relied on implementation-defined conversion to signed int.

prefixRange() in Bench.cpp had the same loop shape. Although TrivialSpan::size() returns int, on an empty (or all-0xff) key the function then asserted and continued executing, allocating a zero-length buffer and writing before its start.

Changes:
- In strinc(), initialize index as signed int(str.size()) - 1 so the loop is skipped for empty input, and return ok=false cleanly.
- In prefixRange(), initialize index the same way and call std::abort() after the assert so invalid input cannot fall through to an out-of-bounds write.
- Replace C-style uint8_t casts with explicit static_casts.

Closes #59
2026-06-29 13:52:02 -04:00
andrew 8f9f345c64 Merge pull request 'Keep Python wrapper key buffers alive in WriteRange/ReadRange' (#43) from weaselbot/conflict-set:weaselbot/issue-42 into main
Reviewed-on: weaselab/conflict-set#43
Reviewed-by: andrew <andrew@weaselab.dev>
2026-06-23 01:17:45 +00:00
andrew b9b2d69dd5 Merge pull request 'Disallow copying ConflictSet in C++98/C++03' (#51) from weaselbot/conflict-set:weaselbot/issue-48 into main
Reviewed-on: weaselab/conflict-set#51
2026-06-22 23:44:59 +00:00
andrew d3c8f4afc6 Merge pull request 'Include <sys/syscall.h> in ServerBench.cpp for SYS_perf_event_open' (#44) from weaselbot/conflict-set:weaselbot/issue-41 into main
Reviewed-on: weaselab/conflict-set#44
2026-06-22 20:01:43 +00:00
andrew 549724f09e Merge pull request 'Assert return values in test_update_zero_should_commit / conflict' (#50) from weaselbot/conflict-set:weaselbot/issue-49 into main
Reviewed-on: weaselab/conflict-set#50
2026-06-22 19:29:55 +00:00
andrew 52eb13cc0b Merge pull request 'Fix CMake Unix Makefiles parallel build race for conflict-set.o' (#52) from weaselbot/conflict-set:weaselbot/issue-47 into main
Reviewed-on: weaselab/conflict-set#52
2026-06-22 19:08:09 +00:00
andrew f30887f280 Merge pull request 'Fix move-assignment leak and self-assignment in ConflictSet' (#55) from weaselbot/conflict-set:weaselbot/issue-54 into main
Reviewed-on: weaselab/conflict-set#55
2026-06-22 18:40:25 +00:00
andrew ccd637deab We already know __cplusplus is defined 2026-06-22 14:18:33 -04:00
weaselbotandandrew 971deb477c Keep Python key buffers alive in WriteRange/ReadRange
`write()` and `read()` created _Key objects from ephemeral ctypes arrays
backed by local bytearray objects. Once the helpers returned, those local
variables were freed, leaving the C library with dangling pointers when
addWrites()/check() later read the keys.

Store the backing bytearray on the returned WriteRange/ReadRange objects
as private `_begin_buf` / `_end_buf` attributes. Python keeps them alive
for the lifetime of the range object, so the C pointer is always valid.

Closes #42
2026-06-22 14:09:18 -04:00
weaselbotandandrew ff0722728a Include <sys/syscall.h> in ServerBench.cpp for SYS_perf_event_open
`ServerBench.cpp` calls `syscall(SYS_perf_event_open, ...)` but did
not include `<sys/syscall.h>`, relying on `<unistd.h>` to transitively
provide the `SYS_*` constants. On toolchains where that does not happen,
the build fails with `SYS_perf_event_open` not declared.

Add the missing include so `server_bench` compiles on any platform that
provides `perf_event_open`.

Resolves weaselab/conflict-set#41.
2026-06-22 14:09:14 -04:00
weaselbotandandrew 60881419b8 Assert return values in test_update_zero_should_commit / conflict
Add the missing [Result.COMMIT] / [Result.CONFLICT] assertions so
that the two regression tests actually verify the expected conflict-set
outcome, not just that the calls do not crash.
2026-06-22 14:09:10 -04:00
weaselbotandandrew 4515af3662 CMake: add target-level dependencies for conflict-set-object
The custom command that links conflict-set.o depends on
$<TARGET_OBJECTS:conflict-set-object>, but that generator expression
does not create a target-level dependency.  With the Unix Makefiles
generator, parallel builds can start building the consuming libraries
before the object library's build rule is available, producing:

  gmake[3]: *** No rule to make target
  'CMakeFiles/conflict-set-object.dir/ConflictSet.cpp.o', needed by
  'conflict-set.o'.  Stop.

Add add_dependencies() so that conflict-set and conflict-set-static
cannot build until conflict-set-object has produced its object files.

Closes #47
2026-06-22 14:09:04 -04:00
weaselbotandandrew 4dc5f7f75c Fix self-move-assignment and leak in ConflictSet move-assignment
The user-declared move-assignment operator overwrote `impl` without first
destroying the existing implementation object, leaking all memory and resources
owned by the left-hand side. Self-move-assignment also set `impl` to nullptr,
leaving the object invalid and leaking the old state.

Fix all three implementations (ConflictSet.cpp, SkipList.cpp, HashTable.cpp)
to guard against self-assignment and to destroy/free the old `impl` before
taking ownership of `other.impl`.
2026-06-22 14:08:59 -04:00
weaselbot 7eaac2a184 Make ConflictSet non-copyable in C++98/C++03
`ConflictSet(const ConflictSet&)` and `operator=(const ConflictSet&)` were
only deleted for C++11 and later. In C++98/C++03 the compiler implicitly
generated public copy operations, so copying a ConflictSet shared the opaque
`Impl*` and caused a double-free on destruction.

Declare both operations private and leave them undefined when
`__cplusplus <= 199711L`, matching the standard pre-C++11 idiom for
move-only types. Guard the declarations with `defined(__cplusplus)` so
they are not exposed to C90 compilation units.

Closes #48
2026-06-22 13:43:04 -04:00
andrew e9c904a86b CMakeLists.txt: don't require hardening-check --help to exit 0
hardening-check --help returns exit code 1, so the previous
hardening_check_help_result EQUAL 0 guard skipped parsing the help
output entirely. As a result, architecture-specific flags such as
--nobranchprotection on x86_64 were never added to the test command.

Parse the help output regardless of exit code; the option-string regex
checks are sufficient.
2026-06-22 13:05:33 -04:00
andrew 789ae8cbb9 ci: install clang/LLVM 21 in Gitea Actions workflows
Switch all CI jobs from the distro-packaged clang to the apt.llvm.org
clang-21 / llvm-21 toolchain, and register the versioned binaries as
alternatives so that CC/CXX=clang/clang++ and tools like llvm-cov and
llvm-objcopy use the newer release automatically.
2026-06-22 12:26:19 -04:00
andrew d70e6a2455 Merge pull request 'Set restype=None for void-returning C functions' (#53) from weaselbot/conflict-set:weaselbot/issue-46 into main
Reviewed-on: weaselab/conflict-set#53
2026-06-22 01:12:47 +00:00
andrew 8a5168f232 CMakeLists: only pass hardening-check arch flags the tool supports
The amd64 CI runner's hardening-check does not recognize
--nobranchprotection, causing the hardening_check test to fail at
configure time. Query the tool's help output and only include the
architecture-specific skip flags when they are advertised.
2026-06-21 21:07:26 -04:00
weaselbotandandrew 742d920aa1 Set restype=None for void-returning C functions in conflict_set.py
ConflictSet_check, ConflictSet_addWrites, ConflictSet_setOldestVersion,
and ConflictSet_destroy return void in C, but the Python wrapper left
their ctypes restype at the default c_int. Set restype = None for each
to match the C API contract and avoid undefined behavior from reading
the return register of void functions.
2026-06-21 19:25:27 -04:00
andrew 6d8b939a81 Replace CI Docker image with inline apt installs
Drops the build-image job and the private registry dependency entirely.
Each job now installs only the packages it needs, caching /var/cache/apt/archives
keyed on the workflow file and ~/.cache/pre-commit keyed on .pre-commit-config.yaml.
2026-06-21 19:21:04 -04:00
andrew 48f9ee46cf Pin CI steps that use bashisms to bash 2026-06-12 18:05:57 -04:00
andrew f13a30e8aa Install biber in the CI image so the paper's bibliography builds 2026-06-12 18:04:49 -04:00
andrew 1394a1a98f Print test summaries to the step log too
Gitea doesn't render GITHUB_STEP_SUMMARY yet
(https://github.com/go-gitea/gitea/issues/36098), so tee the summary
into the log where it's actually visible. Keep writing the summary
file so it lights up if Gitea ships the feature.
2026-06-12 17:21:49 -04:00
andrew c52339a2ba Add markdown test summaries and a coverage HTML report to CI
ctest_summary.py renders a Test.xml as markdown for
GITHUB_STEP_SUMMARY: a one-liner when everything passes, otherwise the
first few failures inline with a link to the full Test.xml on MinIO.
It's also usable locally with --all to list every failure from a
downloaded Test.xml.

The coverage job now also generates and uploads gcovr's html-details
report and links it from the step summary.
2026-06-12 17:06:31 -04:00
andrew 3a82d90914 Work around arm64 clang codegen issues breaking CI
Disable preserve_none under ASan on aarch64: every clang tested (20,
21, trunk 22) miscompiles the continuation chains with that
combination, crashing ~97% of fuzz corpus tests. The chains still work
with the default calling convention. See #38.

Mark the masked Node3 scan results defined for valgrind on aarch64:
clang 21+ lowers the in-bounds tests through flags+csel, which
memcheck models imprecisely, tainting bits the mask provably clears.
See #39.

Skip valgrind tests in the arm64 release job, since -DNVALGRIND
compiles out the client requests the workaround relies on.

Both issues track filing upstream bugs.
2026-06-12 16:35:57 -04:00
andrew 0921d8fedf Fix arch-specific hardening_check and guard CI package upload globs
hardening-check tests both CET (x86-only) and branch protection
(arm64-only), so pass the ignore flag for whichever doesn't apply to
the build arch.

The release upload step runs with if: always() so test results are
uploaded even when an earlier step fails, but in that case cpack never
ran and mc errored on the unmatched package globs. Skip missing
packages instead.
2026-06-12 14:45:23 -04:00
andrew f947d883e7 Only pass -pie when linking executables
Passing -pie globally made the driver link Scrt1.o into shared
libraries, which fails with an undefined reference to main.
2026-06-12 13:58:54 -04:00
andrew 19f430d68f Add nodejs to the CI image
The runner executes JavaScript actions (checkout, cache) with node from
inside the job container, so custom container images must provide it.
2026-06-12 12:33:48 -04:00
andrew 13e9e88e0e Pass registry credentials via env instead of script interpolation
Template interpolation embeds the secret in the generated script file;
env indirection keeps it out of argv and off disk.
2026-06-12 12:21:19 -04:00
andrew 12a62a91cf Track the CI image hash with a label instead of a tag
Pushing per-hash tags accumulates multi-GB versions in the registry,
and package cleanup rules can't distinguish the hash tags that the
latest-{arch} tags currently point to from stale ones. Push only
latest-{arch} and record the Dockerfile hash as an image label; the
skip-rebuild check reads the label back via buildx imagetools.
2026-06-12 12:16:34 -04:00
andrew cbbb23bf9d Shrink the CI image
Replace texlive-full with the texlive packages the paper actually needs
(latexmk, latex-extra, pictures for TikZ, bibtex-extra, recommended
fonts), and use Ubuntu's valgrind instead of building 3.22 from source
- the distro version is newer now. Cuts the image by several GB and
removes the slowest step of the image build.
2026-06-12 12:12:10 -04:00
andrew 04d02261e9 Install clang from Ubuntu and push image from buildx
apt.llvm.org has no repository for the current ubuntu:rolling release,
and nothing pins clang 20 - CC=clang already resolved to the distro
clang. Drop the llvm.sh step along with its helper packages.

docker build now runs under the buildx docker-container driver, which
keeps the result in the build cache unless told otherwise, so push
directly from the build instead of tagging locally.
2026-06-12 12:06:45 -04:00
andrew dd8f006d3f Remove docker socket debug step and unneeded sudo
Jobs run as root in the job container; the earlier permission errors
were SELinux denials on the mounted podman socket, fixed in the runner
config with --security-opt label=disable.
2026-06-12 11:58:30 -04:00
andrew 776d06963b Add temporary docker socket debug step 2026-06-12 11:52:51 -04:00
andrew e03afe0651 Run docker with sudo in build-image
The default job image runs as a non-root user that lacks access to the
mounted docker socket.
2026-06-12 11:30:28 -04:00
andrew 6eecf6e4ac Use a registry bot account for the container registry
Gitea's ephemeral Actions token is not accepted by the container
registry, so docker login and image pulls use REGISTRY_USER /
REGISTRY_TOKEN secrets (a dedicated low-privilege account and its
personal access token with package read/write scope) instead.
2026-06-12 11:16:33 -04:00
andrew d74a12b5a4 Convert Jenkins CI to Gitea Actions
Replace the Jenkinsfile with .gitea/workflows/ci.yml. The CI image is
built from the Dockerfile and pushed to the Gitea container registry,
rebuilt only when the Dockerfile or pre-commit config changes. The
aarch64 release build now runs natively on an arm64 runner instead of
cross-compiling, so the cross toolchain and qemu are dropped from the
image. Artifacts still go to MinIO (via mc, skipped gracefully if
credentials are not configured); ccache uses actions/cache instead of a
host volume.
2026-06-12 11:13:32 -04:00
andrew 755bbcbe56 Fix gcc 15 build 2025-11-06 13:12:10 -05:00
andrew 2642d453dc Remove unnecessary branch for interleaved range writes 2024-12-11 21:53:44 -08:00
14 changed files with 1186 additions and 1194 deletions
+306
View File
@@ -0,0 +1,306 @@
name: CI
on: [push, pull_request]
env:
CC: clang
CXX: clang++
jobs:
pre-commit:
runs-on: ubuntu-latest-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: /var/cache/apt/archives
key: apt-amd64-${{ hashFiles('.gitea/workflows/ci.yml') }}
- name: Install dependencies
run: |
. /etc/os-release
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
echo "deb http://apt.llvm.org/${VERSION_CODENAME}/ llvm-toolchain-${VERSION_CODENAME}-21 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo apt-get update -qq
sudo apt-get install -y \
clang-21 git nodejs pre-commit
for tool in clang clang++; do
sudo update-alternatives --install /usr/bin/${tool} ${tool} /usr/bin/${tool}-21 100
done
- uses: actions/cache@v4
with:
path: ~/.cache/pre-commit
key: pre-commit-${{ hashFiles('.pre-commit-config.yaml') }}
- name: Run pre-commit
run: |
git config --global --add safe.directory "$PWD"
pre-commit run --all-files --show-diff-on-failure
test:
strategy:
fail-fast: false
matrix:
include:
- name: 64-bit-versions
cmake_args: -DCMAKE_CXX_FLAGS=-DUSE_64_BIT=1
- name: debug
cmake_args: -DCMAKE_BUILD_TYPE=Debug
- name: simd-fallback
cmake_args: -DUSE_SIMD_FALLBACK=ON
- name: gcc
cmake_args: -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++
runs-on: ubuntu-latest-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: /var/cache/apt/archives
key: apt-amd64-${{ hashFiles('.gitea/workflows/ci.yml') }}
- name: Install dependencies
run: |
. /etc/os-release
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
echo "deb http://apt.llvm.org/${VERSION_CODENAME}/ llvm-toolchain-${VERSION_CODENAME}-21 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo apt-get update -qq
sudo apt-get install -y \
build-essential ccache clang-21 cmake gcc g++ \
libc6-dbg llvm-21 lld-21 mold ninja-build python3 valgrind zstd
sudo curl -Ls "https://dl.min.io/client/mc/release/linux-amd64/mc" \
-o /usr/local/bin/mc && sudo chmod +x /usr/local/bin/mc
for tool in clang clang++ llvm-ar llvm-nm llvm-ranlib llvm-objcopy llvm-cov llvm-symbolizer lld ld.lld; do
sudo update-alternatives --install /usr/bin/${tool} ${tool} /usr/bin/${tool}-21 100
done
- uses: actions/cache@v4
with:
path: .ccache
key: ccache-${{ matrix.name }}-${{ gitea.sha }}
restore-keys: |
ccache-${{ matrix.name }}-
- name: Build
run: |
export CCACHE_DIR="$GITHUB_WORKSPACE/.ccache"
rm -rf build
cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache ${{ matrix.cmake_args }}
ninja -C build
ccache -s
- name: Test
run: |
cd build
ctest --no-compress-output --test-output-size-passed 100000 --test-output-size-failed 100000 -T Test -j "$(nproc)" --timeout 90 > /dev/null
- name: Upload test results to MinIO
if: always()
env:
MINIO_ACCESS_KEY: ${{ secrets.MINIO_ACCESS_KEY }}
MC_HOST_minio: https://${{ secrets.MINIO_ACCESS_KEY }}:${{ secrets.MINIO_SECRET_KEY }}@minio.weaselab.dev
run: |
if [ -z "$MINIO_ACCESS_KEY" ]; then
echo "MinIO credentials not configured; skipping upload"
exit 0
fi
zstd build/Testing/*/Test.xml
mc cp build/Testing/*/Test.xml.zst "minio/jenkins/conflict-set/${{ gitea.run_number }}/${{ matrix.name }}/"
- name: Test summary
if: always()
run: |
python3 ctest_summary.py build/Testing/*/Test.xml \
--link "https://minio.weaselab.dev/jenkins/conflict-set/${{ gitea.run_number }}/${{ matrix.name }}/Test.xml.zst" \
| tee -a "$GITHUB_STEP_SUMMARY"
release:
strategy:
fail-fast: false
matrix:
include:
- runner: ubuntu-latest-amd64
arch: amd64
- runner: ubuntu-latest-arm64
arch: arm64
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: /var/cache/apt/archives
key: apt-${{ matrix.arch }}-${{ hashFiles('.gitea/workflows/ci.yml') }}
- name: Install dependencies
run: |
. /etc/os-release
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
echo "deb http://apt.llvm.org/${VERSION_CODENAME}/ llvm-toolchain-${VERSION_CODENAME}-21 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo apt-get update -qq
sudo apt-get install -y \
biber build-essential ccache clang-21 cmake devscripts \
latexmk libc6-dbg llvm-21 lld-21 mold ninja-build rpm \
texlive-bibtex-extra texlive-fonts-recommended \
texlive-latex-extra texlive-pictures valgrind zstd
sudo curl -Ls "https://dl.min.io/client/mc/release/linux-$(dpkg --print-architecture)/mc" \
-o /usr/local/bin/mc && sudo chmod +x /usr/local/bin/mc
for tool in clang clang++ llvm-ar llvm-nm llvm-ranlib llvm-objcopy llvm-cov llvm-symbolizer lld ld.lld; do
sudo update-alternatives --install /usr/bin/${tool} ${tool} /usr/bin/${tool}-21 100
done
- uses: actions/cache@v4
with:
path: .ccache
key: ccache-release-${{ matrix.arch }}-${{ gitea.sha }}
restore-keys: |
ccache-release-${{ matrix.arch }}-
- name: Build
run: |
export CCACHE_DIR="$GITHUB_WORKSPACE/.ccache"
rm -rf build
cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_FLAGS=-DNVALGRIND
ninja -C build
ccache -s
- name: Test
run: |
cd build
# On arm64, valgrind needs the MAKE_MEM_DEFINED client requests for
# https://git.weaselab.dev/weaselab/conflict-set/issues/39, but this
# build has -DNVALGRIND, which compiles them out. Skip valgrind
# tests here; they run annotated in the test job.
ctest --no-compress-output --test-output-size-passed 100000 --test-output-size-failed 100000 ${{ matrix.arch == 'arm64' && '-E valgrind' || '' }} -T Test -j "$(nproc)" --timeout 90 > /dev/null
- name: Package
run: |
cd build
cpack -G DEB
cpack -G RPM
- name: Build paper
if: matrix.arch == 'amd64'
run: |
cd paper
make
- name: Upload artifacts to MinIO
if: always()
shell: bash
env:
MINIO_ACCESS_KEY: ${{ secrets.MINIO_ACCESS_KEY }}
MC_HOST_minio: https://${{ secrets.MINIO_ACCESS_KEY }}:${{ secrets.MINIO_SECRET_KEY }}@minio.weaselab.dev
run: |
if [ -z "$MINIO_ACCESS_KEY" ]; then
echo "MinIO credentials not configured; skipping upload"
exit 0
fi
dest="minio/jenkins/conflict-set/${{ gitea.run_number }}/release-${{ matrix.arch }}/"
zstd build/Testing/*/Test.xml
mc cp build/Testing/*/Test.xml.zst "$dest"
if compgen -G "build/*.deb" > /dev/null; then
mc cp build/*.deb "$dest"
fi
if compgen -G "build/*.rpm" > /dev/null; then
mc cp build/*.rpm "$dest"
fi
if compgen -G "paper/*.pdf" > /dev/null; then
mc cp paper/*.pdf "$dest"
fi
- name: Test summary
if: always()
run: |
python3 ctest_summary.py build/Testing/*/Test.xml \
--link "https://minio.weaselab.dev/jenkins/conflict-set/${{ gitea.run_number }}/release-${{ matrix.arch }}/Test.xml.zst" \
| tee -a "$GITHUB_STEP_SUMMARY"
coverage:
runs-on: ubuntu-latest-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: /var/cache/apt/archives
key: apt-amd64-${{ hashFiles('.gitea/workflows/ci.yml') }}
- name: Install dependencies
run: |
. /etc/os-release
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
echo "deb http://apt.llvm.org/${VERSION_CODENAME}/ llvm-toolchain-${VERSION_CODENAME}-21 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo apt-get update -qq
sudo apt-get install -y \
build-essential ccache clang-21 cmake gcovr \
libc6-dbg llvm-21 lld-21 mold ninja-build python3 valgrind zstd
sudo curl -Ls "https://dl.min.io/client/mc/release/linux-amd64/mc" \
-o /usr/local/bin/mc && sudo chmod +x /usr/local/bin/mc
for tool in clang clang++ llvm-ar llvm-nm llvm-ranlib llvm-objcopy llvm-cov llvm-symbolizer lld ld.lld; do
sudo update-alternatives --install /usr/bin/${tool} ${tool} /usr/bin/${tool}-21 100
done
- uses: actions/cache@v4
with:
path: .ccache
key: ccache-coverage-${{ gitea.sha }}
restore-keys: |
ccache-coverage-
- name: Build
run: |
export CCACHE_DIR="$GITHUB_WORKSPACE/.ccache"
rm -rf build
cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_C_FLAGS=--coverage -DCMAKE_CXX_FLAGS=--coverage \
-DCMAKE_BUILD_TYPE=Debug -DDISABLE_TSAN=ON
ninja -C build
ccache -s
- name: Test
run: |
cd build
ctest --no-compress-output --test-output-size-passed 100000 --test-output-size-failed 100000 -T Test -j "$(nproc)" --timeout 90 > /dev/null
- name: Coverage report
shell: bash
run: |
gcov_args=(-f ConflictSet.cpp -f LongestCommonPrefix.h -f Metrics.h
--gcov-executable "llvm-cov gcov" --exclude-noncode-lines)
gcovr "${gcov_args[@]}" --cobertura > build/coverage.xml
gcovr "${gcov_args[@]}"
mkdir -p build/coverage_html
gcovr "${gcov_args[@]}" --html-details build/coverage_html/index.html
gcovr "${gcov_args[@]}" --fail-under-line 100 > /dev/null
- name: Upload results to MinIO
if: always()
env:
MINIO_ACCESS_KEY: ${{ secrets.MINIO_ACCESS_KEY }}
MC_HOST_minio: https://${{ secrets.MINIO_ACCESS_KEY }}:${{ secrets.MINIO_SECRET_KEY }}@minio.weaselab.dev
run: |
if [ -z "$MINIO_ACCESS_KEY" ]; then
echo "MinIO credentials not configured; skipping upload"
exit 0
fi
dest="minio/jenkins/conflict-set/${{ gitea.run_number }}/coverage/"
zstd build/Testing/*/Test.xml
mc cp build/Testing/*/Test.xml.zst "$dest"
if [ -e build/coverage.xml ]; then
mc cp build/coverage.xml "$dest"
fi
if [ -d build/coverage_html ]; then
mc cp -r build/coverage_html "$dest"
fi
- name: Test summary
if: always()
run: |
python3 ctest_summary.py build/Testing/*/Test.xml \
--link "https://minio.weaselab.dev/jenkins/conflict-set/${{ gitea.run_number }}/coverage/Test.xml.zst" \
| tee -a "$GITHUB_STEP_SUMMARY"
echo "" | tee -a "$GITHUB_STEP_SUMMARY"
echo "📊 [Coverage report](https://minio.weaselab.dev/jenkins/conflict-set/${{ gitea.run_number }}/coverage/coverage_html/index.html)" | tee -a "$GITHUB_STEP_SUMMARY"
+5 -4
View File
@@ -37,15 +37,16 @@ ConflictSet::ReadRange singleton(Arena &arena, TrivialSpan key) {
}
ConflictSet::ReadRange prefixRange(Arena &arena, TrivialSpan key) {
int index;
for (index = key.size() - 1; index >= 0; index--)
if ((key[index]) != 255)
int index = key.size() - 1;
for (; index >= 0; index--)
if (key[index] != 255)
break;
// Must not be called with a string that consists only of zero or more '\xff'
// bytes.
// bytes, or with an empty string (which has no finite upper bound).
if (index < 0) {
assert(false);
std::abort();
}
uint8_t *buf = new (arena) uint8_t[index + 1];
+39 -7
View File
@@ -32,8 +32,12 @@ if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
endif()
add_compile_options(
# -Werror=switch-enum
-Wswitch-enum -Wunused-variable -fPIC -fdata-sections -ffunction-sections
-Werror=switch-enum
-Wswitch-enum
-Wunused-variable
-fPIC
-fdata-sections
-ffunction-sections
-fno-jump-tables # https://github.com/llvm/llvm-project/issues/54247
)
@@ -49,6 +53,7 @@ endif()
if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
add_compile_options("-Wno-maybe-uninitialized")
add_compile_options("-Wno-maybe-musttail-local-addr")
endif()
if(NOT APPLE)
@@ -56,12 +61,16 @@ if(NOT APPLE)
add_compile_options(-g -fno-omit-frame-pointer)
endif()
set(full_relro_flags "-pie;LINKER:-z,relro,-z,now,-z,noexecstack")
set(relro_flags "LINKER:-z,relro,-z,now,-z,noexecstack")
set(full_relro_flags "-pie;${relro_flags}")
cmake_push_check_state()
list(APPEND CMAKE_REQUIRED_LINK_OPTIONS ${full_relro_flags})
check_cxx_source_compiles("int main(){}" HAS_FULL_RELRO FAIL_REGEX "warning:")
if(HAS_FULL_RELRO)
add_link_options(${full_relro_flags})
# -pie only applies to executables; passing it when linking a shared library
# makes the driver pull in Scrt1.o, which requires main.
add_link_options("$<$<STREQUAL:$<TARGET_PROPERTY:TYPE>,EXECUTABLE>:-pie>"
${relro_flags})
endif()
cmake_pop_check_state()
@@ -130,6 +139,7 @@ add_custom_command(
COMMAND_EXPAND_LISTS)
add_library(${PROJECT_NAME} SHARED ${CMAKE_BINARY_DIR}/${PROJECT_NAME}.o)
add_dependencies(${PROJECT_NAME} ${PROJECT_NAME}-object)
set_target_properties(
${PROJECT_NAME} PROPERTIES LIBRARY_OUTPUT_DIRECTORY
"${CMAKE_CURRENT_BINARY_DIR}/radix_tree")
@@ -146,6 +156,7 @@ if(HAS_VERSION_SCRIPT)
endif()
add_library(${PROJECT_NAME}-static STATIC ${CMAKE_BINARY_DIR}/${PROJECT_NAME}.o)
add_dependencies(${PROJECT_NAME}-static ${PROJECT_NAME}-object)
if(CMAKE_BUILD_TYPE STREQUAL Debug)
set_target_properties(${PROJECT_NAME}-static PROPERTIES LINKER_LANGUAGE CXX)
else()
@@ -374,9 +385,30 @@ if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR AND BUILD_TESTING)
if(NOT CMAKE_CROSSCOMPILING)
find_program(HARDENING_CHECK hardening-check)
if(HARDENING_CHECK)
add_test(NAME hardening_check
COMMAND ${HARDENING_CHECK} $<TARGET_FILE:${PROJECT_NAME}>
--nofortify --nostackprotector)
# Not all versions of hardening-check support the same options, so query
# the help output before using architecture-specific skips.
execute_process(
COMMAND ${HARDENING_CHECK} --help
OUTPUT_VARIABLE hardening_check_help
ERROR_VARIABLE hardening_check_help
OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_STRIP_TRAILING_WHITESPACE)
set(hardening_check_arch_flags "")
# Control flow integrity (CET) is x86-only and branch protection (PAC/BTI)
# is arm64-only, so ignore whichever doesn't apply.
if(CMAKE_SYSTEM_PROCESSOR STREQUAL aarch64 OR CMAKE_SYSTEM_PROCESSOR
STREQUAL arm64)
if(hardening_check_help MATCHES "nocfprotection")
list(APPEND hardening_check_arch_flags --nocfprotection)
endif()
else()
if(hardening_check_help MATCHES "nobranchprotection")
list(APPEND hardening_check_arch_flags --nobranchprotection)
endif()
endif()
add_test(
NAME hardening_check
COMMAND ${HARDENING_CHECK} $<TARGET_FILE:${PROJECT_NAME}> --nofortify
--nostackprotector ${hardening_check_arch_flags})
endif()
endif()
+631 -983
View File
File diff suppressed because it is too large Load Diff
+13 -25
View File
@@ -8,48 +8,36 @@ RUN chmod -R 777 /tmp
RUN apt-get update
RUN apt-get upgrade -y
RUN TZ=America/Los_Angeles DEBIAN_FRONTEND=noninteractive apt-get install -y \
binutils-aarch64-linux-gnu \
biber \
build-essential \
ccache \
clang \
cmake \
curl \
devscripts \
g++-aarch64-linux-gnu \
gcovr \
git \
gnupg \
latexmk \
libc6-dbg \
lsb-release \
llvm \
mold \
ninja-build \
nodejs \
pre-commit \
python3-requests \
qemu-user \
rpm \
software-properties-common \
texlive-full \
texlive-bibtex-extra \
texlive-fonts-recommended \
texlive-latex-extra \
texlive-pictures \
valgrind \
wget \
zstd
# Install recent valgrind from source
RUN curl -Ls https://sourceware.org/pub/valgrind/valgrind-3.22.0.tar.bz2 -o valgrind.tar.bz2 && \
echo "c811db5add2c5f729944caf47c4e7a65dcaabb9461e472b578765dd7bf6d2d4c valgrind.tar.bz2" > valgrind-sha.txt && \
sha256sum --quiet -c valgrind-sha.txt && \
mkdir valgrind && \
tar --strip-components 1 --no-same-owner --no-same-permissions --directory valgrind -xjf valgrind.tar.bz2 && \
cd valgrind && \
./configure --enable-only64bit --enable-lto && \
make -j`nproc` && \
make install && \
cd .. && \
rm -rf /tmp/*
# MinIO client, for uploading build artifacts
RUN curl -Ls "https://dl.min.io/client/mc/release/linux-$(dpkg --print-architecture)/mc" \
-o /usr/local/bin/mc && chmod +x /usr/local/bin/mc
# Recent clang
RUN wget https://apt.llvm.org/llvm.sh && chmod +x ./llvm.sh && ./llvm.sh 20
RUN apt-get -y install clang llvm
# Set after building valgrind, which doesn't build with clang for some reason
ENV CC=clang
ENV CXX=clang++
+15 -4
View File
@@ -96,7 +96,9 @@ void ConflictSet::setOldestVersion(int64_t oldestVersion) {
return impl->setOldestVersion(oldestVersion);
}
int64_t ConflictSet::getBytes() const { return -1; }
// The hash_table implementation does not track memory usage, so return 0 to
// satisfy the API contract that getBytes() returns a non-negative value.
int64_t ConflictSet::getBytes() const { return 0; }
void ConflictSet::getMetricsV1(MetricsV1 **metrics, int *count) const {
*metrics = nullptr;
@@ -119,7 +121,13 @@ ConflictSet::ConflictSet(ConflictSet &&other) noexcept
: impl(std::exchange(other.impl, nullptr)) {}
ConflictSet &ConflictSet::operator=(ConflictSet &&other) noexcept {
impl = std::exchange(other.impl, nullptr);
if (this != &other) {
if (impl) {
impl->~Impl();
safe_free(impl, sizeof(Impl));
}
impl = std::exchange(other.impl, nullptr);
}
return *this;
}
@@ -155,7 +163,10 @@ __attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) {
}
__attribute__((__visibility__("default"))) int64_t
ConflictSet_getBytes(void *cs) {
using Impl = ConflictSet::Impl;
return -1;
(void)cs;
// The hash_table implementation does not track memory usage, so return 0 to
// satisfy the API contract that ConflictSet_getBytes returns a non-negative
// value.
return 0;
}
}
Vendored
-151
View File
@@ -1,151 +0,0 @@
def CleanBuildAndTest(String cmakeArgs) {
sh """
export CCACHE_DIR=/ccache
rm -rf build
mkdir build
cd build
cmake .. -G Ninja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache ${cmakeArgs}
ninja
ccache -s
"""
catchError {
sh '''
cd build
ctest --no-compress-output --test-output-size-passed 100000 --test-output-size-failed 100000 -T Test -j `nproc` --timeout 90 > /dev/null
zstd Testing/*/Test.xml
'''
}
xunit tools: [CTest(pattern: 'build/Testing/*/Test.xml')], skipPublishingChecks: false
minio bucket: 'jenkins', credentialsId: 'jenkins-minio', excludes: '', host: 'minio.weaselab.dev', includes: 'build/Testing/*/Test.xml.zst', targetFolder: '${JOB_NAME}/${BUILD_NUMBER}/${STAGE_NAME}/'
}
pipeline {
agent any
stages {
stage('Pre-commit') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
script {
env.HOME = env.WORKSPACE
}
sh 'pre-commit run --all-files --show-diff-on-failure'
}
}
stage('64 bit versions') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DCMAKE_CXX_FLAGS=-DUSE_64_BIT=1")
}
}
stage('Debug') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DCMAKE_BUILD_TYPE=Debug")
}
}
stage('SIMD fallback') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DUSE_SIMD_FALLBACK=ON")
}
}
stage('Release [clang]') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DCMAKE_CXX_FLAGS=-DNVALGRIND")
recordIssues(tools: [clang()])
sh '''
cd build
cpack -G DEB
cpack -G RPM
'''
sh '''
cd paper
make
'''
minio bucket: 'jenkins', credentialsId: 'jenkins-minio', excludes: '', host: 'minio.weaselab.dev', includes: 'build/*.deb,build/*.rpm,paper/*.pdf', targetFolder: '${JOB_NAME}/${BUILD_NUMBER}/${STAGE_NAME}/'
}
}
stage('gcc') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++")
recordIssues(tools: [gcc()])
}
}
stage('Release [clang,aarch64]') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DCMAKE_TOOLCHAIN_FILE=../aarch64-toolchain.cmake -DCMAKE_CXX_FLAGS=-DNVALGRIND")
sh '''
cd build
cpack -G DEB
cpack -G RPM
'''
minio bucket: 'jenkins', credentialsId: 'jenkins-minio', excludes: '', host: 'minio.weaselab.dev', includes: 'build/*.deb,build/*.rpm', targetFolder: '${JOB_NAME}/${BUILD_NUMBER}/${STAGE_NAME}'
}
}
stage('Coverage') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
script {
gcov_args = "-f ConflictSet.cpp -f LongestCommonPrefix.h -f Metrics.h --gcov-executable 'llvm-cov gcov' --exclude-noncode-lines"
}
CleanBuildAndTest("-DCMAKE_C_FLAGS=--coverage -DCMAKE_CXX_FLAGS=--coverage -DCMAKE_BUILD_TYPE=Debug -DDISABLE_TSAN=ON")
sh """
gcovr ${gcov_args} --cobertura > build/coverage.xml
"""
recordCoverage qualityGates: [[criticality: 'NOTE', metric: 'MODULE']], tools: [[parser: 'COBERTURA', pattern: 'build/coverage.xml']]
sh """
gcovr ${gcov_args}
gcovr ${gcov_args} --fail-under-line 100 > /dev/null
"""
}
}
}
post {
always {
emailext mimeType: 'text/html', body: '${SCRIPT, template="groovy-html.template"}', subject: "${env.JOB_NAME} - Build# ${env.BUILD_NUMBER} - ${currentBuild.currentResult}", to: 'andrew@weaselab.dev'
}
}
}
+7 -6
View File
@@ -1,5 +1,6 @@
#include <ConflictSet.h>
#include <algorithm>
#include <cerrno>
#include <chrono>
#include <cstdio>
@@ -77,10 +78,10 @@ int main(int argc, const char **argv) {
begin = end + 1;
end = (uint8_t *)memchr(begin, '\n', size);
if (line.size() > 0 && line[0] == 'P') {
write = line.subspan(2, line.size());
} else if (line.size() > 0 && line[0] == 'L') {
reads.push_back(line.subspan(2, line.size()));
if (line.size() >= 2 && line[0] == 'P') {
write = line.subspan(2, line.size() - 2);
} else if (line.size() >= 2 && line[0] == 'L') {
reads.push_back(line.subspan(2, line.size() - 2));
} else if (line.empty()) {
{
readRanges.resize(reads.size());
@@ -90,7 +91,7 @@ int main(int argc, const char **argv) {
iter->begin.len = read.size();
checkBytes += read.size();
iter->end.len = 0;
iter->readVersion = version - 100;
iter->readVersion = std::max<int64_t>(0, version - 100);
++iter;
}
}
@@ -121,7 +122,7 @@ int main(int argc, const char **argv) {
}
timer = now();
cs.setOldestVersion(version - 10000);
cs.setOldestVersion(std::max<int64_t>(0, version - 10000));
gcTime += now() - timer;
}
}
+1
View File
@@ -12,6 +12,7 @@
#include <sys/ioctl.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <thread>
+6 -1
View File
@@ -981,7 +981,12 @@ ConflictSet::ConflictSet(ConflictSet &&other) noexcept
: impl(std::exchange(other.impl, nullptr)) {}
ConflictSet &ConflictSet::operator=(ConflictSet &&other) noexcept {
impl = std::exchange(other.impl, nullptr);
if (this != &other) {
if (impl) {
internal_destroy(impl);
}
impl = std::exchange(other.impl, nullptr);
}
return *this;
}
+27 -9
View File
@@ -27,23 +27,37 @@ class Result(enum.Enum):
TOO_OLD = 2
def write(begin: bytes, end: Optional[bytes] = None) -> WriteRange:
b = (ctypes.c_ubyte * len(begin)).from_buffer(bytearray(begin))
def _make_key(buf: bytes) -> tuple[_Key, bytearray]:
"""Create a _Key and a backing bytearray that must be kept alive."""
backing = bytearray(buf)
array = (ctypes.c_ubyte * len(backing)).from_buffer(backing)
return _Key(array, len(array)), backing
def write(begin: bytes, end: Optional[bytes] = None) -> WriteRange:
begin_key, begin_buf = _make_key(begin)
if end is None:
e = (ctypes.c_ubyte * 0)()
end_key = _Key((ctypes.c_ubyte * 0)(), 0)
end_buf = None
else:
e = (ctypes.c_ubyte * len(end)).from_buffer(bytearray(end))
return WriteRange(_Key(b, len(b)), _Key(e, len(e)))
end_key, end_buf = _make_key(end)
result = WriteRange(begin_key, end_key)
result._begin_buf = begin_buf
result._end_buf = end_buf
return result
def read(version: int, begin: bytes, end: Optional[bytes] = None) -> ReadRange:
b = (ctypes.c_ubyte * len(begin)).from_buffer(bytearray(begin))
begin_key, begin_buf = _make_key(begin)
if end is None:
e = (ctypes.c_ubyte * 0)()
end_key = _Key((ctypes.c_ubyte * 0)(), 0)
end_buf = None
else:
e = (ctypes.c_ubyte * len(end)).from_buffer(bytearray(end))
return ReadRange(_Key(b, len(b)), _Key(e, len(e)), version)
end_key, end_buf = _make_key(end)
result = ReadRange(begin_key, end_key, version)
result._begin_buf = begin_buf
result._end_buf = end_buf
return result
class ConflictSet:
@@ -88,6 +102,7 @@ class ConflictSet:
ctypes.POINTER(ctypes.c_int),
ctypes.c_int,
)
self._lib.ConflictSet_check.restype = None
self._lib.ConflictSet_addWrites.argtypes = (
ctypes.c_void_p,
@@ -95,13 +110,16 @@ class ConflictSet:
ctypes.c_int,
ctypes.c_int64,
)
self._lib.ConflictSet_addWrites.restype = None
self._lib.ConflictSet_setOldestVersion.argtypes = (
ctypes.c_void_p,
ctypes.c_int64,
)
self._lib.ConflictSet_setOldestVersion.restype = None
self._lib.ConflictSet_destroy.argtypes = (ctypes.c_void_p,)
self._lib.ConflictSet_destroy.restype = None
self._lib.ConflictSet_getBytes.argtypes = (ctypes.c_void_p,)
self._lib.ConflictSet_getBytes.restype = ctypes.c_int64
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Summarize a CTest Test.xml as markdown.
Intended for $GITHUB_STEP_SUMMARY in CI, where only the first few failures
are shown inline (pass --link to point at the full Test.xml). Also reusable
locally to print every failure from a downloaded Test.xml with --all.
"""
import argparse
import base64
import gzip
import xml.etree.ElementTree as ET
# Failure output is truncated to this many trailing characters, which is
# usually enough to include e.g. an ASan report's summary.
OUTPUT_TAIL_CHARS = 3000
def test_output(test):
value = test.find("./Results/Measurement/Value")
if value is None or value.text is None:
return ""
text = value.text
if value.get("encoding") == "base64":
raw = base64.b64decode(text)
if value.get("compression") == "gzip":
raw = gzip.decompress(raw)
text = raw.decode(errors="replace")
return text
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("test_xml", help="path to a ctest Testing/*/Test.xml")
parser.add_argument(
"--inline",
type=int,
default=5,
help="how many failures to show inline (default 5)",
)
parser.add_argument("--all", action="store_true", help="show every failure inline")
parser.add_argument(
"--link", help="URL of the full Test.xml, linked when failures are elided"
)
args = parser.parse_args()
testing = ET.parse(args.test_xml).getroot().find("Testing")
tests = testing.findall("Test")
failed = [t for t in tests if t.get("Status") == "failed"]
notrun = sum(1 for t in tests if t.get("Status") == "notrun")
if not failed:
print(f"✅ All {len(tests) - notrun} tests passed")
else:
print(f"{len(failed)} of {len(tests)} tests failed\n")
shown = failed if args.all else failed[: args.inline]
for test in shown:
name = test.findtext("Name")
output = test_output(test)[-OUTPUT_TAIL_CHARS:].strip()
print(f"<details><summary><code>{name}</code></summary>\n")
print("````")
print(output)
print("````")
print("</details>\n")
remaining = len(failed) - len(shown)
if remaining > 0:
more = f"… and {remaining} more"
if args.link:
more += f" — full list in [Test.xml]({args.link})"
print(more)
if notrun:
print(f"\n⚠️ {notrun} tests not run")
if __name__ == "__main__":
main()
+11 -2
View File
@@ -88,7 +88,8 @@ struct __attribute__((__visibility__("default"))) ConflictSet {
~ConflictSet();
/** Returns the total bytes in use by this ConflictSet */
/** Returns the total bytes in use by this ConflictSet. Implementations that
* do not track memory usage return 0. */
int64_t getBytes() const;
/** Experimental! */
@@ -132,6 +133,13 @@ struct __attribute__((__visibility__("default"))) ConflictSet {
private:
Impl *impl;
#if __cplusplus <= 199711L
/* Declared private and left undefined to prevent copying in C++98/C++03.
The compiler would otherwise implicitly generate public copy operations,
which share the opaque Impl* and cause a double-free. */
ConflictSet(const ConflictSet &);
ConflictSet &operator=(const ConflictSet &);
#endif
};
} /* namespace weaselab */
@@ -211,7 +219,8 @@ ConflictSet *ConflictSet_create(int64_t oldestVersion);
void ConflictSet_destroy(ConflictSet *cs);
/** Returns the total bytes in use by this ConflictSet */
/** Returns the total bytes in use by this ConflictSet. Implementations that
* do not track memory usage return 0. */
int64_t ConflictSet_getBytes(const ConflictSet *cs);
#endif
+49 -2
View File
@@ -57,6 +57,53 @@ def test_conflict_set():
assert cs.check(read(0, key), read(1, key)) == [Result.TOO_OLD, Result.COMMIT]
def test_hash_table_getBytes():
# Regression test for issue #62: the hash_table implementation is
# point-query only and does not track memory usage, but getBytes() must
# still return a non-negative value rather than -1.
with ConflictSet(0, build_dir=build_dir, implementation="hash_table") as cs:
assert cs.getBytes() == 0
cs.addWrites(1, write(b"key"))
assert cs.getBytes() >= 0
assert cs.check(read(0, b"key")) == [Result.CONFLICT]
def test_write_read_without_outer_reference():
# Regression test for issue #42: WriteRange/ReadRange must keep their
# backing key buffers alive, because the C library reads the pointer
# stored in _Key while addWrites/check run.
with DebugConflictSet() as cs:
# The bytes literal is not referenced after this expression.
cs.addWrites(1, write(b"key"))
assert cs.check(read(0, b"key")) == [Result.CONFLICT]
cs.addWrites(2, write(b"a", b"z"))
assert cs.check(read(1, b"a", b"z")) == [Result.CONFLICT]
assert cs.check(read(1, b"b")) == [Result.CONFLICT]
assert cs.check(read(1, b"0")) == [Result.COMMIT]
def test_range_keeps_key_buffers_alive():
# Verify the fix for issue #42: returned range objects must retain a
# reference to the backing bytearray so the C pointer stays valid after
# the helper returns.
w = write(b"key")
assert w._begin_buf == bytearray(b"key")
assert w._end_buf is None
w2 = write(b"a", b"z")
assert w2._begin_buf == bytearray(b"a")
assert w2._end_buf == bytearray(b"z")
r = read(0, b"key")
assert r._begin_buf == bytearray(b"key")
assert r._end_buf is None
r2 = read(1, b"a", b"z")
assert r2._begin_buf == bytearray(b"a")
assert r2._end_buf == bytearray(b"z")
def test_update_zero_should_commit():
with DebugConflictSet() as cs1:
with DebugConflictSet() as cs2:
@@ -68,7 +115,7 @@ def test_update_zero_should_commit():
for i in range(256 - 17, 256):
cs2.addWrites(int(1), write(bytes([i])))
# Scan until first point write
cs2.check(read(0, b"\x00", bytes([256 - 17])))
assert cs2.check(read(0, b"\x00", bytes([256 - 17]))) == [Result.COMMIT]
def test_update_zero_should_conflict():
@@ -81,7 +128,7 @@ def test_update_zero_should_conflict():
# "zero" is now 2**31 + 100
cs1.addWrites(2**32 + 101, write(b"", b"\x02"), write(b"\x01"))
# rangeVersion of \x01 is now 2**31 + 100 ("max" of (2**31 + 100, 2**32 + 101))
cs1.check(read(2**32 + 1, b"\x00"))
assert cs1.check(read(2**32 + 1, b"\x00")) == [Result.CONFLICT]
# but 2**32 + 1 ">" 2**31 + 100 , and it incorrectly commits