16 Commits

Author SHA1 Message Date
b7e16b31ff Fill out empty subsections
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-16 12:57:57 -07:00
a324d31518 Second pass at "Checking range reads" 2024-04-16 12:14:04 -07:00
fdb05e0e33 First draft of "Checking range reads" subsection
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-15 17:32:33 -07:00
7c27d4a972 Always target macos 11.0 2024-04-09 14:08:58 -07:00
738de01cb4 Add getBytes to conflict_set.py
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 17:33:51 -07:00
325cab6a95 Target earliest convenient macos version
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 17:16:01 -07:00
0b2821941a Bump version
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 15:52:06 -07:00
a40b5dcd74 Add script to build .pkg file for macos
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 15:21:40 -07:00
193b1926ff Fix python type annotation
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-07 22:28:40 -07:00
1c900c5a8c Use PRE_LINK
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
According to
https://cmake.org/cmake/help/latest/command/add_custom_command.html this
more accurately matches the intent
2024-04-04 17:03:57 -07:00
90fdcdd51a Don't update mtime in privatize_symbols_macos.sh
It confuses ninja and we see things like the following:

ninja explain: stored deps info out of date for 'CMakeFiles/conflict-set-object.dir/ConflictSet.cpp.o' (1712275080328387291 vs 1712275080349012981)
2024-04-04 17:02:51 -07:00
eb3f6823eb Remove prettier
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
It caused a weird jenkins error trying to look at /etc/passwd. Not worth
the trouble for formatting yaml and md, which I'm barely using.
2024-04-04 16:29:26 -07:00
1534e10b75 Fix macos shared library name in conflict_set.py
Some checks failed
weaselab/conflict-set/pipeline/head There was a failure building this commit
2024-04-04 16:17:06 -07:00
3c100ccee8 Add prettier and black as pre-commit hooks
Some checks failed
weaselab/conflict-set/pipeline/head There was a failure building this commit
2024-04-04 15:59:27 -07:00
5cf45d1c35 Preliminary support for writing tests in python 2024-04-04 15:58:57 -07:00
4f97932893 Bump version
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-04 12:02:01 -07:00
11 changed files with 337 additions and 33 deletions

1
.gitignore vendored
View File

@@ -1,2 +1,3 @@
.cache
__pycache__
build

View File

@@ -1,11 +1,11 @@
repos:
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: 6d365699efc33b1b432eab5b4ae331a19e1857de # frozen: v18.1.2
rev: 6d365699efc33b1b432eab5b4ae331a19e1857de # frozen: v18.1.2
hooks:
- id: clang-format
exclude: ".*third_party/.*"
- repo: https://github.com/cheshirekow/cmake-format-precommit
rev: e2c2116d86a80e72e7146a06e68b7c228afc6319 # frozen: v0.6.13
rev: e2c2116d86a80e72e7146a06e68b7c228afc6319 # frozen: v0.6.13
hooks:
- id: cmake-format
- repo: local
@@ -13,7 +13,7 @@ repos:
- id: debug verbose check
name: disallow checking in DEBUG_VERBOSE=1
description: disallow checking in DEBUG_VERBOSE=1
entry: '^#define DEBUG_VERBOSE 1$'
entry: "^#define DEBUG_VERBOSE 1$"
language: pygrep
types: [c++]
- repo: local
@@ -21,10 +21,14 @@ repos:
- id: debug verbose check
name: disallow checking in SHOW_MEMORY=1
description: disallow checking in SHOW_MEMORY=1
entry: '^#define SHOW_MEMORY 1$'
entry: "^#define SHOW_MEMORY 1$"
language: pygrep
types: [c++]
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: a23f6b85d0fdd5bb9d564e2579e678033debbdff # frozen: v0.10.0.1
rev: a23f6b85d0fdd5bb9d564e2579e678033debbdff # frozen: v0.10.0.1
hooks:
- id: shellcheck
- id: shellcheck
- repo: https://github.com/psf/black
rev: 552baf822992936134cbd31a38f69c8cfe7c0f05 # frozen: 24.3.0
hooks:
- id: black

View File

@@ -1,13 +1,15 @@
cmake_minimum_required(VERSION 3.18)
project(
conflict-set
VERSION 0.0.2
VERSION 0.0.4
DESCRIPTION
"A data structure for optimistic concurrency control on ranges of bitwise-lexicographically-ordered keys."
HOMEPAGE_URL "https://git.weaselab.dev/weaselab/conflict-set"
LANGUAGES C CXX)
set(CMAKE_CXX_STANDARD 20)
file(WRITE ${CMAKE_BINARY_DIR}/version.txt ${PROJECT_VERSION})
include(CMakePushCheckState)
include(CheckCXXCompilerFlag)
include(CheckIncludeFileCXX)
@@ -128,7 +130,7 @@ endif()
if(APPLE)
add_custom_command(
TARGET ${PROJECT_NAME}-static
PRE_BUILD
PRE_LINK
COMMAND ${CMAKE_SOURCE_DIR}/privatize_symbols_macos.sh
$<TARGET_OBJECTS:${PROJECT_NAME}-object>)
else()
@@ -333,6 +335,7 @@ endif()
# packaging
set(CPACK_PACKAGE_CONTACT andrew@weaselab.dev)
set(CMAKE_INSTALL_DEFAULT_COMPONENT_NAME all)
set(CPACK_PACKAGE_VENDOR "Weaselab")
set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE")
@@ -353,6 +356,20 @@ else()
set(CPACK_DEBIAN_PACKAGE_DEPENDS "libc6 (>= 2.14)")
endif()
# macos
set(CMAKE_OSX_DEPLOYMENT_TARGET 11.0)
if(APPLE)
find_program(PANDOC_EXE pandoc)
if(PANDOC_EXE)
execute_process(COMMAND ${PANDOC_EXE} ${CMAKE_SOURCE_DIR}/README.md -o
${CMAKE_BINARY_DIR}/README.txt)
set(CPACK_RESOURCE_FILE_README ${CMAKE_BINARY_DIR}/README.txt)
endif()
configure_file(${CMAKE_SOURCE_DIR}/LICENSE ${CMAKE_BINARY_DIR}/LICENSE.txt
COPYONLY)
set(CPACK_RESOURCE_FILE_LICENSE ${CMAKE_BINARY_DIR}/LICENSE.txt)
endif()
include(CPack)
include(GNUInstallDirs)
@@ -377,8 +394,11 @@ install(
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
install(DIRECTORY include/
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME})
install(EXPORT ${PROJECT_NAME}Config
DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/${PROJECT_NAME}/cmake)
cpack_add_component(all)

View File

@@ -58,27 +58,27 @@ Performance counters:
## Skip list
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 246.99 | 4,048,700.59 | 0.2% | 0.01 | `point reads`
| 260.16 | 3,843,784.65 | 0.1% | 0.01 | `prefix reads`
| 493.35 | 2,026,953.19 | 0.1% | 0.01 | `range reads`
| 462.05 | 2,164,289.23 | 0.6% | 0.01 | `point writes`
| 448.19 | 2,231,205.25 | 0.9% | 0.01 | `prefix writes`
| 255.83 | 3,908,845.72 | 1.5% | 0.02 | `range writes`
| 582.63 | 1,716,349.02 | 1.3% | 0.01 | `monotonic increasing point writes`
| ns/op | op/s | err% | total | benchmark |
| -----: | -----------: | ---: | ----: | :---------------------------------- |
| 246.99 | 4,048,700.59 | 0.2% | 0.01 | `point reads` |
| 260.16 | 3,843,784.65 | 0.1% | 0.01 | `prefix reads` |
| 493.35 | 2,026,953.19 | 0.1% | 0.01 | `range reads` |
| 462.05 | 2,164,289.23 | 0.6% | 0.01 | `point writes` |
| 448.19 | 2,231,205.25 | 0.9% | 0.01 | `prefix writes` |
| 255.83 | 3,908,845.72 | 1.5% | 0.02 | `range writes` |
| 582.63 | 1,716,349.02 | 1.3% | 0.01 | `monotonic increasing point writes` |
## Radix tree (this implementation)
| ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:----------
| 19.42 | 51,483,206.67 | 0.3% | 0.01 | `point reads`
| 58.43 | 17,115,612.57 | 0.1% | 0.01 | `prefix reads`
| 216.09 | 4,627,766.60 | 0.2% | 0.01 | `range reads`
| 28.35 | 35,267,567.72 | 0.2% | 0.01 | `point writes`
| 43.43 | 23,026,226.17 | 0.2% | 0.01 | `prefix writes`
| 50.00 | 20,000,000.00 | 0.0% | 0.01 | `range writes`
| 92.38 | 10,824,863.69 | 4.1% | 0.01 | `monotonic increasing point writes`
| ns/op | op/s | err% | total | benchmark |
| -----: | ------------: | ---: | ----: | :---------------------------------- |
| 19.42 | 51,483,206.67 | 0.3% | 0.01 | `point reads` |
| 58.43 | 17,115,612.57 | 0.1% | 0.01 | `prefix reads` |
| 216.09 | 4,627,766.60 | 0.2% | 0.01 | `range reads` |
| 28.35 | 35,267,567.72 | 0.2% | 0.01 | `point writes` |
| 43.43 | 23,026,226.17 | 0.2% | 0.01 | `prefix writes` |
| 50.00 | 20,000,000.00 | 0.0% | 0.01 | `range writes` |
| 92.38 | 10,824,863.69 | 4.1% | 0.01 | `monotonic increasing point writes` |
# "Real data" test

View File

@@ -5,3 +5,4 @@ _free
_malloc
_memcpy
_memmove
dyld_stub_binder

126
conflict_set.py Normal file
View File

@@ -0,0 +1,126 @@
import ctypes
import enum
import os
from typing import Optional
_lib = None
for f in (
os.path.dirname(__file__) + "/build/radix_tree/libconflict-set.so.0",
os.path.dirname(__file__) + "/build/radix_tree/libconflict-set.0.dylib",
):
try:
_lib = ctypes.cdll.LoadLibrary(f)
except:
pass
if _lib is None:
import sys
print("Could not find libconflict-set", file=sys.stderr)
sys.exit(1)
class _Key(ctypes.Structure):
_fields_ = [("p", ctypes.POINTER(ctypes.c_ubyte)), ("len", ctypes.c_int)]
class ReadRange(ctypes.Structure):
_fields_ = [
("begin", _Key),
("end", _Key),
("readVersion", ctypes.c_int64),
]
class WriteRange(ctypes.Structure):
_fields_ = [("begin", _Key), ("end", _Key)]
_lib.ConflictSet_create.argtypes = (ctypes.c_int64,)
_lib.ConflictSet_create.restype = ctypes.c_void_p
_lib.ConflictSet_check.argtypes = (
ctypes.c_void_p,
ctypes.POINTER(ReadRange),
ctypes.POINTER(ctypes.c_int),
ctypes.c_int,
)
_lib.ConflictSet_addWrites.argtypes = (
ctypes.c_void_p,
ctypes.POINTER(WriteRange),
ctypes.c_int,
ctypes.c_int64,
)
_lib.ConflictSet_setOldestVersion.argtypes = (ctypes.c_void_p, ctypes.c_int64)
_lib.ConflictSet_destroy.argtypes = (ctypes.c_void_p,)
_lib.ConflictSet_getBytes.argtypes = (ctypes.c_void_p,)
_lib.ConflictSet_getBytes.restype = ctypes.c_int64
class Result(enum.Enum):
COMMIT = 0
CONFLICT = 1
TOO_OLD = 2
def write(begin: bytes, end: Optional[bytes] = None) -> WriteRange:
b = (ctypes.c_ubyte * len(begin))()
b.value = begin
if end is None:
e = (ctypes.c_ubyte * 0)()
e.value = b""
else:
e = (ctypes.c_ubyte * len(end))()
e.value = end
return WriteRange(_Key(b, len(b)), _Key(e, len(e)))
def read(version: int, begin: bytes, end: Optional[bytes] = None) -> ReadRange:
b = (ctypes.c_ubyte * len(begin))()
b.value = begin
if end is None:
e = (ctypes.c_ubyte * 0)()
e.value = b""
else:
e = (ctypes.c_ubyte * len(end))()
e.value = end
return ReadRange(_Key(b, len(b)), _Key(e, len(e)), version)
class ConflictSet:
def __init__(self, version: int = 0) -> None:
self.p = _lib.ConflictSet_create(version)
def addWrites(self, version: int, *writes: WriteRange):
_lib.ConflictSet_addWrites(
self.p, (WriteRange * len(writes))(*writes), len(writes), version
)
def check(self, *reads: ReadRange) -> list[Result]:
r = (ctypes.c_int * len(reads))()
_lib.ConflictSet_check(self.p, *reads, r, 1)
return [Result(x) for x in r]
def setOldestVersion(self, version: int) -> None:
_lib.ConflictSet_setOldestVersion(self.p, version)
def getBytes(self) -> int:
return _lib.ConflictSet_getBytes(self.p)
def __enter__(self):
return self
def close(self) -> None:
if self.p is not None:
_lib.ConflictSet_destroy(self.p)
self.p = None
def __exit__(self, exception_type, exception_value, exception_traceback):
if self.p is not None:
_lib.ConflictSet_destroy(self.p)
self.p = None

33
package_macos.sh Executable file
View File

@@ -0,0 +1,33 @@
#!/bin/bash
set -euxo pipefail
umask 022
SRC_DIR="${0%/*}"
BUILD_ARM="$(mktemp -d -t conflict-set-arm)"
BUILD_X86="$(mktemp -d -t conflict-set-x86)"
cmake_args=(-DCMAKE_CXX_FLAGS=-DNVALGRIND -DCPACK_PACKAGING_INSTALL_PREFIX=/usr/local)
cmake -S"$SRC_DIR" -B"$BUILD_ARM" -DCMAKE_OSX_ARCHITECTURES=arm64 "${cmake_args[@]}"
cmake --build "$BUILD_ARM" --target conflict-set --target conflict-set-static
cmake -S"$SRC_DIR" -B"$BUILD_X86" -DCMAKE_OSX_ARCHITECTURES=x86_64 "${cmake_args[@]}"
cmake --build "$BUILD_X86" --target conflict-set --target conflict-set-static
VERSION="$(cat "$BUILD_ARM/version.txt")"
lipo -create "$BUILD_ARM/radix_tree/libconflict-set.$VERSION.dylib" "$BUILD_X86/radix_tree/libconflict-set.$VERSION.dylib" -output "libconflict-set.$VERSION.dylib.tmp"
lipo -create "$BUILD_ARM"/libconflict-set-static.a "$BUILD_X86"/libconflict-set-static.a -output libconflict-set-static.a.tmp
mv "libconflict-set.$VERSION.dylib.tmp" "$BUILD_ARM/radix_tree/libconflict-set.$VERSION.dylib"
mv libconflict-set-static.a.tmp "$BUILD_ARM/libconflict-set-static.a"
pushd "$BUILD_ARM"
cpack -G productbuild
popd
mv "$BUILD_ARM/conflict-set-$VERSION-Darwin.pkg" .
rm -rf "$BUILD_ARM" "$BUILD_X86"

View File

@@ -155,3 +155,16 @@ keywords = {data structures, searching, trees}
year={1981},
publisher={ACM New York, NY, USA}
}
@article{Lemire_2018,
title={Roaring bitmaps: Implementation of an optimized software library},
volume={48},
ISSN={1097-024X},
url={http://dx.doi.org/10.1002/spe.2560},
DOI={10.1002/spe.2560},
number={4},
journal={Software: Practice and Experience},
publisher={Wiley},
author={Lemire, Daniel and Kaser, Owen and Kurz, Nathan and Deri, Luca and OHara, Chris and SaintJacques, François and SsiYanKai, Gregory},
year={2018},
month=jan, pages={867895} }

View File

@@ -6,6 +6,7 @@
\usepackage{tikz}
\usepackage{tikzscale}
\usepackage[edges]{forest}
\usepackage{amsmath}
\title{ARTful Conflict Checking for FoundationDB}
\author{Andrew Noyes \thanks{\href{mailto:andrew@weaselab.dev}{andrew@weaselab.dev}}}
@@ -20,7 +21,7 @@
\section*{Abstract}
FoundationDB \cite{DBLP:conf/sigmod/ZhouXSNMTABSLRD21} provides serializability using a specialized data structure called \textit{lastCommit} \footnote{See Algorithm 1 referenced in \cite{DBLP:conf/sigmod/ZhouXSNMTABSLRD21}} to implement optimistic concurrency control \cite{kung1981optimistic}.
FoundationDB \cite{DBLP:conf/sigmod/ZhouXSNMTABSLRD21} provides serializability using a specialized data structure called \textit{lastCommit} \footnote{See Algorithm 1 referenced in \cite{DBLP:conf/sigmod/ZhouXSNMTABSLRD21}.} to implement optimistic concurrency control \cite{kung1981optimistic}.
This data structure encodes the write sets for recent transactions as a map from key ranges (represented as bitwise-lexicographically-ordered half-open intervals) to most recent write versions.
FoundationDB implements \textit{lastCommit} as a version-augmented probabilistic skip list \cite{10.1145/78973.78977}.
In this paper, we propose an alternative implementation of \textit{lastCommit} as a version-augmented Adaptive Radix Tree (ART) \cite{DBLP:conf/icde/LeisK013}, and evaluate its performance.
@@ -68,9 +69,9 @@ See figure \ref{fig:tree} for an example tree after inserting
$\{ANY\} \rightarrow 2$,
$\{ARE\} \rightarrow 3$, and
$\{ART\} \rightarrow 4$.
Each node shows its partial prefix annotated with $(max,point,range)$.
Each node shows its partial prefix annotated with $max$ or $max,point,range$.
\subsection{Checking point reads}
\subsection{Checking point reads} \label{Checking point reads}
The algorithm for checking point reads follows directly from the definitions of the \emph{point} and \emph{range} fields.
Our input is a key $k$ and a read version $r$, and we must report whether or not the write version $v_{k}$ of $k$ is less than or equal to $r$.
@@ -84,10 +85,101 @@ As an optimization, during the search phase for a point read we can inspect the
If the max version among all keys starting with a prefix of $k$ is less than or equal to $r$, then $v_{k} \leq r$.
\subsection{Checking range reads}
\subsection{Adding point writes}
\subsection{Adding range writes}
Checking range reads is more involved. Logically the idea is to partition the range read so that each subrange in the partition is a single point or coincides with the set of keys beginning with a prefix (a \emph{prefix range}).
The max version of a single point is $v$ as described in \ref{Checking point reads}.
The max version of a prefix range is the $max$ of the node associated with the prefix if such a node exists, and $range$ of the next node with a $range$ field otherwise.
If there is no next node with a range field, then we ignore that subrange in our max version calculation.
The max version among all versions and max versions of subranges in this partition is the max version of the whole range, which we compare to $r$.
Let's start with partitioning the range in the case where the beginning of the range is a prefix of the end of the range.
We'll be able to use this as a subroutine in the general case.
Suppose our range is $[a_{0}\dots a_{k}, a_{0}\dots a_{n})$ where $k < n$, and $a_{i} \in [0, 256)$.
The partition starts with the singleton set containing the first key in the range.
\[
\{a_{0}\dots a_{k}\}
\]
or equivalently
\[
[a_{0}\dots a_{k}, a_{0}\dots a_{k} 0)
\]
and continues with a sequence of prefix ranges ending in each digit up until $a_{k+1}$.
Recall that the range $[a_{0}\dots a_{k} 0, a_{0}\dots a_{k} 1)$ is equivalent to the set of keys starting with $a_{0}\dots a_{k} 0$.
\begin{align*}
\dots \quad \cup \quad & [a_{0}\dots a_{k} 0, a_{0}\dots a_{k} 1) \quad \cup \\
& [a_{0}\dots a_{k} 1, a_{0}\dots a_{k} 2) \quad \cup \\
& \dots \\
& [a_{0}\dots a_{k} (a_{k+1}-1), a_{0}\dots a_{k+1})
\end{align*}
The remainder of the partition begins with the singleton set
\[
\dots \quad \cup \quad [a_{0}\dots a_{k + 1}, a_{0}\dots a_{k + 1} 0) \quad \cup\ \quad \dots
\]
and proceeds as above until a range ending at $a_{0}\dots a_{n}$.
Let's now consider a range where begin is not a prefix of end.
\[
[a_{0}\dots a_{m}, b_{0}\dots b_{n})
\]
Let $i$ be the lowest index such that $a_{i} \neq b_{i}$.
For brevity we will elide the common prefix up until $i$ in the following discussion.
We'll start with partitioning this range coarsely:
\begin{align*}
& [a_{i}\dots a_{m}, a_{i} + 1) \quad \cup \\
& [a_{i} + 1, a_{i} + 2) \quad \cup \\
& \dots \\
& [b_{i} - 1, b_{i}) \quad \cup \\
& [b_{i}, b_{i}\dots b_{n})
\end{align*}
The last range has a begin that's a prefix of end, and so we'll partition that as before.
The inner ranges are already prefix ranges.
This leaves only $[a_{i}\dots a_{m}, a_{i} + 1)$.
If $m = i$, then this range is adjacent to the first inner range above, and we're done.
Otherwise we'll partition this into
\begin{align*}
& [a_{i}\dots a_{m}, a_{i}\dots (a_{m} + 1)) \quad \cup \\
& [a_{i}\dots (a_{m} + 1), a_{i}\dots (a_{m} + 2)) \quad \cup \\
& \dots \\
& [a_{i}\dots 254, a_{i}\dots 255) \quad \cup \\
& [a_{i}\dots 255, a_{i}\dots (a_{m-1} + 1) )
\end{align*}
and repeat starting at \footnote{This doesn't explicitly describe how to handle the case where $a_{m-1} = 255$. In this case we would skip to the largest $j < m$ such that $a_{j} \neq 255$. We know $j \geq i$ since if $a_{i} = 255$ then the range is inverted.}
\[
\dots \quad \cup \quad [a_{i}\dots (a_{m-1} + 1), a_{i}\dots (a_{m-1} + 2))
\]
until we end at $a_{i} + 1$, adjacent to the first inner range.
A few notes on implementation:
\begin{itemize}
\item{For clarity, the above algorithm decouples the logical partitioning from the physical structure of the tree. An optimized implementation would merge adjacent prefix ranges that don't correspond to nodes in the tree as it scans, so that it only calculates the version of such merged ranges once. Additionally, our implementation stores an index of which child pointers are valid as a bitset for Node48 and Node256 to speed up this scan using techniques inspired by \cite{Lemire_2018}.}
\item{In order to avoid many costly pointer indirections, we can store the max version not in each node itself but next to each node's parent pointer. Without this, the range read performance is not competetive with the skip list.}
\item{An optimized implementation would construct the partition of $[a_{i}\dots a_{m}, a_{i} + 1)$ in reverse order, as it descends along the search path to $a_{i}\dots a_{m}$}
\item{An optimized implementation would search for the common prefix first, and return early if any prefix of the common prefix has a $max \leq r$.}
\end{itemize}
\subsection{Reclaiming old entries}
In order to bound memory usage, we track an \emph{oldest version}, reject transactions with read versions before \emph{oldest version}, and reclaim nodes made redundant by \emph{oldest version}.
We track the rate of insertions of new nodes and make sure that our incremental reclaiming of old nodes according to \emph{oldest version} outpaces inserts.
\subsection{Adding point writes}
A point write of $k$ at version $v$ simply sets $max \gets v$ \footnote{Recall that write versions are non-decreasing.} for every node along $k$'s search path, and sets $range$ for $k$'s node to the $range$ of the first node greater than $k$, or the \emph{oldest version} if none exists.
\subsection{Adding range writes}
A range write of $[b, e)$ at version $v$ performs a point write of $b$ at $v$, and then inserts a node at $e$ with $range$ set to $v$, and $point$ set such that the result of checking a read of $e$ is unaffected.
Nodes along the search path to $e$ that are a strict prefix of $e$ get $max$ set to $v$, and all nodes between $b$ and $e$ are removed.
\begin{figure}
\caption{}
\label{fig:tree}

View File

@@ -2,5 +2,7 @@
# This has the effect of making visibility=hidden symbols private in object files
for obj in "$@" ; do
ld -r "$obj" -o "$obj.tmp" && mv "$obj.tmp" "$obj"
ld -r "$obj" -o "$obj.tmp"
touch -r "$obj" "$obj.tmp"
mv "$obj.tmp" "$obj"
done

12
test_conflict_set.py Normal file
View File

@@ -0,0 +1,12 @@
from conflict_set import *
def test_conflict_set():
with ConflictSet() as cs:
before = cs.getBytes()
key = b"a key"
cs.addWrites(1, write(key))
assert cs.getBytes() - before > 0
assert cs.check(read(0, key)) == [Result.CONFLICT]
cs.setOldestVersion(1)
assert cs.check(read(0, key)) == [Result.TOO_OLD]