19 Commits

Author SHA1 Message Date
d2e1863593 Account for every uncovered line in the implementation
All checks were successful
Tests / Clang total: 1162, passed: 1162
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1162, passed: 1162
Tests / Release [gcc] total: 1162, passed: 1162
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 868, passed: 868
Tests / Coverage total: 872, passed: 872
weaselab/conflict-set/pipeline/head This commit looks good
Closes #23
2024-04-18 12:43:48 -07:00
bf91bca16d Include long common prefix in fuzz test 2024-04-18 12:43:48 -07:00
08ed17f47b Fail jenkins build if not 100% line coverage
Some checks failed
Tests / Clang total: 1093, passed: 1093
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1093, passed: 1093
Tests / Release [gcc] total: 1093, passed: 1093
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 817, passed: 817
Tests / Coverage total: 820, passed: 820
weaselab/conflict-set/pipeline/head There was a failure building this commit
2024-04-18 12:43:48 -07:00
76a45f16ad Exercise freelist size limiting code
All checks were successful
Tests / Clang total: 1093, passed: 1093
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1093, passed: 1093
Tests / Release [gcc] total: 1093, passed: 1093
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 817, passed: 817
Tests / Coverage total: 820, passed: 820
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-17 18:27:01 -07:00
c15d296432 Exercise copyChildrenAndKeyFrom for Node{48,256} to itself
All checks were successful
Tests / Clang total: 1092, passed: 1092
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1092, passed: 1092
Tests / Release [gcc] total: 1092, passed: 1092
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 817, passed: 817
Tests / Coverage total: 819, passed: 819
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-17 17:13:22 -07:00
64a98c529c Fix conflict_set.py bug, and add full inner words test
All checks were successful
Tests / Clang total: 1091, passed: 1091
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1091, passed: 1091
Tests / Release [gcc] total: 1091, passed: 1091
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 817, passed: 817
Tests / Coverage total: 818, passed: 818
weaselab/conflict-set/pipeline/head This commit looks good
Apparently all bytes were 0
2024-04-17 14:11:26 -07:00
ed1388ed21 Disable script tests when cross compiling
All checks were successful
Tests / Clang total: 1090, passed: 1090
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1090, passed: 1090
Tests / Release [gcc] total: 1090, passed: 1090
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 817, passed: 817
Tests / Coverage total: 817, passed: 817
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-17 12:25:16 -07:00
309d315956 Add DebugConflictSet, which asserts using skip list as a reference
Some checks failed
Tests / Clang total: 1090, passed: 1090
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1090, passed: 1090
Tests / Release [gcc] total: 1090, passed: 1090
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 818, failed: 1, passed: 817
Tests / Coverage total: 817, passed: 817
weaselab/conflict-set/pipeline/head There was a failure building this commit
CC #23
2024-04-17 12:08:39 -07:00
eab2e46a56 Mention that we modified SkipList.cpp 2024-04-17 12:08:33 -07:00
85db1a8786 Allow to choose implementation in python wrapper
And fix a few minor bugs to make the python tests pass for skip_list.

CC #23
2024-04-17 12:08:28 -07:00
717f9d6829 Remove ScriptTest.cpp, replace with test_conflict_set.py
CC #23
2024-04-17 12:08:08 -07:00
fd93300ce8 Tweaks and add more planned sections
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-16 17:36:29 -07:00
b7e16b31ff Fill out empty subsections
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-16 12:57:57 -07:00
a324d31518 Second pass at "Checking range reads" 2024-04-16 12:14:04 -07:00
fdb05e0e33 First draft of "Checking range reads" subsection
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-15 17:32:33 -07:00
7c27d4a972 Always target macos 11.0 2024-04-09 14:08:58 -07:00
738de01cb4 Add getBytes to conflict_set.py
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 17:33:51 -07:00
325cab6a95 Target earliest convenient macos version
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 17:16:01 -07:00
0b2821941a Bump version
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-04-08 15:52:06 -07:00
576 changed files with 429 additions and 1096 deletions

View File

@@ -1,7 +1,7 @@
cmake_minimum_required(VERSION 3.18)
project(
conflict-set
VERSION 0.0.3
VERSION 0.0.4
DESCRIPTION
"A data structure for optimistic concurrency control on ranges of bitwise-lexicographically-ordered keys."
HOMEPAGE_URL "https://git.weaselab.dev/weaselab/conflict-set"
@@ -261,14 +261,22 @@ if(BUILD_TESTING)
# scripted tests. Written manually to fill in anything libfuzzer couldn't
# find.
add_executable(script_test ScriptTest.cpp)
target_compile_options(script_test PRIVATE ${TEST_FLAGS})
target_link_libraries(script_test PRIVATE ${PROJECT_NAME})
file(GLOB SCRIPT_TESTS ${CMAKE_SOURCE_DIR}/script_tests/*)
foreach(TEST ${SCRIPT_TESTS})
get_filename_component(name ${TEST} NAME)
add_test(NAME conflict_set_script_${name} COMMAND script_test ${TEST})
endforeach()
if(NOT CMAKE_CROSSCOMPILING)
find_package(Python3 REQUIRED COMPONENTS Interpreter)
set_property(
DIRECTORY
APPEND
PROPERTY CMAKE_CONFIGURE_DEPENDS ${CMAKE_SOURCE_DIR}/test_conflict_set.py)
execute_process(
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/test_conflict_set.py
list OUTPUT_VARIABLE SCRIPT_TESTS)
foreach(TEST ${SCRIPT_TESTS})
add_test(
NAME script_test_${TEST}
COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/test_conflict_set.py
test ${TEST} --build-dir ${CMAKE_BINARY_DIR})
endforeach()
endif()
find_program(VALGRIND_EXE valgrind)
if(VALGRIND_EXE AND NOT CMAKE_CROSSCOMPILING)
@@ -357,6 +365,7 @@ else()
endif()
# macos
set(CMAKE_OSX_DEPLOYMENT_TARGET 11.0)
if(APPLE)
find_program(PANDOC_EXE pandoc)
if(PANDOC_EXE)

View File

@@ -770,7 +770,7 @@ template <class NodeT> int getChildGeqSimd(NodeT *self, int child) {
// cachegrind says the plain loop is fewer instructions and more mis-predicted
// branches. Microbenchmark says plain loop is faster. It's written in this
// weird "generic" way though in case someday we can use the simd
// weird "generic" way though so that someday we can use the simd
// implementation easily if we want.
if constexpr (std::is_same_v<NodeT, Node3>) {
Node3 *n = (Node3 *)self;
@@ -1146,8 +1146,14 @@ void freeAndMakeCapacityAtLeast(Node *&self, int capacity,
// capacity.
void maybeDecreaseCapacity(Node *&self, NodeAllocators *allocators,
ConflictSet::Impl *impl) {
const int maxCapacity =
(self->numChildren + int(self->entryPresent)) * (self->partialKeyLen + 1);
#if DEBUG_VERBOSE && !defined(NDEBUG)
fprintf(stderr, "maybeDecreaseCapacity: current: %d, max: %d, key: %s\n",
self->getCapacity(), maxCapacity,
getSearchPathPrintable(self).c_str());
#endif
if (self->getCapacity() <= maxCapacity) {
return;
}
@@ -2389,7 +2395,11 @@ Iterator firstGeq(Node *n, const std::span<const uint8_t> key) {
} else {
n = nextSibling(n);
if (n == nullptr) {
return {nullptr, 1};
// This line is genuinely unreachable from any entry point of the
// final library, since we can't remove a key without introducing a
// key after it, and the only production caller of firstGeq is for
// resuming the setOldestVersion scan.
return {nullptr, 1}; // GCOVR_EXCL_LINE
}
goto downLeftSpine;
}
@@ -2510,6 +2520,10 @@ struct __attribute__((visibility("hidden"))) ConflictSet::Impl {
}
explicit Impl(int64_t oldestVersion) : oldestVersion(oldestVersion) {
#if DEBUG_VERBOSE
fprintf(stderr, "radix_tree: create\n");
#endif
// Insert ""
root = allocators.node0.allocate(0);
root->numChildren = 0;
@@ -2524,7 +2538,12 @@ struct __attribute__((visibility("hidden"))) ConflictSet::Impl {
root->entry.pointVersion = oldestVersion;
root->entry.rangeVersion = oldestVersion;
}
~Impl() { destroyTree(root); }
~Impl() {
#if DEBUG_VERBOSE
fprintf(stderr, "radix_tree: destroy\n");
#endif
destroyTree(root);
}
NodeAllocators allocators;
@@ -2647,17 +2666,27 @@ ConflictSet_check(void *cs, const ConflictSet_ReadRange *reads,
__attribute__((__visibility__("default"))) void
ConflictSet_addWrites(void *cs, const ConflictSet_WriteRange *writes, int count,
int64_t writeVersion) {
auto *impl = ((ConflictSet::Impl *)cs);
auto *impl = (ConflictSet::Impl *)cs;
mallocBytesDelta = 0;
impl->addWrites(writes, count, writeVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
}
__attribute__((__visibility__("default"))) void
ConflictSet_setOldestVersion(void *cs, int64_t oldestVersion) {
auto *impl = ((ConflictSet::Impl *)cs);
auto *impl = (ConflictSet::Impl *)cs;
mallocBytesDelta = 0;
impl->setOldestVersion(oldestVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
}
__attribute__((__visibility__("default"))) void *
ConflictSet_create(int64_t oldestVersion) {
@@ -2995,6 +3024,7 @@ void removeKey(Node *n) {
struct __attribute__((visibility("default"))) PeakPrinter {
~PeakPrinter() {
printf("--- radix_tree ---\n");
printf("malloc bytes: %g\n", double(mallocBytes));
printf("Peak malloc bytes: %g\n", double(peakMallocBytes));
printf("Node bytes: %g\n", double(nodeBytes));

View File

@@ -530,10 +530,13 @@ template <class ConflictSetImpl> struct TestDriver {
ConflictSetImpl cs{oldestVersion};
ReferenceImpl refImpl{oldestVersion};
constexpr static auto kMaxKeyLen = 8;
constexpr static auto kMaxKeySuffixLen = 8;
bool ok = true;
const int prefixLen = arbitrary.bounded(512);
const int prefixByte = arbitrary.randT<uint8_t>();
// Call until it returns true, for "done". Check internal invariants etc
// between calls to next.
bool next() {
@@ -552,9 +555,10 @@ template <class ConflictSetImpl> struct TestDriver {
if (!arbitrary.hasEntropy()) {
return true;
}
int keyLen = arbitrary.bounded(kMaxKeyLen);
int keyLen = prefixLen + arbitrary.bounded(kMaxKeySuffixLen);
auto *begin = new (arena) uint8_t[keyLen];
arbitrary.randomBytes(begin, keyLen);
memset(begin, prefixByte, prefixLen);
arbitrary.randomBytes(begin + prefixLen, keyLen - prefixLen);
keys.insert(std::string_view((const char *)begin, keyLen));
}
@@ -619,9 +623,10 @@ template <class ConflictSetImpl> struct TestDriver {
if (!arbitrary.hasEntropy()) {
return true;
}
int keyLen = arbitrary.bounded(kMaxKeyLen);
int keyLen = prefixLen + arbitrary.bounded(kMaxKeySuffixLen);
auto *begin = new (arena) uint8_t[keyLen];
arbitrary.randomBytes(begin, keyLen);
memset(begin, prefixByte, prefixLen);
arbitrary.randomBytes(begin + prefixLen, keyLen - prefixLen);
keys.insert(std::string_view((const char *)begin, keyLen));
}

3
Jenkinsfile vendored
View File

@@ -111,6 +111,9 @@ pipeline {
gcovr -f ConflictSet.cpp --cobertura > build/coverage.xml
'''
cobertura autoUpdateHealth: false, autoUpdateStability: false, coberturaReportFile: 'build/coverage.xml', conditionalCoverageTargets: '70, 0, 0', failUnhealthy: false, failUnstable: false, lineCoverageTargets: '80, 0, 0', maxNumberOfBuilds: 0, methodCoverageTargets: '80, 0, 0', onlyStable: false, sourceEncoding: 'ASCII', zoomCoverageChart: false
sh '''
gcovr -f ConflictSet.cpp --fail-under-line 100 > /dev/null
'''
}
}
}

View File

@@ -1,155 +0,0 @@
#include "Internal.h"
#include <ConflictSet.h>
#include <chrono>
#include <cstring>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <vector>
inline size_t getPageSize() {
static size_t kPageSize = sysconf(_SC_PAGESIZE);
return kPageSize;
}
/// Helper for rounding up to page size (or some other alignment)
constexpr inline size_t rightAlign(size_t offset, size_t alignment) {
return offset % alignment == 0 ? offset
: ((offset / alignment) + 1) * alignment;
}
using StringView = std::basic_string_view<uint8_t>;
inline StringView operator"" _v(const char *str, size_t size) {
return {reinterpret_cast<const uint8_t *>(str), size};
}
int main(int argc, const char **argv) {
ConflictSet cs{0};
ReferenceImpl ref{0};
for (int i = 1; i < argc; ++i) {
int fd = open(argv[i], O_RDONLY);
struct stat st;
if (fstat(fd, &st) == -1) {
int err = errno;
fprintf(stderr, "stat error %s - %s\n", argv[i], strerror(err));
fflush(stderr);
abort();
}
int64_t size = rightAlign(st.st_size, getPageSize());
const uint8_t *begin =
(uint8_t *)mmap(0, size, PROT_READ, MAP_PRIVATE, fd, 0);
madvise((void *)begin, size, MADV_SEQUENTIAL);
auto *const mapOriginal = begin;
const auto sizeOriginal = size;
StringView b;
StringView e;
int64_t v = 0;
int64_t lastWriteVersion = 0;
int64_t lastOldestVersion = 0;
std::vector<ConflictSet::WriteRange> writeRanges;
std::vector<ConflictSet::ReadRange> readRanges;
std::vector<ConflictSet::Result> results;
for (uint8_t *end = (uint8_t *)memchr(begin, '\n', size); end != nullptr;) {
StringView line{begin, static_cast<size_t>(end - begin)};
size -= end - begin + 1;
begin = end + 1;
end = (uint8_t *)memchr(begin, '\n', size);
if (line.starts_with("begin"_v)) {
b = line.substr("begin "_v.size(), line.size());
printf("b <- %.*s\n", int(b.size()), b.data());
} else if (line.starts_with("end"_v)) {
e = line.substr("end "_v.size(), line.size());
printf("e <- %.*s\n", int(e.size()), e.data());
} else if (line.starts_with("version"_v)) {
line = line.substr("version "_v.size(), line.size());
v = 0;
for (auto c : line) {
v = v * 10 + int(c) - int('0');
}
printf("v <- %" PRId64 "\n", v);
} else if (line.starts_with("pointread"_v)) {
printf("pointread\n");
ConflictSet::ReadRange r;
r.begin.p = b.data();
r.begin.len = b.size();
r.end.len = 0;
r.readVersion = v;
readRanges.push_back(r);
} else if (line.starts_with("pointwrite"_v)) {
printf("pointwrite\n");
assert(writeRanges.empty() ||
(writeRanges.back().end.len == 0 ? writeRanges.back().begin
: writeRanges.back().end) < b);
ConflictSet::WriteRange w;
w.begin.p = b.data();
w.begin.len = b.size();
w.end.len = 0;
writeRanges.push_back(w);
} else if (line.starts_with("rangeread"_v)) {
printf("rangeread\n");
ConflictSet::ReadRange r;
r.begin.p = b.data();
r.begin.len = b.size();
r.end.p = e.data();
r.end.len = e.size();
r.readVersion = v;
readRanges.push_back(r);
} else if (line.starts_with("rangewrite"_v)) {
printf("rangewrite\n");
assert(b < e);
assert(writeRanges.empty() ||
(writeRanges.back().end.len == 0 ? writeRanges.back().begin
: writeRanges.back().end) < b);
ConflictSet::WriteRange w;
w.begin.p = b.data();
w.begin.len = b.size();
w.end.p = e.data();
w.end.len = e.size();
writeRanges.push_back(w);
} else if (line.starts_with("check"_v)) {
printf("check\n");
Arena arena;
auto *expected = new (arena) ConflictSet::Result[readRanges.size()];
auto *actual = new (arena) ConflictSet::Result[readRanges.size()];
ref.check(readRanges.data(), expected, readRanges.size());
cs.check(readRanges.data(), actual, readRanges.size());
for (int i = 0; i < int(readRanges.size()); ++i) {
if (expected[i] != actual[i]) {
fprintf(stderr, "Expected %s, got %s at index %d\n",
resultToStr(expected[i]), resultToStr(actual[i]), i);
return 1;
}
}
readRanges = {};
} else if (line.starts_with("addwrites"_v)) {
printf("addwrites\n");
assert(v > lastWriteVersion);
lastWriteVersion = v;
cs.addWrites(writeRanges.data(), writeRanges.size(), v);
ref.addWrites(writeRanges.data(), writeRanges.size(), v);
writeRanges = {};
} else if (line.starts_with("setoldest"_v)) {
printf("setoldest\n");
assert(v > lastOldestVersion);
lastOldestVersion = v;
cs.setOldestVersion(v);
ref.setOldestVersion(v);
} else if (line.empty() || line.starts_with(";"_v)) {
// skip
} else {
printf("Unrecognized line: %.*s\n", int(line.size()), line.data());
}
}
munmap((void *)mapOriginal, sizeOriginal);
close(fd);
}
}

View File

@@ -16,6 +16,8 @@
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* This source code is modified to compile outside of FoundationDB
*/
#include "ConflictSet.h"
@@ -268,13 +270,21 @@ public:
}
explicit SkipList(Version version = 0) {
#if DEBUG_VERBOSE
fprintf(stderr, "skip_list: create\n");
#endif
header = Node::create(StringRef(), MaxLevels - 1);
for (int l = 0; l < MaxLevels; l++) {
header->setNext(l, nullptr);
header->setMaxVersion(l, version);
}
}
~SkipList() { destroy(); }
~SkipList() {
#if DEBUG_VERBOSE
fprintf(stderr, "skip_list: destroy\n");
#endif
destroy();
}
SkipList(SkipList &&other) noexcept : header(other.header) {
other.header = nullptr;
}
@@ -603,6 +613,10 @@ struct __attribute__((visibility("hidden"))) ConflictSet::Impl {
for (int s = stripes - 1; s >= 0; s--) {
for (int i = 0; i * 2 < ss; ++i) {
const auto &w = writes[s * stripeSize / 2 + i];
#if DEBUG_VERBOSE
printf("Write begin: %s\n", printable(w.begin).c_str());
fflush(stdout);
#endif
values[i * 2] = {w.begin.p, size_t(w.begin.len)};
values[i * 2 + 1] = w.end.len > 0
? StringRef{w.end.p, size_t(w.end.len)}
@@ -702,16 +716,35 @@ ConflictSet_check(void *cs, const ConflictSet_ReadRange *reads,
__attribute__((__visibility__("default"))) void
ConflictSet_addWrites(void *cs, const ConflictSet_WriteRange *writes, int count,
int64_t writeVersion) {
((ConflictSet::Impl *)cs)->addWrites(writes, count, writeVersion);
auto *impl = (ConflictSet::Impl *)cs;
mallocBytesDelta = 0;
impl->addWrites(writes, count, writeVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
}
__attribute__((__visibility__("default"))) void
ConflictSet_setOldestVersion(void *cs, int64_t oldestVersion) {
((ConflictSet::Impl *)cs)->setOldestVersion(oldestVersion);
auto *impl = (ConflictSet::Impl *)cs;
mallocBytesDelta = 0;
impl->setOldestVersion(oldestVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
}
__attribute__((__visibility__("default"))) void *
ConflictSet_create(int64_t oldestVersion) {
return new (safe_malloc(sizeof(ConflictSet::Impl)))
mallocBytesDelta = 0;
auto *result = new (safe_malloc(sizeof(ConflictSet::Impl)))
ConflictSet::Impl{oldestVersion};
result->totalBytes += mallocBytesDelta;
return result;
}
__attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) {
using Impl = ConflictSet::Impl;
@@ -728,6 +761,7 @@ ConflictSet_getBytes(void *cs) {
#if SHOW_MEMORY
struct __attribute__((visibility("default"))) PeakPrinter {
~PeakPrinter() {
printf("--- skip_list ---\n");
printf("malloc bytes: %g\n", double(mallocBytes));
printf("Peak malloc bytes: %g\n", double(peakMallocBytes));
}

View File

@@ -4,4 +4,5 @@ _bzero
_free
_malloc
_memcpy
_memmove
_memmove
dyld_stub_binder

View File

@@ -4,22 +4,6 @@ import os
from typing import Optional
_lib = None
for f in (
os.path.dirname(__file__) + "/build/radix_tree/libconflict-set.so.0",
os.path.dirname(__file__) + "/build/radix_tree/libconflict-set.0.dylib",
):
try:
_lib = ctypes.cdll.LoadLibrary(f)
except:
pass
if _lib is None:
import sys
print("Could not find libconflict-set", file=sys.stderr)
sys.exit(1)
class _Key(ctypes.Structure):
_fields_ = [("p", ctypes.POINTER(ctypes.c_ubyte)), ("len", ctypes.c_int)]
@@ -37,28 +21,6 @@ class WriteRange(ctypes.Structure):
_fields_ = [("begin", _Key), ("end", _Key)]
_lib.ConflictSet_create.argtypes = (ctypes.c_int64,)
_lib.ConflictSet_create.restype = ctypes.c_void_p
_lib.ConflictSet_check.argtypes = (
ctypes.c_void_p,
ctypes.POINTER(ReadRange),
ctypes.POINTER(ctypes.c_int),
ctypes.c_int,
)
_lib.ConflictSet_addWrites.argtypes = (
ctypes.c_void_p,
ctypes.POINTER(WriteRange),
ctypes.c_int,
ctypes.c_int64,
)
_lib.ConflictSet_setOldestVersion.argtypes = (ctypes.c_void_p, ctypes.c_int64)
_lib.ConflictSet_destroy.argtypes = (ctypes.c_void_p,)
class Result(enum.Enum):
COMMIT = 0
CONFLICT = 1
@@ -66,55 +28,109 @@ class Result(enum.Enum):
def write(begin: bytes, end: Optional[bytes] = None) -> WriteRange:
b = (ctypes.c_ubyte * len(begin))()
b.value = begin
b = (ctypes.c_ubyte * len(begin)).from_buffer(bytearray(begin))
if end is None:
e = (ctypes.c_ubyte * 0)()
e.value = b""
else:
e = (ctypes.c_ubyte * len(end))()
e.value = end
e = (ctypes.c_ubyte * len(end)).from_buffer(bytearray(end))
return WriteRange(_Key(b, len(b)), _Key(e, len(e)))
def read(version: int, begin: bytes, end: Optional[bytes] = None) -> ReadRange:
b = (ctypes.c_ubyte * len(begin))()
b.value = begin
b = (ctypes.c_ubyte * len(begin)).from_buffer(bytearray(begin))
if end is None:
e = (ctypes.c_ubyte * 0)()
e.value = b""
else:
e = (ctypes.c_ubyte * len(end))()
e.value = end
e = (ctypes.c_ubyte * len(end)).from_buffer(bytearray(end))
return ReadRange(_Key(b, len(b)), _Key(e, len(e)), version)
class ConflictSet:
def __init__(self, version: int = 0) -> None:
self.p = _lib.ConflictSet_create(version)
def __init__(
self,
version: int = 0,
build_dir: Optional[str] = None,
implementation: Optional[str] = None,
) -> None:
self._lib = None
if build_dir is None:
build_dir = os.path.dirname(__file__) + "/build"
if implementation is None:
implementation = "radix_tree"
for f in (
build_dir + "/" + implementation + "/libconflict-set.so.0",
os.path.dirname(__file__)
+ "/build/"
+ implementation
+ "/libconflict-set.0.dylib",
):
try:
self._lib = ctypes.cdll.LoadLibrary(f)
except:
pass
if self._lib is None:
import sys
print(
"Could not find libconflict-set implementation " + implementation,
file=sys.stderr,
)
sys.exit(1)
self._lib.ConflictSet_create.argtypes = (ctypes.c_int64,)
self._lib.ConflictSet_create.restype = ctypes.c_void_p
self._lib.ConflictSet_check.argtypes = (
ctypes.c_void_p,
ctypes.POINTER(ReadRange),
ctypes.POINTER(ctypes.c_int),
ctypes.c_int,
)
self._lib.ConflictSet_addWrites.argtypes = (
ctypes.c_void_p,
ctypes.POINTER(WriteRange),
ctypes.c_int,
ctypes.c_int64,
)
self._lib.ConflictSet_setOldestVersion.argtypes = (
ctypes.c_void_p,
ctypes.c_int64,
)
self._lib.ConflictSet_destroy.argtypes = (ctypes.c_void_p,)
self._lib.ConflictSet_getBytes.argtypes = (ctypes.c_void_p,)
self._lib.ConflictSet_getBytes.restype = ctypes.c_int64
self.p = self._lib.ConflictSet_create(version)
def addWrites(self, version: int, *writes: WriteRange):
_lib.ConflictSet_addWrites(
self._lib.ConflictSet_addWrites(
self.p, (WriteRange * len(writes))(*writes), len(writes), version
)
def check(self, *reads: ReadRange) -> list[Result]:
r = (ctypes.c_int * len(reads))()
_lib.ConflictSet_check(self.p, *reads, r, 1)
self._lib.ConflictSet_check(self.p, *reads, r, 1)
return [Result(x) for x in r]
def setOldestVersion(self, version: int) -> None:
_lib.ConflictSet_setOldestVersion(self.p, version)
self._lib.ConflictSet_setOldestVersion(self.p, version)
def getBytes(self) -> int:
return self._lib.ConflictSet_getBytes(self.p)
def __enter__(self):
return self
def close(self) -> None:
if self.p is not None:
_lib.ConflictSet_destroy(self.p)
self._lib.ConflictSet_destroy(self.p)
self.p = None
def __exit__(self, exception_type, exception_value, exception_traceback):
if self.p is not None:
_lib.ConflictSet_destroy(self.p)
self.p = None
self.close()

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More