31 Commits

Author SHA1 Message Date
71c39f9955 Opt-in to rpm/deb default package filenames
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-29 15:56:58 -07:00
8cc17158fd Fix preprocessing instructions for linux
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-28 14:58:29 -07:00
ab211c646a Apply compiler-appeasing syntax changes from Taoxi 2024-03-28 14:57:31 -07:00
7af961f141 Fix jenkins build
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-28 11:44:01 -07:00
a91df62608 Add USE_SIMD_FALLBACK build in jenkins
Some checks failed
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / SIMD fallback total: 1096, passed: 1096
weaselab/conflict-set/pipeline/head There was a failure building this commit
2024-03-28 11:33:53 -07:00
0a1843a161 Add USE_SIMD_FALLBACK
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-28 11:12:50 -07:00
4edf0315d9 Find insertion point for Node16 with simd
Closes #13
2024-03-28 10:47:20 -07:00
14515e186a Update readme benchmarks again
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
I think the last skiplisttest benchmark looked bad because I had vscode
+ firefox + ?? open
2024-03-27 16:47:36 -07:00
b0085df5ad Update symbols.txt
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-27 16:29:30 -07:00
76a7e17b29 Update readme benchmarks
Some checks failed
Tests / Clang total: 1096, failed: 2, passed: 1094
Tests / Release [gcc] total: 1096, failed: 2, passed: 1094
Tests / Release [gcc,aarch64] total: 824, failed: 2, passed: 822
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head There was a failure building this commit
I was incorrectly linking to an old build of conflict set, before the
conflict-set name change. I don't know what the regression is from, but
update the README for transparency now.
2024-03-27 16:12:45 -07:00
5cf43d1bfa Add weaselab namespace 2024-03-27 16:07:05 -07:00
25cc427ec5 Assert safe_free size is correct in debug builds
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
Closes #16
2024-03-27 15:34:24 -07:00
c15c2e7b44 Remove redundant static assert
Closes #14
2024-03-27 12:41:45 -07:00
a4d1f91670 Update README benchmark after adding getBytes
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-20 16:26:05 -07:00
b7cdecaf71 Document thread-safety in terms of constness 2024-03-20 16:16:15 -07:00
cda28643a6 Fix ConflictSet_getBytes 2024-03-20 16:15:43 -07:00
cdb5360b9a Allow _GLOBAL_OFFSET_TABLE_ usage
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-20 12:39:18 -07:00
ef224a60f4 Allow use of __tls_get_addr
Some checks failed
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, failed: 1, passed: 1095
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head There was a failure building this commit
2024-03-20 12:29:18 -07:00
6222b74787 Fix tests
Some checks failed
Tests / Clang total: 1096, failed: 2, passed: 1094
Tests / Release [gcc] total: 1096, failed: 2, passed: 1094
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head There was a failure building this commit
Add the new symbol, and update the valgrind client request so that
Node::partialKeyCapacity is defined.
2024-03-20 12:21:59 -07:00
19edc6f78f Interface change! Add ConflictSet::getBytes
Some checks failed
Tests / Clang total: 1096, failed: 3, passed: 1093
Tests / Release [gcc] total: 1096, failed: 2, passed: 1094
Tests / Release [gcc,aarch64] total: 824, failed: 1, passed: 823
Tests / Coverage total: 823, failed: 1, passed: 822
weaselab/conflict-set/pipeline/head There was a failure building this commit
Closes #12
2024-03-20 12:11:34 -07:00
3f9d01c46a Users will now do find_package(conflict-set)
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-19 19:05:54 -07:00
db03c6f901 Fix invalid package name for debian
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-19 17:43:58 -07:00
c1698b040b Disable tsan for debug builds
All checks were successful
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
Too slow
2024-03-19 16:54:29 -07:00
2e08b54785 Update README.md with 1.0 benchmarks
Some checks reported errors
weaselab/conflict-set/pipeline/head Something is wrong with the build of this commit
Closes #7
2024-03-19 16:41:59 -07:00
aa6f237d50 Document and test thread safety properties
Some checks reported errors
Tests / Clang total: 1096, passed: 1096
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 1096, passed: 1096
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
weaselab/conflict-set/pipeline/head Something is wrong with the build of this commit
Closes #2
2024-03-19 16:27:24 -07:00
becfd25139 De-templatize kUseFreeList
This results in smaller code. This is part of the "let the compiler be
in charge of inlining decisions" theme.
2024-03-19 15:12:31 -07:00
d78b36821b Remove more redundant nullptr checks 2024-03-19 15:05:34 -07:00
ce79b47fbe Revert 303b368fc5
This is a code-size / speed tradeoff. Maybe it's a good idea here, but
it's a bit weird to do this in some places and not others (there are
many places we can avoid switching on type this way). The compiler can
inline and then dead code eliminate to achieve the same effect, so we'll
just let the compiler be in charge of inlining decisions.
2024-03-19 15:01:13 -07:00
727b7e642a Save more redundant nullptr checks 2024-03-19 14:34:59 -07:00
cb4c2b7e1e Avoid redundant null check in some cases
All checks were successful
Tests / Clang total: 825, passed: 825
Clang |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc] total: 825, passed: 825
GNU C Compiler (gcc) |Total|New|Outstanding|Fixed|Trend |:-:|:-:|:-:|:-:|:-: |0|0|0|0|:clap:
Tests / Release [gcc,aarch64] total: 824, passed: 824
Tests / Coverage total: 823, passed: 823
weaselab/conflict-set/pipeline/head This commit looks good
2024-03-19 14:29:13 -07:00
ef9b789745 Remove unused code 2024-03-19 11:12:03 -07:00
17 changed files with 504 additions and 355 deletions

View File

@@ -1,2 +1,2 @@
CompileFlags: CompileFlags:
Add: [-DENABLE_MAIN, -UNDEBUG, -DENABLE_FUZZ, -fexceptions] Add: [-DENABLE_MAIN, -UNDEBUG, -DENABLE_FUZZ, -DTHREAD_TEST, -fexceptions]

View File

@@ -34,7 +34,7 @@ ConflictSet::ReadRange singleton(Arena &arena, std::span<const uint8_t> key) {
std::span<uint8_t>(new (arena) uint8_t[key.size() + 1], key.size() + 1); std::span<uint8_t>(new (arena) uint8_t[key.size() + 1], key.size() + 1);
memcpy(r.data(), key.data(), key.size()); memcpy(r.data(), key.data(), key.size());
r[key.size()] = 0; r[key.size()] = 0;
return {key.data(), int(key.size()), r.data(), int(r.size())}; return {{key.data(), int(key.size())}, {r.data(), int(r.size())}, 0};
} }
ConflictSet::ReadRange prefixRange(Arena &arena, std::span<const uint8_t> key) { ConflictSet::ReadRange prefixRange(Arena &arena, std::span<const uint8_t> key) {
@@ -52,7 +52,7 @@ ConflictSet::ReadRange prefixRange(Arena &arena, std::span<const uint8_t> key) {
auto r = std::span<uint8_t>(new (arena) uint8_t[index + 1], index + 1); auto r = std::span<uint8_t>(new (arena) uint8_t[index + 1], index + 1);
memcpy(r.data(), key.data(), index + 1); memcpy(r.data(), key.data(), index + 1);
r[r.size() - 1]++; r[r.size() - 1]++;
return {key.data(), int(key.size()), r.data(), int(r.size())}; return {{key.data(), int(key.size())}, {r.data(), int(r.size())}, 0};
} }
void benchConflictSet() { void benchConflictSet() {
@@ -258,4 +258,4 @@ void benchConflictSet() {
} }
} }
int main(void) { benchConflictSet(); } int main(void) { benchConflictSet(); }

View File

@@ -1,6 +1,6 @@
cmake_minimum_required(VERSION 3.18) cmake_minimum_required(VERSION 3.18)
project( project(
conflict_set conflict-set
VERSION 0.0.1 VERSION 0.0.1
DESCRIPTION DESCRIPTION
"A data structure for optimistic concurrency control on ranges of bitwise-lexicographically-ordered keys." "A data structure for optimistic concurrency control on ranges of bitwise-lexicographically-ordered keys."
@@ -25,6 +25,9 @@ endif()
add_compile_options(-fdata-sections -ffunction-sections -Wswitch-enum add_compile_options(-fdata-sections -ffunction-sections -Wswitch-enum
-Werror=switch-enum) -Werror=switch-enum)
option(USE_SIMD_FALLBACK
"Use fallback implementations of functions that use SIMD" OFF)
# This is encouraged according to # This is encouraged according to
# https://valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.clientreq # https://valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.clientreq
include_directories(SYSTEM ${CMAKE_SOURCE_DIR}/third_party/valgrind) include_directories(SYSTEM ${CMAKE_SOURCE_DIR}/third_party/valgrind)
@@ -43,18 +46,20 @@ endif()
include(CheckIncludeFileCXX) include(CheckIncludeFileCXX)
include(CMakePushCheckState) include(CMakePushCheckState)
cmake_push_check_state() if(NOT USE_SIMD_FALLBACK)
list(APPEND CMAKE_REQUIRED_FLAGS -mavx) cmake_push_check_state()
check_include_file_cxx("immintrin.h" HAS_AVX) list(APPEND CMAKE_REQUIRED_FLAGS -mavx)
if(HAS_AVX) check_include_file_cxx("immintrin.h" HAS_AVX)
add_compile_options(-mavx) if(HAS_AVX)
add_compile_definitions(HAS_AVX) add_compile_options(-mavx)
endif() add_compile_definitions(HAS_AVX)
cmake_pop_check_state() endif()
cmake_pop_check_state()
check_include_file_cxx("arm_neon.h" HAS_ARM_NEON) check_include_file_cxx("arm_neon.h" HAS_ARM_NEON)
if(HAS_ARM_NEON) if(HAS_ARM_NEON)
add_compile_definitions(HAS_ARM_NEON) add_compile_definitions(HAS_ARM_NEON)
endif()
endif() endif()
set(CMAKE_CXX_IMPLICIT_LINK_LIBRARIES "") set(CMAKE_CXX_IMPLICIT_LINK_LIBRARIES "")
@@ -78,19 +83,19 @@ if(NOT APPLE)
LINKER:--version-script=${CMAKE_SOURCE_DIR}/linker.map) LINKER:--version-script=${CMAKE_SOURCE_DIR}/linker.map)
endif() endif()
add_library(${PROJECT_NAME}_static STATIC add_library(${PROJECT_NAME}-static STATIC
$<TARGET_OBJECTS:${PROJECT_NAME}_object>) $<TARGET_OBJECTS:${PROJECT_NAME}_object>)
if(NOT CMAKE_BUILD_TYPE STREQUAL Debug) if(NOT CMAKE_BUILD_TYPE STREQUAL Debug)
set_target_properties(${PROJECT_NAME}_static PROPERTIES LINKER_LANGUAGE C) set_target_properties(${PROJECT_NAME}-static PROPERTIES LINKER_LANGUAGE C)
endif() endif()
if(NOT APPLE AND CMAKE_OBJCOPY) if(NOT APPLE AND CMAKE_OBJCOPY)
add_custom_command( add_custom_command(
TARGET conflict_set_static TARGET conflict-set-static
POST_BUILD POST_BUILD
COMMAND COMMAND
${CMAKE_OBJCOPY} --keep-global-symbols=${CMAKE_SOURCE_DIR}/symbols.txt ${CMAKE_OBJCOPY} --keep-global-symbols=${CMAKE_SOURCE_DIR}/symbols.txt
$<TARGET_FILE:${PROJECT_NAME}_static>) $<TARGET_FILE:${PROJECT_NAME}-static>)
endif() endif()
set(TEST_FLAGS -Wall -Wextra -Wunreachable-code -Wpedantic -UNDEBUG) set(TEST_FLAGS -Wall -Wextra -Wunreachable-code -Wpedantic -UNDEBUG)
@@ -167,6 +172,21 @@ if(BUILD_TESTING)
add_test(NAME conflict_set_fuzz_${hash} COMMAND fuzz_driver ${TEST}) add_test(NAME conflict_set_fuzz_${hash} COMMAND fuzz_driver ${TEST})
endforeach() endforeach()
# tsan
if(NOT CMAKE_CROSSCOMPILING AND NOT CMAKE_BUILD_TYPE STREQUAL Debug)
add_executable(tsan_driver ConflictSet.cpp FuzzTestDriver.cpp)
target_compile_options(tsan_driver PRIVATE ${TEST_FLAGS} -fsanitize=thread)
target_link_options(tsan_driver PRIVATE -fsanitize=thread)
target_compile_definitions(tsan_driver PRIVATE ENABLE_FUZZ THREAD_TEST)
target_include_directories(tsan_driver
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include)
foreach(TEST ${CORPUS_TESTS})
get_filename_component(hash ${TEST} NAME)
add_test(NAME conflict_set_tsan_${hash} COMMAND tsan_driver ${TEST})
endforeach()
endif()
add_executable(driver TestDriver.cpp) add_executable(driver TestDriver.cpp)
target_compile_options(driver PRIVATE ${TEST_FLAGS}) target_compile_options(driver PRIVATE ${TEST_FLAGS})
target_link_libraries(driver PRIVATE ${PROJECT_NAME}) target_link_libraries(driver PRIVATE ${PROJECT_NAME})
@@ -227,7 +247,7 @@ if(BUILD_TESTING)
NAME conflict_set_static_symbols NAME conflict_set_static_symbols
COMMAND COMMAND
${CMAKE_SOURCE_DIR}/test_symbols.sh ${CMAKE_SOURCE_DIR}/test_symbols.sh
$<TARGET_FILE:${PROJECT_NAME}_static> ${CMAKE_SOURCE_DIR}/symbols.txt) $<TARGET_FILE:${PROJECT_NAME}-static> ${CMAKE_SOURCE_DIR}/symbols.txt)
endif() endif()
# bench # bench
@@ -252,6 +272,10 @@ set(CPACK_RESOURCE_FILE_README "${CMAKE_CURRENT_SOURCE_DIR}/README.md")
set(CPACK_RPM_PACKAGE_ARCHITECTURE ${CMAKE_SYSTEM_PROCESSOR}) set(CPACK_RPM_PACKAGE_ARCHITECTURE ${CMAKE_SYSTEM_PROCESSOR})
set(CPACK_RPM_SPEC_INSTALL_POST "/bin/true") # avoid stripping set(CPACK_RPM_SPEC_INSTALL_POST "/bin/true") # avoid stripping
set(CPACK_RPM_PACKAGE_LICENSE "Apache 2.0") set(CPACK_RPM_PACKAGE_LICENSE "Apache 2.0")
set(CPACK_RPM_FILE_NAME RPM-DEFAULT)
# deb
set(CPACK_DEBIAN_FILE_NAME DEB-DEFAULT)
include(CPack) include(CPack)
@@ -263,7 +287,7 @@ target_include_directories(
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME}>) $<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME}>)
target_include_directories( target_include_directories(
${PROJECT_NAME}_static ${PROJECT_NAME}-static
PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include> PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME}>) $<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME}>)
@@ -272,13 +296,13 @@ set_target_properties(
SOVERSION ${PROJECT_VERSION_MAJOR}) SOVERSION ${PROJECT_VERSION_MAJOR})
install( install(
TARGETS ${PROJECT_NAME} ${PROJECT_NAME}_static TARGETS ${PROJECT_NAME} ${PROJECT_NAME}-static
EXPORT ConflictSetConfig EXPORT conflict-setConfig
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
install(DIRECTORY include/ install(DIRECTORY include/
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME}) DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/${PROJECT_NAME})
install(EXPORT ConflictSetConfig install(EXPORT conflict-setConfig
DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/ConflictSet/cmake) DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/conflict-set/cmake)

View File

@@ -29,6 +29,7 @@ limitations under the License.
#include <span> #include <span>
#include <string> #include <string>
#include <string_view> #include <string_view>
#include <type_traits>
#include <utility> #include <utility>
#ifdef HAS_AVX #ifdef HAS_AVX
@@ -39,6 +40,8 @@ limitations under the License.
#include <memcheck.h> #include <memcheck.h>
using namespace weaselab;
// Use assert for checking potentially complex properties during tests. // Use assert for checking potentially complex properties during tests.
// Use assume to hint simple properties to the optimizer. // Use assume to hint simple properties to the optimizer.
@@ -216,6 +219,7 @@ struct Node {
/* end section that's copied to the next node */ /* end section that's copied to the next node */
uint8_t *partialKey(); uint8_t *partialKey();
size_t size() const;
Type getType() const { return type; } Type getType() const { return type; }
int32_t getCapacity() const { return partialKeyCapacity; } int32_t getCapacity() const { return partialKeyCapacity; }
@@ -249,6 +253,8 @@ struct Node0 : Node {
void copyChildrenAndKeyFrom(const Node0 &other); void copyChildrenAndKeyFrom(const Node0 &other);
void copyChildrenAndKeyFrom(const struct Node3 &other); void copyChildrenAndKeyFrom(const struct Node3 &other);
size_t size() const { return sizeof(Node0) + getCapacity(); }
}; };
struct Node3 : Node { struct Node3 : Node {
@@ -262,6 +268,8 @@ struct Node3 : Node {
void copyChildrenAndKeyFrom(const Node0 &other); void copyChildrenAndKeyFrom(const Node0 &other);
void copyChildrenAndKeyFrom(const Node3 &other); void copyChildrenAndKeyFrom(const Node3 &other);
void copyChildrenAndKeyFrom(const struct Node16 &other); void copyChildrenAndKeyFrom(const struct Node16 &other);
size_t size() const { return sizeof(Node3) + getCapacity(); }
}; };
struct Node16 : Node { struct Node16 : Node {
@@ -275,6 +283,8 @@ struct Node16 : Node {
void copyChildrenAndKeyFrom(const Node3 &other); void copyChildrenAndKeyFrom(const Node3 &other);
void copyChildrenAndKeyFrom(const Node16 &other); void copyChildrenAndKeyFrom(const Node16 &other);
void copyChildrenAndKeyFrom(const struct Node48 &other); void copyChildrenAndKeyFrom(const struct Node48 &other);
size_t size() const { return sizeof(Node16) + getCapacity(); }
}; };
struct Node48 : Node { struct Node48 : Node {
@@ -290,6 +300,8 @@ struct Node48 : Node {
void copyChildrenAndKeyFrom(const Node16 &other); void copyChildrenAndKeyFrom(const Node16 &other);
void copyChildrenAndKeyFrom(const Node48 &other); void copyChildrenAndKeyFrom(const Node48 &other);
void copyChildrenAndKeyFrom(const struct Node256 &other); void copyChildrenAndKeyFrom(const struct Node256 &other);
size_t size() const { return sizeof(Node48) + getCapacity(); }
}; };
struct Node256 : Node { struct Node256 : Node {
@@ -299,6 +311,8 @@ struct Node256 : Node {
uint8_t *partialKey() { return (uint8_t *)(this + 1); } uint8_t *partialKey() { return (uint8_t *)(this + 1); }
void copyChildrenAndKeyFrom(const Node48 &other); void copyChildrenAndKeyFrom(const Node48 &other);
void copyChildrenAndKeyFrom(const Node256 &other); void copyChildrenAndKeyFrom(const Node256 &other);
size_t size() const { return sizeof(Node256) + getCapacity(); }
}; };
inline void Node0::copyChildrenAndKeyFrom(const Node0 &other) { inline void Node0::copyChildrenAndKeyFrom(const Node0 &other) {
@@ -535,7 +549,7 @@ template <class T> struct BoundedFreeListAllocator {
} else { } else {
// The intent is to filter out too-small nodes in the freelist // The intent is to filter out too-small nodes in the freelist
removeNode(n); removeNode(n);
safe_free(n); safe_free(n, sizeof(T) + n->partialKeyCapacity);
} }
} }
@@ -547,10 +561,9 @@ template <class T> struct BoundedFreeListAllocator {
} }
void release(T *p) { void release(T *p) {
static_assert(std::is_trivially_destructible_v<T>);
if (freeListBytes >= kFreeListMaxMemory) { if (freeListBytes >= kFreeListMaxMemory) {
removeNode(p); removeNode(p);
return safe_free(p); return safe_free(p, sizeof(T) + p->partialKeyCapacity);
} }
memcpy((void *)p, &freeList, sizeof(freeList)); memcpy((void *)p, &freeList, sizeof(freeList));
freeList = p; freeList = p;
@@ -560,11 +573,11 @@ template <class T> struct BoundedFreeListAllocator {
~BoundedFreeListAllocator() { ~BoundedFreeListAllocator() {
for (void *iter = freeList; iter != nullptr;) { for (void *iter = freeList; iter != nullptr;) {
VALGRIND_MAKE_MEM_DEFINED(iter, sizeof(iter)); VALGRIND_MAKE_MEM_DEFINED(iter, sizeof(Node));
auto *tmp = iter; auto *tmp = (T *)iter;
memcpy(&iter, iter, sizeof(void *)); memcpy(&iter, iter, sizeof(void *));
removeNode(((T *)tmp)); removeNode((tmp));
safe_free(tmp); safe_free(tmp, sizeof(T) + tmp->partialKeyCapacity);
} }
} }
@@ -590,6 +603,23 @@ uint8_t *Node::partialKey() {
} }
} }
size_t Node::size() const {
switch (type) {
case Type_Node0:
return ((Node0 *)this)->size();
case Type_Node3:
return ((Node3 *)this)->size();
case Type_Node16:
return ((Node16 *)this)->size();
case Type_Node48:
return ((Node48 *)this)->size();
case Type_Node256:
return ((Node256 *)this)->size();
default: // GCOVR_EXCL_LINE
__builtin_unreachable(); // GCOVR_EXCL_LINE
}
}
struct NodeAllocators { struct NodeAllocators {
BoundedFreeListAllocator<Node0> node0; BoundedFreeListAllocator<Node0> node0;
BoundedFreeListAllocator<Node3> node3; BoundedFreeListAllocator<Node3> node3;
@@ -925,6 +955,42 @@ Node *&getOrCreateChild(Node *&self, uint8_t index,
assert(self->getType() == Type_Node16); assert(self->getType() == Type_Node16);
++self->numChildren; ++self->numChildren;
#ifdef HAS_AVX
__m128i key_vec = _mm_set1_epi8(index);
__m128i indices;
memcpy(&indices, self16->index, sizeof(self16->index));
__m128i results = _mm_cmpeq_epi8(key_vec, _mm_min_epu8(key_vec, indices));
int mask = (1 << (self->numChildren - 1)) - 1;
uint32_t bitfield = _mm_movemask_epi8(results) & mask;
bitfield |= uint32_t(1) << (self->numChildren - 1);
int i = std::countr_zero(bitfield);
if (i < self->numChildren - 1) {
memmove(self16->index + i + 1, self16->index + i,
self->numChildren - (i + 1));
memmove(self16->children + i + 1, self16->children + i,
(self->numChildren - (i + 1)) * sizeof(Child));
}
#elif defined(HAS_ARM_NEON)
uint8x16_t indices;
memcpy(&indices, self16->index, sizeof(self16->index));
// 0xff for each leq
auto results = vcleq_u8(vdupq_n_u8(index), indices);
uint64_t mask = (uint64_t(1) << ((self->numChildren - 1) * 4)) - 1;
// 0xf for each 0xff (within mask)
uint64_t bitfield =
vget_lane_u64(
vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(results), 4)),
0) &
mask;
bitfield |= uint64_t(0xf) << ((self->numChildren - 1) * 4);
int i = std::countr_zero(bitfield) / 4;
if (i < self->numChildren - 1) {
memmove(self16->index + i + 1, self16->index + i,
self->numChildren - (i + 1));
memmove(self16->children + i + 1, self16->children + i,
(self->numChildren - (i + 1)) * sizeof(Child));
}
#else
int i = 0; int i = 0;
for (; i < int(self->numChildren) - 1; ++i) { for (; i < int(self->numChildren) - 1; ++i) {
if (int(self16->index[i]) > int(index)) { if (int(self16->index[i]) > int(index)) {
@@ -935,6 +1001,7 @@ Node *&getOrCreateChild(Node *&self, uint8_t index,
break; break;
} }
} }
#endif
self16->index[i] = index; self16->index[i] = index;
auto &result = self16->children[i].child; auto &result = self16->children[i].child;
result = nullptr; result = nullptr;
@@ -950,8 +1017,8 @@ Node *&getOrCreateChild(Node *&self, uint8_t index,
self = newSelf; self = newSelf;
goto insert256; goto insert256;
} }
insert48:
insert48:
auto *self48 = static_cast<Node48 *>(self); auto *self48 = static_cast<Node48 *>(self);
self48->bitSet.set(index); self48->bitSet.set(index);
++self->numChildren; ++self->numChildren;
@@ -963,6 +1030,7 @@ Node *&getOrCreateChild(Node *&self, uint8_t index,
return result; return result;
} }
case Type_Node256: { case Type_Node256: {
insert256: insert256:
auto *self256 = static_cast<Node256 *>(self); auto *self256 = static_cast<Node256 *>(self);
++self->numChildren; ++self->numChildren;
@@ -998,21 +1066,21 @@ Node *nextLogical(Node *node) {
// Invalidates `self`, replacing it with a node of at least capacity. // Invalidates `self`, replacing it with a node of at least capacity.
// Does not return nodes to freelists when kUseFreeList is false. // Does not return nodes to freelists when kUseFreeList is false.
template <bool kUseFreeList>
void freeAndMakeCapacityAtLeast(Node *&self, int capacity, void freeAndMakeCapacityAtLeast(Node *&self, int capacity,
NodeAllocators *allocators, NodeAllocators *allocators,
ConflictSet::Impl *impl) { ConflictSet::Impl *impl,
const bool kUseFreeList) {
switch (self->getType()) { switch (self->getType()) {
case Type_Node0: { case Type_Node0: {
auto *self0 = (Node0 *)self; auto *self0 = (Node0 *)self;
auto *newSelf = allocators->node0.allocate(capacity); auto *newSelf = allocators->node0.allocate(capacity);
newSelf->copyChildrenAndKeyFrom(*self0); newSelf->copyChildrenAndKeyFrom(*self0);
getInTree(self, impl) = newSelf; getInTree(self, impl) = newSelf;
if constexpr (kUseFreeList) { if (kUseFreeList) {
allocators->node0.release(self0); allocators->node0.release(self0);
} else { } else {
removeNode(self0); removeNode(self0);
safe_free(self0); safe_free(self0, self0->size());
} }
self = newSelf; self = newSelf;
} break; } break;
@@ -1021,11 +1089,11 @@ void freeAndMakeCapacityAtLeast(Node *&self, int capacity,
auto *newSelf = allocators->node3.allocate(capacity); auto *newSelf = allocators->node3.allocate(capacity);
newSelf->copyChildrenAndKeyFrom(*self3); newSelf->copyChildrenAndKeyFrom(*self3);
getInTree(self, impl) = newSelf; getInTree(self, impl) = newSelf;
if constexpr (kUseFreeList) { if (kUseFreeList) {
allocators->node3.release(self3); allocators->node3.release(self3);
} else { } else {
removeNode(self3); removeNode(self3);
safe_free(self3); safe_free(self3, self3->size());
} }
self = newSelf; self = newSelf;
} break; } break;
@@ -1034,11 +1102,11 @@ void freeAndMakeCapacityAtLeast(Node *&self, int capacity,
auto *newSelf = allocators->node16.allocate(capacity); auto *newSelf = allocators->node16.allocate(capacity);
newSelf->copyChildrenAndKeyFrom(*self16); newSelf->copyChildrenAndKeyFrom(*self16);
getInTree(self, impl) = newSelf; getInTree(self, impl) = newSelf;
if constexpr (kUseFreeList) { if (kUseFreeList) {
allocators->node16.release(self16); allocators->node16.release(self16);
} else { } else {
removeNode(self16); removeNode(self16);
safe_free(self16); safe_free(self16, self16->size());
} }
self = newSelf; self = newSelf;
} break; } break;
@@ -1047,11 +1115,11 @@ void freeAndMakeCapacityAtLeast(Node *&self, int capacity,
auto *newSelf = allocators->node48.allocate(capacity); auto *newSelf = allocators->node48.allocate(capacity);
newSelf->copyChildrenAndKeyFrom(*self48); newSelf->copyChildrenAndKeyFrom(*self48);
getInTree(self, impl) = newSelf; getInTree(self, impl) = newSelf;
if constexpr (kUseFreeList) { if (kUseFreeList) {
allocators->node48.release(self48); allocators->node48.release(self48);
} else { } else {
removeNode(self48); removeNode(self48);
safe_free(self48); safe_free(self48, self48->size());
} }
self = newSelf; self = newSelf;
} break; } break;
@@ -1060,11 +1128,11 @@ void freeAndMakeCapacityAtLeast(Node *&self, int capacity,
auto *newSelf = allocators->node256.allocate(capacity); auto *newSelf = allocators->node256.allocate(capacity);
newSelf->copyChildrenAndKeyFrom(*self256); newSelf->copyChildrenAndKeyFrom(*self256);
getInTree(self, impl) = newSelf; getInTree(self, impl) = newSelf;
if constexpr (kUseFreeList) { if (kUseFreeList) {
allocators->node256.release(self256); allocators->node256.release(self256);
} else { } else {
removeNode(self256); removeNode(self256);
safe_free(self256); safe_free(self256, self256->size());
} }
self = newSelf; self = newSelf;
} break; } break;
@@ -1083,11 +1151,9 @@ void maybeDecreaseCapacity(Node *&self, NodeAllocators *allocators,
if (self->getCapacity() <= maxCapacity) { if (self->getCapacity() <= maxCapacity) {
return; return;
} }
freeAndMakeCapacityAtLeast</*kUseFreeList*/ false>(self, maxCapacity, freeAndMakeCapacityAtLeast(self, maxCapacity, allocators, impl, false);
allocators, impl);
} }
template <Type kType>
void maybeDownsize(Node *self, NodeAllocators *allocators, void maybeDownsize(Node *self, NodeAllocators *allocators,
ConflictSet::Impl *impl, Node *&dontInvalidate) { ConflictSet::Impl *impl, Node *&dontInvalidate) {
@@ -1095,9 +1161,7 @@ void maybeDownsize(Node *self, NodeAllocators *allocators,
fprintf(stderr, "maybeDownsize: %s\n", getSearchPathPrintable(self).c_str()); fprintf(stderr, "maybeDownsize: %s\n", getSearchPathPrintable(self).c_str());
#endif #endif
assert(self->getType() == kType); switch (self->getType()) {
static_assert(kType != Type_Node0);
switch (kType) {
case Type_Node0: // GCOVR_EXCL_LINE case Type_Node0: // GCOVR_EXCL_LINE
__builtin_unreachable(); // GCOVR_EXCL_LINE __builtin_unreachable(); // GCOVR_EXCL_LINE
case Type_Node3: { case Type_Node3: {
@@ -1113,8 +1177,7 @@ void maybeDownsize(Node *self, NodeAllocators *allocators,
if (minCapacity > child->getCapacity()) { if (minCapacity > child->getCapacity()) {
const bool update = child == dontInvalidate; const bool update = child == dontInvalidate;
freeAndMakeCapacityAtLeast</*kUseFreeList*/ true>(child, minCapacity, freeAndMakeCapacityAtLeast(child, minCapacity, allocators, impl, true);
allocators, impl);
if (update) { if (update) {
dontInvalidate = child; dontInvalidate = child;
} }
@@ -1201,24 +1264,7 @@ Node *erase(Node *self, NodeAllocators *allocators, ConflictSet::Impl *impl,
if (self->numChildren != 0) { if (self->numChildren != 0) {
const bool update = result == dontInvalidate; const bool update = result == dontInvalidate;
switch (self->getType()) { maybeDownsize(self, allocators, impl, result);
case Type_Node0: // GCOVR_EXCL_LINE
__builtin_unreachable(); // GCOVR_EXCL_LINE
case Type_Node3:
maybeDownsize<Type_Node3>(self, allocators, impl, result);
break;
case Type_Node16:
maybeDownsize<Type_Node16>(self, allocators, impl, result);
break;
case Type_Node48:
maybeDownsize<Type_Node48>(self, allocators, impl, result);
break;
case Type_Node256:
maybeDownsize<Type_Node256>(self, allocators, impl, result);
break;
default: // GCOVR_EXCL_LINE
__builtin_unreachable(); // GCOVR_EXCL_LINE
}
if (update) { if (update) {
dontInvalidate = result; dontInvalidate = result;
} }
@@ -1244,11 +1290,6 @@ Node *erase(Node *self, NodeAllocators *allocators, ConflictSet::Impl *impl,
--parent->numChildren; --parent->numChildren;
assert(parent->numChildren > 0 || parent->entryPresent); assert(parent->numChildren > 0 || parent->entryPresent);
const bool update = result == dontInvalidate;
maybeDownsize<Type_Node3>(parent, allocators, impl, result);
if (update) {
dontInvalidate = result;
}
} break; } break;
case Type_Node16: { case Type_Node16: {
auto *parent16 = static_cast<Node16 *>(parent); auto *parent16 = static_cast<Node16 *>(parent);
@@ -1265,13 +1306,6 @@ Node *erase(Node *self, NodeAllocators *allocators, ConflictSet::Impl *impl,
// By kMinChildrenNode16 // By kMinChildrenNode16
assert(parent->numChildren > 0); assert(parent->numChildren > 0);
const bool update = result == dontInvalidate;
maybeDownsize<Type_Node16>(parent, allocators, impl, result);
if (update) {
dontInvalidate = result;
}
} break; } break;
case Type_Node48: { case Type_Node48: {
auto *parent48 = static_cast<Node48 *>(parent); auto *parent48 = static_cast<Node48 *>(parent);
@@ -1292,12 +1326,6 @@ Node *erase(Node *self, NodeAllocators *allocators, ConflictSet::Impl *impl,
// By kMinChildrenNode48 // By kMinChildrenNode48
assert(parent->numChildren > 0); assert(parent->numChildren > 0);
const bool update = result == dontInvalidate;
maybeDownsize<Type_Node48>(parent, allocators, impl, result);
if (update) {
dontInvalidate = result;
}
} break; } break;
case Type_Node256: { case Type_Node256: {
auto *parent256 = static_cast<Node256 *>(parent); auto *parent256 = static_cast<Node256 *>(parent);
@@ -1309,16 +1337,17 @@ Node *erase(Node *self, NodeAllocators *allocators, ConflictSet::Impl *impl,
// By kMinChildrenNode256 // By kMinChildrenNode256
assert(parent->numChildren > 0); assert(parent->numChildren > 0);
const bool update = result == dontInvalidate;
maybeDownsize<Type_Node256>(parent, allocators, impl, result);
if (update) {
dontInvalidate = result;
}
} break; } break;
default: // GCOVR_EXCL_LINE default: // GCOVR_EXCL_LINE
__builtin_unreachable(); // GCOVR_EXCL_LINE __builtin_unreachable(); // GCOVR_EXCL_LINE
} }
const bool update = result == dontInvalidate;
maybeDownsize(parent, allocators, impl, result);
if (update) {
dontInvalidate = result;
}
return result; return result;
} }
@@ -1540,6 +1569,9 @@ bool checkPointRead(Node *n, const std::span<const uint8_t> key,
goto downLeftSpine; goto downLeftSpine;
} else { } else {
n = nextSibling(n); n = nextSibling(n);
if (n == nullptr) {
return true;
}
goto downLeftSpine; goto downLeftSpine;
} }
} }
@@ -1556,6 +1588,9 @@ bool checkPointRead(Node *n, const std::span<const uint8_t> key,
goto downLeftSpine; goto downLeftSpine;
} else { } else {
n = nextSibling(n); n = nextSibling(n);
if (n == nullptr) {
return true;
}
goto downLeftSpine; goto downLeftSpine;
} }
} }
@@ -1570,9 +1605,6 @@ bool checkPointRead(Node *n, const std::span<const uint8_t> key,
} }
} }
downLeftSpine: downLeftSpine:
if (n == nullptr) {
return true;
}
for (;;) { for (;;) {
if (n->entryPresent) { if (n->entryPresent) {
return n->entry.rangeVersion <= readVersion; return n->entry.rangeVersion <= readVersion;
@@ -1695,6 +1727,9 @@ bool checkRangeStartsWith(Node *n, std::span<const uint8_t> key, int begin,
goto downLeftSpine; goto downLeftSpine;
} else { } else {
n = nextSibling(n); n = nextSibling(n);
if (n == nullptr) {
return true;
}
goto downLeftSpine; goto downLeftSpine;
} }
} }
@@ -1712,6 +1747,9 @@ bool checkRangeStartsWith(Node *n, std::span<const uint8_t> key, int begin,
goto downLeftSpine; goto downLeftSpine;
} else { } else {
n = nextSibling(n); n = nextSibling(n);
if (n == nullptr) {
return true;
}
goto downLeftSpine; goto downLeftSpine;
} }
} }
@@ -1729,9 +1767,6 @@ bool checkRangeStartsWith(Node *n, std::span<const uint8_t> key, int begin,
__builtin_unreachable(); // GCOVR_EXCL_LINE __builtin_unreachable(); // GCOVR_EXCL_LINE
downLeftSpine: downLeftSpine:
if (n == nullptr) {
return true;
}
for (;;) { for (;;) {
if (n->entryPresent) { if (n->entryPresent) {
return n->entry.rangeVersion <= readVersion; return n->entry.rangeVersion <= readVersion;
@@ -1799,6 +1834,10 @@ struct CheckRangeLeftSide {
return true; return true;
} else { } else {
n = nextSibling(n); n = nextSibling(n);
if (n == nullptr) {
ok = true;
return true;
}
return downLeftSpine(); return downLeftSpine();
} }
} }
@@ -1826,6 +1865,10 @@ struct CheckRangeLeftSide {
return true; return true;
} else { } else {
n = nextSibling(n); n = nextSibling(n);
if (n == nullptr) {
ok = true;
return true;
}
return downLeftSpine(); return downLeftSpine();
} }
} }
@@ -1862,10 +1905,6 @@ struct CheckRangeLeftSide {
bool downLeftSpine() { bool downLeftSpine() {
phase = DownLeftSpine; phase = DownLeftSpine;
if (n == nullptr) {
ok = true;
return true;
}
return false; return false;
} }
}; };
@@ -2207,7 +2246,7 @@ void destroyTree(Node *root) {
assert(c != nullptr); assert(c != nullptr);
toFree.push_back(c); toFree.push_back(c);
} }
safe_free(n); safe_free(n, n->size());
} }
} }
@@ -2464,10 +2503,10 @@ struct __attribute__((visibility("hidden"))) ConflictSet::Impl {
} }
if (n == nullptr) { if (n == nullptr) {
removalKey = {}; removalKey = {};
return; } else {
removalKeyArena = Arena();
removalKey = getSearchPath(removalKeyArena, n);
} }
removalKeyArena = Arena();
removalKey = getSearchPath(removalKeyArena, n);
} }
explicit Impl(int64_t oldestVersion) : oldestVersion(oldestVersion) { explicit Impl(int64_t oldestVersion) : oldestVersion(oldestVersion) {
@@ -2496,6 +2535,7 @@ struct __attribute__((visibility("hidden"))) ConflictSet::Impl {
Node *root; Node *root;
int64_t rootMaxVersion; int64_t rootMaxVersion;
int64_t oldestVersion; int64_t oldestVersion;
int64_t totalBytes = 0;
}; };
// Precondition - an entry for index must exist in the node // Precondition - an entry for index must exist in the node
@@ -2549,20 +2589,39 @@ void ConflictSet::check(const ReadRange *reads, Result *results,
void ConflictSet::addWrites(const WriteRange *writes, int count, void ConflictSet::addWrites(const WriteRange *writes, int count,
int64_t writeVersion) { int64_t writeVersion) {
return impl->addWrites(writes, count, writeVersion); mallocBytesDelta = 0;
impl->addWrites(writes, count, writeVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
} }
void ConflictSet::setOldestVersion(int64_t oldestVersion) { void ConflictSet::setOldestVersion(int64_t oldestVersion) {
return impl->setOldestVersion(oldestVersion); mallocBytesDelta = 0;
impl->setOldestVersion(oldestVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
} }
int64_t ConflictSet::getBytes() const { return impl->totalBytes; }
ConflictSet::ConflictSet(int64_t oldestVersion) ConflictSet::ConflictSet(int64_t oldestVersion)
: impl(new (safe_malloc(sizeof(Impl))) Impl{oldestVersion}) {} : impl((mallocBytesDelta = 0,
new (safe_malloc(sizeof(Impl))) Impl{oldestVersion})) {
impl->totalBytes += mallocBytesDelta;
}
ConflictSet::~ConflictSet() { ConflictSet::~ConflictSet() {
if (impl) { if (impl) {
impl->~Impl(); impl->~Impl();
safe_free(impl); safe_free(impl, sizeof(*impl));
} }
} }
@@ -2588,24 +2647,44 @@ ConflictSet_check(void *cs, const ConflictSet_ReadRange *reads,
__attribute__((__visibility__("default"))) void __attribute__((__visibility__("default"))) void
ConflictSet_addWrites(void *cs, const ConflictSet_WriteRange *writes, int count, ConflictSet_addWrites(void *cs, const ConflictSet_WriteRange *writes, int count,
int64_t writeVersion) { int64_t writeVersion) {
((ConflictSet::Impl *)cs)->addWrites(writes, count, writeVersion); auto *impl = ((ConflictSet::Impl *)cs);
mallocBytesDelta = 0;
impl->addWrites(writes, count, writeVersion);
impl->totalBytes += mallocBytesDelta;
} }
__attribute__((__visibility__("default"))) void __attribute__((__visibility__("default"))) void
ConflictSet_setOldestVersion(void *cs, int64_t oldestVersion) { ConflictSet_setOldestVersion(void *cs, int64_t oldestVersion) {
((ConflictSet::Impl *)cs)->setOldestVersion(oldestVersion); auto *impl = ((ConflictSet::Impl *)cs);
mallocBytesDelta = 0;
impl->setOldestVersion(oldestVersion);
impl->totalBytes += mallocBytesDelta;
} }
__attribute__((__visibility__("default"))) void * __attribute__((__visibility__("default"))) void *
ConflictSet_create(int64_t oldestVersion) { ConflictSet_create(int64_t oldestVersion) {
return new (safe_malloc(sizeof(ConflictSet::Impl))) mallocBytesDelta = 0;
auto *result = new (safe_malloc(sizeof(ConflictSet::Impl)))
ConflictSet::Impl{oldestVersion}; ConflictSet::Impl{oldestVersion};
result->totalBytes += mallocBytesDelta;
return result;
} }
__attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) { __attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) {
using Impl = ConflictSet::Impl; using Impl = ConflictSet::Impl;
((Impl *)cs)->~Impl(); ((Impl *)cs)->~Impl();
safe_free(cs); safe_free(cs, sizeof(Impl));
}
__attribute__((__visibility__("default"))) int64_t
ConflictSet_getBytes(void *cs) {
using Impl = ConflictSet::Impl;
return ((Impl *)cs)->totalBytes;
} }
} }
// Make sure abi is well-defined
static_assert(std::is_standard_layout_v<ConflictSet::Result>);
static_assert(std::is_standard_layout_v<ConflictSet::Key>);
static_assert(std::is_standard_layout_v<ConflictSet::ReadRange>);
static_assert(std::is_standard_layout_v<ConflictSet::WriteRange>);
namespace { namespace {
std::string getSearchPathPrintable(Node *n) { std::string getSearchPathPrintable(Node *n) {
@@ -2818,8 +2897,8 @@ Iterator firstGeq(Node *n, std::string_view key) {
} }
} }
bool checkCorrectness(Node *node, int64_t oldestVersion, [[maybe_unused]] bool checkCorrectness(Node *node, int64_t oldestVersion,
ConflictSet::Impl *impl) { ConflictSet::Impl *impl) {
bool success = true; bool success = true;
checkParentPointers(node, success); checkParentPointers(node, success);

View File

@@ -2,15 +2,25 @@
#include <cstdint> #include <cstdint>
#include <fstream> #include <fstream>
#include <sstream> #include <sstream>
#include <thread>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);
int main(int argc, char **argv) { int main(int argc, char **argv) {
for (int i = 1; i < argc; ++i) { auto doTest = [&]() {
std::ifstream t(argv[i], std::ios::binary); for (int i = 1; i < argc; ++i) {
std::stringstream buffer; std::ifstream t(argv[i], std::ios::binary);
buffer << t.rdbuf(); std::stringstream buffer;
auto str = buffer.str(); buffer << t.rdbuf();
LLVMFuzzerTestOneInput((const uint8_t *)str.data(), str.size()); auto str = buffer.str();
} LLVMFuzzerTestOneInput((const uint8_t *)str.data(), str.size());
}
};
#ifdef THREAD_TEST
std::thread thread2{doTest};
#endif
doTest();
#ifdef THREAD_TEST
thread2.join();
#endif
} }

View File

@@ -96,13 +96,15 @@ void ConflictSet::setOldestVersion(int64_t oldestVersion) {
return impl->setOldestVersion(oldestVersion); return impl->setOldestVersion(oldestVersion);
} }
int64_t ConflictSet::getBytes() const { return -1; }
ConflictSet::ConflictSet(int64_t oldestVersion) ConflictSet::ConflictSet(int64_t oldestVersion)
: impl(new (safe_malloc(sizeof(Impl))) Impl{oldestVersion}) {} : impl(new (safe_malloc(sizeof(Impl))) Impl{oldestVersion}) {}
ConflictSet::~ConflictSet() { ConflictSet::~ConflictSet() {
if (impl) { if (impl) {
impl->~Impl(); impl->~Impl();
safe_free(impl); safe_free(impl, sizeof(Impl));
} }
} }
@@ -142,6 +144,11 @@ ConflictSet_create(int64_t oldestVersion) {
__attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) { __attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) {
using Impl = ConflictSet::Impl; using Impl = ConflictSet::Impl;
((Impl *)cs)->~Impl(); ((Impl *)cs)->~Impl();
safe_free(cs); safe_free(cs, sizeof(Impl));
}
__attribute__((__visibility__("default"))) int64_t
ConflictSet_getBytes(void *cs) {
using Impl = ConflictSet::Impl;
return -1;
} }
} }

View File

@@ -2,6 +2,8 @@
#include "ConflictSet.h" #include "ConflictSet.h"
using namespace weaselab;
#include <bit> #include <bit>
#include <cassert> #include <cassert>
#include <compare> #include <compare>
@@ -10,10 +12,12 @@
#include <cstdlib> #include <cstdlib>
#include <cstring> #include <cstring>
#include <inttypes.h> #include <inttypes.h>
#include <latch>
#include <map> #include <map>
#include <set> #include <set>
#include <span> #include <span>
#include <string> #include <string>
#include <thread>
#include <unordered_set> #include <unordered_set>
#include <utility> #include <utility>
#include <vector> #include <vector>
@@ -54,43 +58,55 @@ operator<=>(const std::span<const uint8_t> &lhs,
#if SHOW_MEMORY #if SHOW_MEMORY
inline int64_t mallocBytes = 0; inline int64_t mallocBytes = 0;
inline int64_t peakMallocBytes = 0; inline int64_t peakMallocBytes = 0;
constexpr auto kIntMallocHeaderSize = 16; #endif
inline thread_local int64_t mallocBytesDelta = 0;
#ifndef NDEBUG
constexpr auto kMallocHeaderSize = 16;
#endif #endif
// malloc that aborts on OOM and thus always returns a non-null pointer. Must be // malloc that aborts on OOM and thus always returns a non-null pointer. Must be
// paired with `safe_free`. // paired with `safe_free`.
__attribute__((always_inline)) inline void *safe_malloc(size_t s) { __attribute__((always_inline)) inline void *safe_malloc(size_t s) {
mallocBytesDelta += s;
#if SHOW_MEMORY #if SHOW_MEMORY
mallocBytes += s; mallocBytes += s;
if (mallocBytes > peakMallocBytes) { if (mallocBytes > peakMallocBytes) {
peakMallocBytes = mallocBytes; peakMallocBytes = mallocBytes;
} }
void *p = malloc(s + kIntMallocHeaderSize);
if (p == nullptr) {
abort();
}
memcpy(p, &s, sizeof(s));
return (char *)p + kIntMallocHeaderSize;
#else
void *p = malloc(s);
if (p == nullptr) {
abort();
}
return p;
#endif #endif
void *p = malloc(s
#ifndef NDEBUG
+ kMallocHeaderSize
#endif
);
if (p == nullptr) {
abort();
}
#ifndef NDEBUG
memcpy(p, &s, sizeof(s));
(char *&)p += kMallocHeaderSize;
#endif
return p;
} }
// Must be paired with `safe_malloc`. // Must be paired with `safe_malloc`.
// //
// There's nothing safer about this than free. Only called safe_free for // There's nothing safer about this than free. Only called safe_free for
// symmetry with safe_malloc. // symmetry with safe_malloc.
__attribute__((always_inline)) inline void safe_free(void *p) { __attribute__((always_inline)) inline void safe_free(void *p, size_t s) {
mallocBytesDelta -= s;
#if SHOW_MEMORY #if SHOW_MEMORY
size_t s;
memcpy(&s, (char *)p - kIntMallocHeaderSize, sizeof(s));
mallocBytes -= s; mallocBytes -= s;
free((char *)p - kIntMallocHeaderSize); free(p);
#else #else
#ifndef NDEBUG
(char *&)p -= kMallocHeaderSize;
size_t expected;
memcpy(&expected, p, sizeof(expected));
assert(s == expected);
#endif
free(p); free(p);
#endif #endif
} }
@@ -177,7 +193,7 @@ inline Arena::Arena(int initialSize) : impl(nullptr) {
inline void onDestroy(Arena::ArenaImpl *impl) { inline void onDestroy(Arena::ArenaImpl *impl) {
while (impl) { while (impl) {
auto *prev = impl->prev; auto *prev = impl->prev;
safe_free(impl); safe_free(impl, sizeof(Arena::ArenaImpl) + impl->capacity);
impl = prev; impl = prev;
} }
} }
@@ -397,34 +413,6 @@ inline uint32_t Arbitrary::bounded(uint32_t s) {
// ==================== END ARBITRARY IMPL ==================== // ==================== END ARBITRARY IMPL ====================
// ==================== BEGIN UTILITIES IMPL ====================
// Call Stepwise::step for each element of remaining until it returns true.
// Applies a permutation to `remaining` as a side effect.
template <class Stepwise> void runInterleaved(std::span<Stepwise> remaining) {
while (remaining.size() > 0) {
for (int i = 0; i < int(remaining.size());) {
bool done = remaining[i].step();
if (done) {
if (i != int(remaining.size()) - 1) {
using std::swap;
swap(remaining[i], remaining.back());
}
remaining = remaining.subspan(0, remaining.size() - 1);
} else {
++i;
}
}
}
};
template <class Stepwise> void runSequential(std::span<Stepwise> remaining) {
for (auto &r : remaining) {
while (!r.step()) {
}
}
}
struct ReferenceImpl { struct ReferenceImpl {
explicit ReferenceImpl(int64_t oldestVersion) : oldestVersion(oldestVersion) { explicit ReferenceImpl(int64_t oldestVersion) : oldestVersion(oldestVersion) {
writeVersionMap[""] = oldestVersion; writeVersionMap[""] = oldestVersion;
@@ -680,32 +668,60 @@ template <class ConflictSetImpl> struct TestDriver {
auto *results2 = auto *results2 =
new (arena) ConflictSet::Result[numPointReads + numRangeReads]; new (arena) ConflictSet::Result[numPointReads + numRangeReads];
#ifdef THREAD_TEST
auto *results3 =
new (arena) ConflictSet::Result[numPointReads + numRangeReads];
std::latch ready{1};
std::thread thread2{[&]() {
ready.count_down();
cs.check(reads, results3, numPointReads + numRangeReads);
}};
ready.wait();
#endif
CALLGRIND_START_INSTRUMENTATION; CALLGRIND_START_INSTRUMENTATION;
cs.check(reads, results1, numPointReads + numRangeReads); cs.check(reads, results1, numPointReads + numRangeReads);
CALLGRIND_STOP_INSTRUMENTATION; CALLGRIND_STOP_INSTRUMENTATION;
refImpl.check(reads, results2, numPointReads + numRangeReads); refImpl.check(reads, results2, numPointReads + numRangeReads);
for (int i = 0; i < numPointReads + numRangeReads; ++i) {
if (results1[i] != results2[i]) { auto compareResults = [reads](ConflictSet::Result *results1,
if (reads[i].end.len == 0) { ConflictSet::Result *results2, int count) {
fprintf(stderr, for (int i = 0; i < count; ++i) {
"Expected %s, got %s for read of {%s} at version %" PRId64 if (results1[i] != results2[i]) {
"\n", if (reads[i].end.len == 0) {
resultToStr(results2[i]), resultToStr(results1[i]), fprintf(stderr,
printable(reads[i].begin).c_str(), reads[i].readVersion); "Expected %s, got %s for read of {%s} at version %" PRId64
} else { "\n",
fprintf( resultToStr(results2[i]), resultToStr(results1[i]),
stderr, printable(reads[i].begin).c_str(), reads[i].readVersion);
"Expected %s, got %s for read of [%s, %s) at version %" PRId64 } else {
"\n", fprintf(
resultToStr(results2[i]), resultToStr(results1[i]), stderr,
printable(reads[i].begin).c_str(), "Expected %s, got %s for read of [%s, %s) at version %" PRId64
printable(reads[i].end).c_str(), reads[i].readVersion); "\n",
resultToStr(results2[i]), resultToStr(results1[i]),
printable(reads[i].begin).c_str(),
printable(reads[i].end).c_str(), reads[i].readVersion);
}
return false;
} }
ok = false;
return true;
} }
return true;
};
if (!compareResults(results1, results2, numPointReads + numRangeReads)) {
ok = false;
return true;
} }
#ifdef THREAD_TEST
thread2.join();
if (!compareResults(results3, results2, numPointReads + numRangeReads)) {
ok = false;
return true;
}
#endif
} }
return false; return false;
} }

11
Jenkinsfile vendored
View File

@@ -48,6 +48,17 @@ pipeline {
recordIssues(tools: [clang()]) recordIssues(tools: [clang()])
} }
} }
stage('SIMD fallback') {
agent {
dockerfile {
args '-v /home/jenkins/ccache:/ccache'
reuseNode true
}
}
steps {
CleanBuildAndTest("-DUSE_SIMD_FALLBACK=ON")
}
}
stage('Release [gcc]') { stage('Release [gcc]') {
agent { agent {
dockerfile { dockerfile {

100
README.md
View File

@@ -9,49 +9,49 @@ Hardware for all benchmarks is a mac m1 2020.
## Skip list ## Skip list
``` ```
New conflict set: 1.927 sec New conflict set: 1.957 sec
0.649 Mtransactions/sec 0.639 Mtransactions/sec
2.595 Mkeys/sec 2.555 Mkeys/sec
Detect only: 1.838 sec Detect only: 1.845 sec
0.680 Mtransactions/sec 0.678 Mtransactions/sec
2.721 Mkeys/sec 2.710 Mkeys/sec
Skiplist only: 1.256 sec Skiplist only: 1.263 sec
0.995 Mtransactions/sec 0.990 Mtransactions/sec
3.981 Mkeys/sec 3.960 Mkeys/sec
Performance counters: Performance counters:
Build: 0.0381 Build: 0.0546
Add: 0.0499 Add: 0.0563
Detect: 1.84 Detect: 1.84
D.Sort: 0.411 D.Sort: 0.412
D.Combine: 0.0141 D.Combine: 0.0141
D.CheckRead: 0.667 D.CheckRead: 0.671
D.CheckIntraBatch: 0.00673 D.CheckIntraBatch: 0.0068
D.MergeWrite: 0.589 D.MergeWrite: 0.592
D.RemoveBefore: 0.146 D.RemoveBefore: 0.146
``` ```
## Radix tree (this implementation) ## Radix tree (this implementation)
``` ```
New conflict set: 1.318 sec New conflict set: 1.366 sec
0.949 Mtransactions/sec 0.915 Mtransactions/sec
3.795 Mkeys/sec 3.660 Mkeys/sec
Detect only: 1.202 sec Detect only: 1.248 sec
1.040 Mtransactions/sec 1.002 Mtransactions/sec
4.160 Mkeys/sec 4.007 Mkeys/sec
Skiplist only: 0.542 sec Skiplist only: 0.573 sec
2.307 Mtransactions/sec 2.182 Mtransactions/sec
9.227 Mkeys/sec 8.730 Mkeys/sec
Performance counters: Performance counters:
Build: 0.0566 Build: 0.0594
Add: 0.058 Add: 0.0572
Detect: 1.2 Detect: 1.25
D.Sort: 0.411 D.Sort: 0.418
D.Combine: 0.0136 D.Combine: 0.0149
D.CheckRead: 0.22 D.CheckRead: 0.232
D.CheckIntraBatch: 0.00659 D.CheckIntraBatch: 0.0067
D.MergeWrite: 0.322 D.MergeWrite: 0.341
D.RemoveBefore: 0.226 D.RemoveBefore: 0.232
``` ```
# Our benchmark # Our benchmark
@@ -60,25 +60,25 @@ Performance counters:
| ns/op | op/s | err% | total | benchmark | ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:---------- |--------------------:|--------------------:|--------:|----------:|:----------
| 257.12 | 3,889,241.18 | 0.2% | 0.01 | `point reads` | 246.99 | 4,048,700.59 | 0.2% | 0.01 | `point reads`
| 276.38 | 3,618,145.21 | 0.3% | 0.01 | `prefix reads` | 260.16 | 3,843,784.65 | 0.1% | 0.01 | `prefix reads`
| 494.19 | 2,023,531.84 | 0.2% | 0.01 | `range reads` | 493.35 | 2,026,953.19 | 0.1% | 0.01 | `range reads`
| 451.22 | 2,216,229.54 | 1.3% | 0.01 | `point writes` | 462.05 | 2,164,289.23 | 0.6% | 0.01 | `point writes`
| 435.80 | 2,294,622.46 | 0.3% | 0.01 | `prefix writes` | 448.19 | 2,231,205.25 | 0.9% | 0.01 | `prefix writes`
| 246.67 | 4,053,999.27 | 4.2% | 0.02 | `range writes` | 255.83 | 3,908,845.72 | 1.5% | 0.02 | `range writes`
| 555.46 | 1,800,304.91 | 0.9% | 0.01 | `monotonic increasing point writes` | 582.63 | 1,716,349.02 | 1.3% | 0.01 | `monotonic increasing point writes`
## Radix tree (this implementation) ## Radix tree (this implementation)
| ns/op | op/s | err% | total | benchmark | ns/op | op/s | err% | total | benchmark
|--------------------:|--------------------:|--------:|----------:|:---------- |--------------------:|--------------------:|--------:|----------:|:----------
| 19.40 | 51,554,711.61 | 0.2% | 0.01 | `point reads` | 19.42 | 51,483,206.67 | 0.3% | 0.01 | `point reads`
| 57.10 | 17,514,573.13 | 0.4% | 0.01 | `prefix reads` | 58.43 | 17,115,612.57 | 0.1% | 0.01 | `prefix reads`
| 215.65 | 4,637,096.77 | 0.4% | 0.01 | `range reads` | 216.09 | 4,627,766.60 | 0.2% | 0.01 | `range reads`
| 27.52 | 36,340,784.38 | 0.2% | 0.01 | `point writes` | 28.35 | 35,267,567.72 | 0.2% | 0.01 | `point writes`
| 42.16 | 23,720,515.40 | 0.7% | 0.01 | `prefix writes` | 43.43 | 23,026,226.17 | 0.2% | 0.01 | `prefix writes`
| 48.33 | 20,691,082.14 | 2.7% | 0.01 | `range writes` | 50.00 | 20,000,000.00 | 0.0% | 0.01 | `range writes`
| 87.93 | 11,372,164.55 | 2.5% | 0.01 | `monotonic increasing point writes` | 92.38 | 10,824,863.69 | 4.1% | 0.01 | `monotonic increasing point writes`
# "Real data" test # "Real data" test
@@ -87,13 +87,13 @@ Point queries only, best of three runs. Gc ratio is the ratio of time spent doin
## skip list ## skip list
``` ```
Check: 11.3839 seconds, 328.404 MB/s, Add: 5.32878 seconds, 131.745 MB/s, Gc ratio: 45.5903% Check: 11.3385 seconds, 329.718 MB/s, Add: 5.35612 seconds, 131.072 MB/s, Gc ratio: 45.7173%
``` ```
## radix tree ## radix tree
``` ```
Check: 2.55069 seconds, 1465.69 MB/s, Add: 2.08443 seconds, 336.801 MB/s, Gc ratio: 41.748% Check: 2.48583 seconds, 1503.93 MB/s, Add: 2.12768 seconds, 329.954 MB/s, Gc ratio: 41.7943%
``` ```
## hash table ## hash table
@@ -101,5 +101,5 @@ Check: 2.55069 seconds, 1465.69 MB/s, Add: 2.08443 seconds, 336.801 MB/s, Gc rat
(The hash table implementation doesn't work on range queries, and its purpose is to provide an idea of how fast point queries can be) (The hash table implementation doesn't work on range queries, and its purpose is to provide an idea of how fast point queries can be)
``` ```
Check: 1.84205 seconds, 2029.54 MB/s, Add: 0.60281 seconds, 1164.61 MB/s, Gc ratio: 48.8159% Check: 1.83386 seconds, 2038.6 MB/s, Add: 0.601411 seconds, 1167.32 MB/s, Gc ratio: 48.9776%
``` ```

View File

@@ -8,6 +8,8 @@
#include <unistd.h> #include <unistd.h>
#include <vector> #include <vector>
using namespace weaselab;
double now() { double now() {
return std::chrono::duration_cast<std::chrono::nanoseconds>( return std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now().time_since_epoch()) std::chrono::steady_clock::now().time_since_epoch())
@@ -28,7 +30,7 @@ constexpr inline size_t rightAlign(size_t offset, size_t alignment) {
int main(int argc, const char **argv) { int main(int argc, const char **argv) {
// Use with this dataset https://snap.stanford.edu/data/memetracker9.html // Use with this dataset https://snap.stanford.edu/data/memetracker9.html
// Preprocess the files with `sed -i '' '/^Q/d'` // Preprocess the files with `sed -i'' '/^Q/d'`
double checkTime = 0; double checkTime = 0;
double addTime = 0; double addTime = 0;
@@ -40,6 +42,8 @@ int main(int argc, const char **argv) {
int64_t version = 0; int64_t version = 0;
double timer = 0; double timer = 0;
int64_t peakMemory = 0;
for (int i = 1; i < argc; ++i) { for (int i = 1; i < argc; ++i) {
int fd = open(argv[i], O_RDONLY); int fd = open(argv[i], O_RDONLY);
struct stat st; struct stat st;
@@ -109,6 +113,10 @@ int main(int argc, const char **argv) {
write = {}; write = {};
reads.clear(); reads.clear();
if (cs.getBytes() > peakMemory) {
peakMemory = cs.getBytes();
}
timer = now(); timer = now();
cs.setOldestVersion(version - 10000); cs.setOldestVersion(version - 10000);
gcTime += now() - timer; gcTime += now() - timer;
@@ -118,8 +126,9 @@ int main(int argc, const char **argv) {
close(fd); close(fd);
} }
printf( printf("Check: %g seconds, %g MB/s, Add: %g seconds, %g MB/s, Gc ratio: "
"Check: %g seconds, %g MB/s, Add: %g seconds, %g MB/s, Gc ratio: %g%%\n", "%g%%, Peak idle memory: %g\n",
checkTime, checkBytes / checkTime * 1e-6, addTime, checkTime, checkBytes / checkTime * 1e-6, addTime,
addBytes / addTime * 1e-6, gcTime / (gcTime + addTime) * 1e2); addBytes / addTime * 1e-6, gcTime / (gcTime + addTime) * 1e2,
double(peakMemory));
} }

View File

@@ -149,7 +149,7 @@ private:
setMaxVersion(level, v); setMaxVersion(level, v);
} }
void destroy() { safe_free(this); } void destroy() { safe_free(this, getNodeSize()); }
private: private:
int getNodeSize() const { int getNodeSize() const {
@@ -627,6 +627,8 @@ struct __attribute__((visibility("hidden"))) ConflictSet::Impl {
removalArena, {finger.getValue().data(), finger.getValue().size()}); removalArena, {finger.getValue().data(), finger.getValue().size()});
} }
int64_t totalBytes = 0;
private: private:
int64_t keyUpdates = 10; int64_t keyUpdates = 10;
Arena removalArena; Arena removalArena;
@@ -637,25 +639,44 @@ private:
void ConflictSet::check(const ReadRange *reads, Result *results, void ConflictSet::check(const ReadRange *reads, Result *results,
int count) const { int count) const {
return impl->check(reads, results, count); impl->check(reads, results, count);
} }
void ConflictSet::addWrites(const WriteRange *writes, int count, void ConflictSet::addWrites(const WriteRange *writes, int count,
int64_t writeVersion) { int64_t writeVersion) {
return impl->addWrites(writes, count, writeVersion); mallocBytesDelta = 0;
impl->addWrites(writes, count, writeVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
} }
void ConflictSet::setOldestVersion(int64_t oldestVersion) { void ConflictSet::setOldestVersion(int64_t oldestVersion) {
return impl->setOldestVersion(oldestVersion); mallocBytesDelta = 0;
impl->setOldestVersion(oldestVersion);
impl->totalBytes += mallocBytesDelta;
#if SHOW_MEMORY
if (impl->totalBytes != mallocBytes) {
abort();
}
#endif
} }
int64_t ConflictSet::getBytes() const { return impl->totalBytes; }
ConflictSet::ConflictSet(int64_t oldestVersion) ConflictSet::ConflictSet(int64_t oldestVersion)
: impl(new (safe_malloc(sizeof(Impl))) Impl{oldestVersion}) {} : impl((mallocBytesDelta = 0,
new (safe_malloc(sizeof(Impl))) Impl{oldestVersion})) {
impl->totalBytes += mallocBytesDelta;
}
ConflictSet::~ConflictSet() { ConflictSet::~ConflictSet() {
if (impl) { if (impl) {
impl->~Impl(); impl->~Impl();
safe_free(impl); safe_free(impl, sizeof(Impl));
} }
} }
@@ -695,7 +716,12 @@ ConflictSet_create(int64_t oldestVersion) {
__attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) { __attribute__((__visibility__("default"))) void ConflictSet_destroy(void *cs) {
using Impl = ConflictSet::Impl; using Impl = ConflictSet::Impl;
((Impl *)cs)->~Impl(); ((Impl *)cs)->~Impl();
safe_free(cs); safe_free(cs, sizeof(Impl));
}
__attribute__((__visibility__("default"))) int64_t
ConflictSet_getBytes(void *cs) {
using Impl = ConflictSet::Impl;
return ((Impl *)cs)->totalBytes;
} }
} }

View File

@@ -7,6 +7,7 @@ int main(void) {
ConflictSet_WriteRange w; ConflictSet_WriteRange w;
ConflictSet_Result result; ConflictSet_Result result;
ConflictSet_ReadRange r; ConflictSet_ReadRange r;
int64_t bytes;
w.begin.p = (const uint8_t *)"0000"; w.begin.p = (const uint8_t *)"0000";
w.begin.len = 4; w.begin.len = 4;
w.end.len = 0; w.end.len = 0;
@@ -17,6 +18,8 @@ int main(void) {
r.readVersion = 0; r.readVersion = 0;
ConflictSet_check(cs, &r, &result, 1); ConflictSet_check(cs, &r, &result, 1);
assert(result == ConflictSet_Conflict); assert(result == ConflictSet_Conflict);
bytes = ConflictSet_getBytes(cs);
assert(bytes > 0);
ConflictSet_destroy(cs); ConflictSet_destroy(cs);
return 0; return 0;
} }

View File

@@ -2,6 +2,8 @@
#include <cassert> #include <cassert>
using namespace weaselab;
int main(void) { int main(void) {
ConflictSet cs(0); ConflictSet cs(0);
ConflictSet::WriteRange w; ConflictSet::WriteRange w;
@@ -17,4 +19,6 @@ int main(void) {
r.readVersion = 0; r.readVersion = 0;
cs.check(&r, &result, 1); cs.check(&r, &result, 1);
assert(result == ConflictSet::Conflict); assert(result == ConflictSet::Conflict);
int64_t bytes = cs.getBytes();
assert(bytes > 0);
} }

View File

@@ -1,32 +1,19 @@
giff --git a/fdbserver/CMakeLists.txt b/fdbserver/CMakeLists.txt diff --git a/fdbserver/CMakeLists.txt b/fdbserver/CMakeLists.txt
index 3f353c2ef..cd0834761 100644 index 3f353c2ef..074a18628 100644
--- a/fdbserver/CMakeLists.txt --- a/fdbserver/CMakeLists.txt
+++ b/fdbserver/CMakeLists.txt +++ b/fdbserver/CMakeLists.txt
@@ -22,6 +22,9 @@ file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/workloads) @@ -22,6 +22,9 @@ file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/workloads)
add_flow_target(EXECUTABLE NAME fdbserver SRCS ${FDBSERVER_SRCS}) add_flow_target(EXECUTABLE NAME fdbserver SRCS ${FDBSERVER_SRCS})
+find_package(ConflictSet) +find_package(conflict-set)
+target_link_libraries(fdbserver PRIVATE conflict_set_static) +target_link_libraries(fdbserver PRIVATE conflict-set-static)
+ +
if (WITH_SWIFT) if (WITH_SWIFT)
# Setup the Swift sources in FDBServer. # Setup the Swift sources in FDBServer.
include(FindSwiftLibs) include(FindSwiftLibs)
diff --git a/fdbserver/Resolver.actor.cpp b/fdbserver/Resolver.actor.cpp
index bf4118f5f..d3b4eaad8 100644
--- a/fdbserver/Resolver.actor.cpp
+++ b/fdbserver/Resolver.actor.cpp
@@ -132,7 +132,7 @@ struct Resolver : ReferenceCounted<Resolver> {
AsyncVar<int64_t> totalStateBytes;
AsyncTrigger checkNeededVersion;
std::map<NetworkAddress, ProxyRequestsInfo> proxyInfoMap;
- ConflictSet* conflictSet;
+ ConflictSet2* conflictSet;
TransientStorageMetricSample iopsSample;
// Use LogSystem as backend for txnStateStore. However, the real commit
diff --git a/fdbserver/SkipList.cpp b/fdbserver/SkipList.cpp diff --git a/fdbserver/SkipList.cpp b/fdbserver/SkipList.cpp
index b48d32c6b..da106b5d2 100644 index b48d32c6b..da99e03aa 100644
--- a/fdbserver/SkipList.cpp --- a/fdbserver/SkipList.cpp
+++ b/fdbserver/SkipList.cpp +++ b/fdbserver/SkipList.cpp
@@ -25,6 +25,7 @@ @@ -25,6 +25,7 @@
@@ -46,50 +33,35 @@ index b48d32c6b..da106b5d2 100644
static std::vector<PerfDoubleCounter*> skc; static std::vector<PerfDoubleCounter*> skc;
static thread_local uint32_t g_seed = 0; static thread_local uint32_t g_seed = 0;
@@ -782,26 +785,34 @@ private: @@ -783,10 +786,14 @@ private:
}
}; };
-struct ConflictSet { struct ConflictSet {
- ConflictSet() : removalKey(makeString(0)), oldestVersion(0) {} - ConflictSet() : removalKey(makeString(0)), oldestVersion(0) {}
- ~ConflictSet() {} + ConflictSet() : versionHistory(0), removalKey(makeString(0)), oldestVersion(0) {}
+struct ConflictSet2 { ~ConflictSet() {}
+ ConflictSet2() : versionHistory(0), removalKey(makeString(0)), oldestVersion(0) {}
+ ~ConflictSet2() {}
+#if USE_RADIX_TREE +#if USE_RADIX_TREE
+ ConflictSet versionHistory; + weaselab::ConflictSet versionHistory;
+#else +#else
SkipList versionHistory; SkipList versionHistory;
+#endif +#endif
Key removalKey; Key removalKey;
Version oldestVersion; Version oldestVersion;
}; };
@@ -795,7 +802,11 @@ ConflictSet* newConflictSet() {
-ConflictSet* newConflictSet() { return new ConflictSet;
- return new ConflictSet;
+ConflictSet2* newConflictSet() {
+ return new ConflictSet2;
} }
-void clearConflictSet(ConflictSet* cs, Version v) { void clearConflictSet(ConflictSet* cs, Version v) {
- SkipList(v).swap(cs->versionHistory); - SkipList(v).swap(cs->versionHistory);
+void clearConflictSet(ConflictSet2* cs, Version v) {
+#if USE_RADIX_TREE +#if USE_RADIX_TREE
+ cs->versionHistory = ConflictSet{ 0 }; + cs->versionHistory = weaselab::ConflictSet{ 0 };
+#else +#else
+ SkipList().swap(cs->versionHistory); + SkipList().swap(cs->versionHistory);
+#endif +#endif
} }
-void destroyConflictSet(ConflictSet* cs) { void destroyConflictSet(ConflictSet* cs) {
+void destroyConflictSet(ConflictSet2* cs) {
delete cs; delete cs;
}
-ConflictBatch::ConflictBatch(ConflictSet* cs,
+ConflictBatch::ConflictBatch(ConflictSet2* cs,
std::map<int, VectorRef<int>>* conflictingKeyRangeMap,
Arena* resolveBatchReplyArena)
: cs(cs), transactionCount(0), conflictingKeyRangeMap(conflictingKeyRangeMap),
@@ -971,11 +982,15 @@ void ConflictBatch::detectConflicts(Version now, @@ -971,11 +982,15 @@ void ConflictBatch::detectConflicts(Version now,
t = timer(); t = timer();
if (newOldestVersion > cs->oldestVersion) { if (newOldestVersion > cs->oldestVersion) {
@@ -112,7 +84,7 @@ index b48d32c6b..da106b5d2 100644
+#if USE_RADIX_TREE +#if USE_RADIX_TREE
+ Arena arena; + Arena arena;
+ auto* reads = new (arena) ConflictSet::ReadRange[combinedReadConflictRanges.size()]; + auto* reads = new (arena) weaselab::ConflictSet::ReadRange[combinedReadConflictRanges.size()];
+ +
+ for (int i = 0; i < combinedReadConflictRanges.size(); ++i) { + for (int i = 0; i < combinedReadConflictRanges.size(); ++i) {
+ auto& read = reads[i]; + auto& read = reads[i];
@@ -122,11 +94,11 @@ index b48d32c6b..da106b5d2 100644
+ read.end.p = combinedReadConflictRanges[i].end.begin(); + read.end.p = combinedReadConflictRanges[i].end.begin();
+ read.end.len = combinedReadConflictRanges[i].end.size(); + read.end.len = combinedReadConflictRanges[i].end.size();
+ } + }
+ auto* results = new (arena) ConflictSet::Result[combinedReadConflictRanges.size()]; + auto* results = new (arena) weaselab::ConflictSet::Result[combinedReadConflictRanges.size()];
+ cs->versionHistory.check(reads, results, combinedReadConflictRanges.size()); + cs->versionHistory.check(reads, results, combinedReadConflictRanges.size());
+ +
+ for (int i = 0; i < combinedReadConflictRanges.size(); ++i) { + for (int i = 0; i < combinedReadConflictRanges.size(); ++i) {
+ if (results[i] == ConflictSet::Conflict) { + if (results[i] == weaselab::ConflictSet::Conflict) {
+ transactionConflictStatus[combinedReadConflictRanges[i].transaction] = true; + transactionConflictStatus[combinedReadConflictRanges[i].transaction] = true;
+ if (combinedReadConflictRanges[i].conflictingKeyRange != nullptr) { + if (combinedReadConflictRanges[i].conflictingKeyRange != nullptr) {
+ combinedReadConflictRanges[i].conflictingKeyRange->push_back(*combinedReadConflictRanges[i].cKRArena, + combinedReadConflictRanges[i].conflictingKeyRange->push_back(*combinedReadConflictRanges[i].cKRArena,
@@ -147,7 +119,7 @@ index b48d32c6b..da106b5d2 100644
+#if USE_RADIX_TREE +#if USE_RADIX_TREE
+ Arena arena; + Arena arena;
+ auto* writes = new (arena) ConflictSet::WriteRange[combinedWriteConflictRanges.size()]; + auto* writes = new (arena) weaselab::ConflictSet::WriteRange[combinedWriteConflictRanges.size()];
+ +
+ for (int i = 0; i < combinedWriteConflictRanges.size(); ++i) { + for (int i = 0; i < combinedWriteConflictRanges.size(); ++i) {
+ auto& write = writes[i]; + auto& write = writes[i];
@@ -164,15 +136,6 @@ index b48d32c6b..da106b5d2 100644
} }
void ConflictBatch::combineWriteConflictRanges() { void ConflictBatch::combineWriteConflictRanges() {
@@ -1115,7 +1171,7 @@ void skipListTest() {
double start;
- ConflictSet* cs = newConflictSet();
+ ConflictSet2* cs = newConflictSet();
Arena testDataArena;
VectorRef<VectorRef<KeyRangeRef>> testData;
@@ -1197,6 +1253,4 @@ void skipListTest() { @@ -1197,6 +1253,4 @@ void skipListTest() {
for (const auto& counter : skc) { for (const auto& counter : skc) {
printf("%20s: %s\n", counter->getMetric().name().c_str(), counter->getMetric().formatted().c_str()); printf("%20s: %s\n", counter->getMetric().name().c_str(), counter->getMetric().formatted().c_str());
@@ -180,35 +143,3 @@ index b48d32c6b..da106b5d2 100644
- -
- printf("%d entries in version history\n", cs->versionHistory.count()); - printf("%d entries in version history\n", cs->versionHistory.count());
} }
diff --git a/fdbserver/include/fdbserver/ConflictSet.h b/fdbserver/include/fdbserver/ConflictSet.h
index 90ed2c406..b7e31217c 100644
--- a/fdbserver/include/fdbserver/ConflictSet.h
+++ b/fdbserver/include/fdbserver/ConflictSet.h
@@ -28,13 +28,13 @@
#include "fdbclient/CommitTransaction.h"
#include "fdbserver/ResolverBug.h"
-struct ConflictSet;
-ConflictSet* newConflictSet();
-void clearConflictSet(ConflictSet*, Version);
-void destroyConflictSet(ConflictSet*);
+struct ConflictSet2;
+ConflictSet2* newConflictSet();
+void clearConflictSet(ConflictSet2*, Version);
+void destroyConflictSet(ConflictSet2*);
struct ConflictBatch {
- explicit ConflictBatch(ConflictSet*,
+ explicit ConflictBatch(ConflictSet2*,
std::map<int, VectorRef<int>>* conflictingKeyRangeMap = nullptr,
Arena* resolveBatchReplyArena = nullptr);
~ConflictBatch();
@@ -54,7 +54,7 @@ struct ConflictBatch {
void GetTooOldTransactions(std::vector<int>& tooOldTransactions);
private:
- ConflictSet* cs;
+ ConflictSet2* cs;
Standalone<VectorRef<struct TransactionInfo*>> transactionInfo;
std::vector<struct KeyInfo> points;
int transactionCount;

View File

@@ -19,7 +19,16 @@ limitations under the License.
#include <stdint.h> #include <stdint.h>
#ifdef __cplusplus #ifdef __cplusplus
namespace weaselab {
/** A data structure for optimistic concurrency control on ranges of
* bitwise-lexicographically-ordered keys.
*
* Thread safety:
* - It's safe to operate on two different ConflictSets in two different
* threads concurrently
* - It's safe to have multiple threads operating on the same ConflictSet
* concurrently if and only if all threads only call const methods
*/
struct __attribute__((__visibility__("default"))) ConflictSet { struct __attribute__((__visibility__("default"))) ConflictSet {
enum Result { enum Result {
/** The result of a check which does not intersect any conflicting writes */ /** The result of a check which does not intersect any conflicting writes */
@@ -76,6 +85,9 @@ struct __attribute__((__visibility__("default"))) ConflictSet {
~ConflictSet(); ~ConflictSet();
/** Returns the total bytes in use by this ConflictSet */
int64_t getBytes() const;
#if __cplusplus > 199711L #if __cplusplus > 199711L
ConflictSet(ConflictSet &&) noexcept; ConflictSet(ConflictSet &&) noexcept;
ConflictSet &operator=(ConflictSet &&) noexcept; ConflictSet &operator=(ConflictSet &&) noexcept;
@@ -89,9 +101,20 @@ struct __attribute__((__visibility__("default"))) ConflictSet {
private: private:
Impl *impl; Impl *impl;
}; };
} /* namespace weaselab */
#else #else
/** A data structure for optimistic concurrency control on ranges of
* bitwise-lexicographically-ordered keys.
*
* Thread safety:
* - It's safe to operate on two different ConflictSets in two different
* threads concurrently
* - It's safe to have multiple threads operating on the same ConflictSet
* concurrently if and only if all threads only call functions that accept a
* const ConflictSet pointer
*/
typedef struct ConflictSet ConflictSet; typedef struct ConflictSet ConflictSet;
typedef enum { typedef enum {
@@ -130,7 +153,8 @@ typedef struct {
} ConflictSet_WriteRange; } ConflictSet_WriteRange;
/** The result of checking reads[i] is written in results[i] */ /** The result of checking reads[i] is written in results[i] */
void ConflictSet_check(ConflictSet *cs, const ConflictSet_ReadRange *reads, void ConflictSet_check(const ConflictSet *cs,
const ConflictSet_ReadRange *reads,
ConflictSet_Result *results, int count); ConflictSet_Result *results, int count);
/** `writes` must be sorted ascending, and must not have adjacent or /** `writes` must be sorted ascending, and must not have adjacent or
@@ -152,4 +176,7 @@ ConflictSet *ConflictSet_create(int64_t oldestVersion);
void ConflictSet_destroy(ConflictSet *cs); void ConflictSet_destroy(ConflictSet *cs);
/** Returns the total bytes in use by this ConflictSet */
int64_t ConflictSet_getBytes(const ConflictSet *cs);
#endif #endif

View File

@@ -2,14 +2,16 @@ ConflictSet_addWrites
ConflictSet_check ConflictSet_check
ConflictSet_create ConflictSet_create
ConflictSet_destroy ConflictSet_destroy
ConflictSet_getBytes
ConflictSet_setOldestVersion ConflictSet_setOldestVersion
_ZN11ConflictSet16setOldestVersionEl _ZN8weaselab11ConflictSet16setOldestVersionEl
_ZN11ConflictSet9addWritesEPKNS_10WriteRangeEil _ZN8weaselab11ConflictSet9addWritesEPKNS0_10WriteRangeEil
_ZN11ConflictSetaSEOS_ _ZN8weaselab11ConflictSetaSEOS0_
_ZN11ConflictSetC1El _ZN8weaselab11ConflictSetC1El
_ZN11ConflictSetC1EOS_ _ZN8weaselab11ConflictSetC1EOS0_
_ZN11ConflictSetC2El _ZN8weaselab11ConflictSetC2El
_ZN11ConflictSetC2EOS_ _ZN8weaselab11ConflictSetC2EOS0_
_ZN11ConflictSetD1Ev _ZN8weaselab11ConflictSetD1Ev
_ZN11ConflictSetD2Ev _ZN8weaselab11ConflictSetD2Ev
_ZNK11ConflictSet5checkEPKNS_9ReadRangeEPNS_6ResultEi _ZNK8weaselab11ConflictSet5checkEPKNS0_9ReadRangeEPNS0_6ResultEi
_ZNK8weaselab11ConflictSet8getBytesEv

View File

@@ -3,4 +3,4 @@
set -euo pipefail set -euo pipefail
diff -u <(sort < "$2") <(nm "$1" | grep " T " | cut -f3 -d " " | sort) diff -u <(sort < "$2") <(nm "$1" | grep " T " | cut -f3 -d " " | sort)
nm "$1" | grep " U " | (! grep -Pv 'abort|free|malloc|mem[a-z]*|__ashlti3|__stack_chk_[a-z]*') nm "$1" | grep " U " | (! grep -Pv 'abort|free|malloc|mem[a-z]*|__ashlti3|__stack_chk_[a-z]*|__tls_get_addr|_GLOBAL_OFFSET_TABLE_')