# WeaselDB Development Guide

## Table of Contents

1. [Project Overview](#project-overview)
1. [Quick Start](#quick-start)
1. [Architecture](#architecture)
1. [Development Guidelines](#development-guidelines)
1. [Common Patterns](#common-patterns)
1. [Reference](#reference)

**IMPORTANT:** Read [style.md](style.md) first - contains mandatory C++ coding standards, threading rules, and testing guidelines that must be followed for all code changes.

______________________________________________________________________

## Project Overview

WeaselDB is a high-performance write-side database component designed for systems where reading and writing are decoupled. The system focuses exclusively on handling transactional commits with optimistic concurrency control, while readers are expected to maintain their own queryable representations by subscribing to change streams.

### Key Features

- **Ultra-fast arena allocation** (~1ns vs ~20-270ns for malloc)
- **High-performance JSON parsing** with streaming support and SIMD optimization
- **Multi-threaded networking** using multiple epoll instances with unified I/O thread pool
- **Configurable epoll instances** to eliminate kernel-level contention
- **Optimized memory management** with arena allocation and efficient copying
- **Factory pattern safety** ensuring correct object lifecycle management

______________________________________________________________________

## Quick Start

### Build System

Use CMake with C++20 and **always use ninja** (see [style.md](style.md) for build details):

```bash
mkdir -p build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
ninja
```

### Testing & Development

**Run all tests:**

```bash
ninja test    # or ctest
```

**Individual targets:**

- `./test_arena` - Arena allocator unit tests
- `./test_commit_request` - JSON parsing and validation tests
- `./test_http_handler` - HTTP protocol handling tests
- `./test_metric` - Metrics system tests
- `./test_api_url_parser` - API URL parsing tests
- `./test_reference` - Reference counting system tests
- `./test_server_connection_return` - Connection lifecycle tests

**Benchmarking:**

- `./bench_arena` - Memory allocation performance
- `./bench_commit_request` - JSON parsing performance
- `./bench_cpu_work` - CPU work benchmarking utility
- `./bench_format_comparison` - String formatting performance
- `./bench_metric` - Metrics system performance
- `./bench_parser_comparison` - Compare vs nlohmann::json and RapidJSON
- `./bench_reference` - Reference counting performance
- `./bench_thread_pipeline` - Lock-free pipeline performance

**Debug tools:**

- `./debug_arena` - Analyze arena allocator behavior

**Load Testing:**

- `./load_tester` - A tool to generate load against the server for performance and stability analysis.

### Dependencies

**System requirements:**

- **weaseljson** - Must be installed system-wide (high-performance JSON parser)
- **gperf** - System requirement for perfect hash generation

**Auto-fetched:**

- **simdutf** - SIMD base64 encoding/decoding
- **toml11** - TOML configuration parsing
- **doctest** - Testing framework
- **nanobench** - Benchmarking library
- **nlohmann/json** - JSON library (used in benchmarks)
- **RapidJSON** - High-performance JSON library (used in benchmarks)
- **llhttp** - Fast HTTP parser

______________________________________________________________________

## Architecture

### Core Components

#### **Arena Allocator** (`src/arena.hpp`)

Ultra-fast memory allocator optimized for request/response patterns:

- **~1ns allocation time** vs ~20-270ns for malloc
- **Lazy initialization** with geometric block growth (doubling strategy)
- **Intrusive linked list design** for minimal memory overhead
- **Memory-efficient reset** that keeps the first block and frees others
- **STL-compatible interface** via `ArenaStlAllocator`
- **O(1) amortized allocation** with proper alignment handling
- **Move semantics** for efficient transfers
- **Thread-safe per-connection** usage via exclusive ownership model

#### **Networking Layer**

**Server** (`src/server.{hpp,cpp}`):

- **High-performance multi-threaded networking** using multiple epoll instances with unified I/O thread pool
- **Configurable epoll instances** to eliminate kernel-level epoll_ctl contention (default: 2, max: io_threads)
- **Round-robin thread-to-epoll assignment** distributes I/O threads across epoll instances
- **Connection distribution** keeps accepted connections on same epoll, returns via round-robin
- **Factory pattern construction** via `Server::create()` ensures you can only get a `Ref<Server>`
- **Safe shutdown mechanism** with async-signal-safe shutdown() method
- **Connection ownership management** with automatic cleanup on server destruction
- **Pluggable protocol handlers** via ConnectionHandler interface
- **EPOLL_EXCLUSIVE** on listen socket across all epoll instances prevents thundering herd

**Connection** (`src/connection.{hpp,cpp}`):

- **Efficient per-connection state management** with arena-based memory allocation
- **Safe ownership transfer** between server threads and protocol handlers
- **Automatic cleanup** on connection closure or server shutdown
- **Handler interface isolation** - only exposes necessary methods to protocol handlers
- **Protocol-specific data:** `user_data` `void*` for custom handler data

**ConnectionHandler Interface** (`src/connection_handler.hpp`):

- **Abstract protocol interface** decoupling networking from application logic
- **Ownership transfer support** allowing handlers to take connections for async processing
- **Streaming data processing** with partial message handling
- **Connection lifecycle hooks** for initialization and cleanup

#### **Thread Pipeline** (`src/thread_pipeline.hpp`)

A high-performance, multi-stage, lock-free pipeline for inter-thread communication.

- **Lock-Free Design**: Uses a shared ring buffer with atomic counters for coordination, avoiding locks for maximum throughput.
- **Multi-Stage Processing**: Allows items (like connections or data packets) to flow through a series of processing stages (e.g., from I/O threads to worker threads).
- **Batching Support**: Enables efficient batch processing of items to reduce overhead.
- **RAII Guards**: Utilizes RAII (`StageGuard`, `ProducerGuard`) to ensure thread-safe publishing and consumption of items in the pipeline, even in the presence of exceptions.

#### **Parsing Layer**

**JSON Commit Request Parser** (`src/json_commit_request_parser.{hpp,cpp}`):

- **High-performance JSON parser** using `weaseljson` library
- **Streaming parser support** for incremental parsing of network data
- **gperf-optimized token recognition** for fast JSON key parsing
- **Base64 decoding** using SIMD-accelerated simdutf
- **Comprehensive validation** of transaction structure
- **Perfect hash table lookup** for JSON keys using gperf
- **Zero hash collisions** for known JSON tokens eliminates branching

**Parser Interface** (`src/commit_request_parser.hpp`):

- **Abstract base class** for commit request parsers
- **Format-agnostic parsing interface** supporting multiple serialization formats
- **Streaming and one-shot parsing modes**
- **Standardized error handling** across parser implementations

#### **Data Model**

**Commit Request Data Model** (`src/commit_request.hpp`):

- **Format-agnostic data structure** for representing transactional commits
- **Arena-backed string storage** with efficient memory management
- **Move-only semantics** for optimal performance
- **Builder pattern** for constructing commit requests
- **String views** pointing to arena-allocated memory to avoid unnecessary copying

#### **Metrics System** (`src/metric.{hpp,cpp}`)

**High-Performance Metrics Implementation:**

- **Thread-local counters/histograms** with single writer for performance
- **Global gauges** with lock-free atomic CAS operations for multi-writer scenarios
- **SIMD-optimized histogram bucket updates** using AVX instructions for high throughput
- **Arena allocator integration** for efficient memory management during rendering

**Threading Model:**

- **Counters**: Per-thread storage, single writer, atomic write in `Counter::inc()`, atomic read in render thread
- **Histograms**: Per-thread storage, single writer, per-histogram mutex serializes all access (observe and render)
- **Gauges**: Lock-free atomic operations using `std::bit_cast` for double precision
- **Thread cleanup**: Automatic accumulation of thread-local state into global state on destruction

**Prometheus Compatibility:**

- **Standard metric types** with proper label handling and validation
- **Bucket generation helpers** for linear/exponential histogram distributions
- **Callback-based metrics** for dynamic values
- **UTF-8 validation** using simdutf for label values

#### **Configuration & Optimization**

**Configuration System** (`src/config.{hpp,cpp}`):

- **TOML-based configuration** using `toml11` library
- **Structured configuration** with server, commit, and subscription sections
- **Default fallback values** for all configuration options
- **Type-safe parsing** with validation and bounds checking
- See `config.md` for complete configuration documentation

**JSON Token Optimization** (`src/json_tokens.gperf`, `src/json_token_enum.hpp`):

- **Perfect hash table** generated by gperf for O(1) JSON key lookup
- **Compile-time token enumeration** for type-safe key identification
- **Minimal perfect hash** reduces memory overhead and improves cache locality
- **Build-time code generation** ensures optimal performance

### Transaction Data Model

#### CommitRequest Structure

```
CommitRequest {
  - request_id: Optional unique identifier
  - leader_id: Expected leader for consistency
  - read_version: Snapshot version for preconditions
  - preconditions[]: Optimistic concurrency checks
    - point_read: Single key existence/content validation
    - range_read: Range-based consistency validation
  - operations[]: Ordered mutation operations
    - write: Set key-value pair
    - delete: Remove single key
    - range_delete: Remove key range
}
```

### Memory Management Model

#### Connection Ownership Lifecycle

1. **Creation**: Server creates connections and stores them in registry
1. **Processing**: I/O threads access connections via registry lookup
1. **Handler Access**: Handlers receive Connection& references, server retains ownership
1. **Async Processing**: Handlers use WeakRef<Connection> for safe async access
1. **Safety**: Connection mutex synchronizes concurrent access between threads
1. **Cleanup**: RAII ensures proper resource cleanup when connections are destroyed

#### Arena Memory Lifecycle

1. **Request Processing**: Handler creates request-scoped arena for parsing request data
1. **Response Generation**: Handler uses same arena for response construction (headers, JSON, etc.)
1. **Response Queuing**: Handler calls `conn->append_message()` passing span + arena ownership
1. **Response Writing**: I/O thread writes messages to socket, arena freed after completion

> **Note**: Call `conn->reset()` periodically to reclaim arena memory. Best practice is after all outgoing bytes have been written.

#### Threading Model and Server-Owned Connections

**Server-Owned Connection Design:**
WeaselDB uses a server-owned connection model where the server retains ownership of all connections while providing safe concurrent access to handlers:

**Key Benefits:**

1. **Simplified ownership** - Server always owns connections, eliminating complex ownership transfers
1. **Safe concurrent access** - Connection mutexes synchronize access between I/O threads and handlers
1. **WeakRef pattern** - Handlers use WeakRef<Connection> for safe async processing without ownership

**Threading Flow:**

1. **Event Trigger**: Network thread gets epoll event and processes data
1. **Handler Invocation**: Handler receives Connection& reference - server retains ownership
1. **Async Processing**: Handler obtains WeakRef<Connection> for safe background processing
1. **Connection Cleanup**: Server manages connection lifecycle including file descriptor operations

**Performance Benefits:**

- **Reduced syscalls**: Eliminates epoll_ctl(MOD) calls needed for ownership transfer
- **Simplified synchronization**: Connection mutexes provide clear concurrent access patterns
- **Memory efficiency**: No unique_ptr overhead for ownership management

**Safe Async Processing:**

- WeakRef<Connection> prevents use-after-free in background threads
- Connection mutex ensures thread-safe access to connection state
- Server handles all file descriptor management automatically

This design provides high performance concurrent processing while maintaining thread safety through clear ownership boundaries and synchronization primitives.

### API Endpoints

The system implements a RESTful API. See [api.md](api.md) for comprehensive API documentation including request/response formats, examples, and error codes.

### Design Principles

1. **Performance-first** - Every component optimized for high throughput
1. **Scalable concurrency** - Multiple epoll instances eliminate kernel contention
1. **Memory efficiency** - Arena allocation eliminates fragmentation
1. **Efficient copying** - Minimize unnecessary copies while accepting required ones
1. **Streaming-ready** - Support incremental processing
1. **Type safety** - Compile-time validation where possible
1. **Resource management** - RAII and move semantics throughout

______________________________________________________________________

## Development Guidelines

### Code Style & Conventions

See [style.md](style.md) for comprehensive C++ coding standards and conventions.

### Critical Implementation Rules

- **Server Creation**: Always use `Server::create()` factory method - direct construction is impossible
- **Connection Creation**: Only the Server can create connections - no public constructor or factory method
- **Connection Ownership**: Server retains ownership, handlers use Connection& references
- **Arena Allocator Pattern**: Always use `Arena` for temporary allocations within request processing
- **String View Usage**: Prefer `std::string_view` over `std::string` when pointing to arena-allocated memory
- **Async Processing**: Use `conn.get_weak_ref()` for safe background processing without ownership
- **JSON Token Lookup**: Use the gperf-generated perfect hash table in `json_tokens.hpp` for O(1) key recognition
- **Base64 Handling**: Always use simdutf for base64 encoding/decoding for performance
- **Thread Safety**: Connection mutexes provide safe concurrent access between threads

### Project Structure

- **`src/`** - Core headers and implementation files
- **`tests/`** - doctest-based unit tests
- **`benchmarks/`** - nanobench performance tests
- **`tools/`** - Debugging and analysis utilities
- **`build/`** - CMake-generated files including `json_tokens.cpp`

### Extension Points

#### Adding New Protocol Handlers

1. Inherit from `ConnectionHandler` in `src/connection_handler.hpp`
1. Implement `on_data_arrived()` using Connection& reference parameter
1. Use connection's arena allocator for temporary allocations: `conn.get_arena()`
1. Handle partial messages and streaming protocols appropriately
1. Use `conn.get_weak_ref()` for safe async processing without ownership transfer
1. Add corresponding test cases and integration tests
1. Consider performance implications of concurrent access patterns

#### Adding New Parsers

1. Inherit from `CommitRequestParser` in `src/commit_request_parser.hpp`
1. Implement both streaming and one-shot parsing modes
1. Use arena allocation for all temporary string storage
1. Add corresponding test cases in `tests/`
1. Add benchmark comparisons in `benchmarks/`

### Performance Guidelines

- **Memory**: Arena allocation eliminates fragmentation - use it for all request-scoped data
- **CPU**: Perfect hashing and SIMD operations are critical paths - avoid alternatives
- **I/O**: Streaming parser design supports incremental network data processing
- **Cache**: String views avoid copying, keeping data cache-friendly

### Configuration & Testing

- **Configuration**: All configuration is TOML-based using `config.toml` (see `config.md`)
- **Testing Strategy**: Run unit tests, benchmarks, and debug tools before submitting changes
- **Build System**: CMake generates gperf hash tables at build time
- **Testing Guidelines**: See [style.md](style.md) for comprehensive testing standards including synchronization rules

______________________________________________________________________

## Common Patterns

### Factory Method Patterns

#### Server Creation

```cpp
// Server must be created via factory method
auto server = Server::create(config, handler);

// Never create on stack or with make_shared (won't compile):
// Server server(config, handler);  // Compiler error - constructor private
// auto server = std::make_shared<Server>(config, handler);  // Compiler error
```

#### Connection Creation (Server-Only)

Only Server can create connections (using private constructor via friend access)

### ConnectionHandler Implementation Patterns

#### Simple Synchronous Handler

```cpp
class HttpHandler : ConnectionHandler {
public:
  void on_data_arrived(std::string_view data, Connection& conn) override {
    // Parse HTTP request using connection's arena
    Arena& arena = conn.get_arena();

    // Generate response
    conn.append_message("HTTP/1.1 200 OK\r\n\r\nHello World");

    // Server retains ownership
  }
};
```

#### Async Handler with WeakRef

```cpp
class AsyncHandler : ConnectionHandler {
public:
  void on_data_arrived(std::string_view data, Connection& conn) override {
    // Get weak reference for async processing
    auto weak_conn = conn.get_weak_ref();

    work_queue.push([weak_conn, data = std::string(data)]() {
      // Process asynchronously - connection may be closed
      if (auto conn_ref = weak_conn.lock()) {
        conn_ref->append_message("Async response");
      }
    });
  }
};
```

#### Batching Handler with User Data

```cpp
class BatchingHandler : ConnectionHandler {
public:
  void on_connection_established(Connection &conn) override {
    // Allocate some protocol-specific data and attach it to the connection
    conn.user_data = new MyProtocolData();
  }

  void on_connection_closed(Connection &conn) override {
    // Free the protocol-specific data
    delete static_cast<MyProtocolData*>(conn.user_data);
  }

  void on_data_arrived(std::string_view data, Connection& conn) override {
    // Process data and maybe store some results in the user_data
    auto* proto_data = static_cast<MyProtocolData*>(conn.user_data);
    proto_data->process(data);
  }

  void on_batch_complete(std::span<Connection *const> batch) override {
    // Process a batch of connections
    for (auto* conn : batch) {
      if (conn) {
        auto* proto_data = static_cast<MyProtocolData*>(conn->user_data);
        if (proto_data->is_ready()) {
          // This connection is ready for the next stage, get weak ref for pipeline
          pipeline_.push(conn->get_weak_ref());
        }
      }
    }
  }

private:
  MyProcessingPipeline pipeline_;
};
```

#### Streaming "yes" Handler

```cpp
class YesHandler : ConnectionHandler {
public:
  void on_connection_established(Connection &conn) override {
    // Write an initial "y\n"
    conn.append_message("y\n");
  }

  void on_write_progress(Connection &conn) override {
    if (conn.outgoing_bytes_queued() == 0) {
      // Don't use an unbounded amount of memory
      conn.reset();
      // Write "y\n" repeatedly
      conn.append_message("y\n");
    }
  }
};
```

### Memory Management Patterns

#### Arena-Based String Handling

```cpp
// Preferred: String view with arena allocation to minimize copying
std::string_view process_json_key(const char* data, Arena& arena);

// Avoid: Unnecessary string copies
std::string process_json_key(const char* data);
```

#### Safe Async Connection Processing

```cpp
// In handler - get weak reference for background processing
auto weak_conn = conn.get_weak_ref();

// Process on worker thread
background_processor.submit([weak_conn]() {
  // Do work...
  if (auto conn_ref = weak_conn.lock()) {
    conn_ref->append_message("Background result");
  }
  // Connection automatically cleaned up by server
});
```

### Data Construction Patterns

#### Builder Pattern Usage

```cpp
CommitRequest request = CommitRequestBuilder(arena)
    .request_id("example-id")
    .leader_id("leader-123")
    .read_version(42)
    .build();
```

#### Error Handling Pattern

```cpp
enum class ParseResult { Success, InvalidJson, MissingField };
ParseResult parse_commit_request(const char* json, CommitRequest& out);
```

______________________________________________________________________

## Reference

### Build Targets

**Test Executables:**

- `test_arena` - Arena allocator functionality tests
- `test_commit_request` - JSON parsing and validation tests
- `test_metric` - Metrics system functionality tests
- Main server executable (compiled from `src/main.cpp`)

**Benchmark Executables:**

- `bench_arena` - Arena allocator performance benchmarks
- `bench_commit_request` - JSON parsing performance benchmarks
- `bench_parser_comparison` - Comparison benchmarks vs nlohmann::json and RapidJSON
- `bench_metric` - Metrics system performance benchmarks

**Debug Tools:**

- `debug_arena` - Debug tool for arena allocator analysis

### Performance Characteristics

**Memory Allocation:**

- **~1ns allocation time** vs standard allocators
- **Bulk deallocation** eliminates individual free() calls
- **Optimized geometric growth** uses current block size for doubling strategy
- **Alignment-aware** allocation prevents performance penalties

**JSON Parsing:**

- **Streaming parser** handles large payloads efficiently
- **Incremental processing** suitable for network protocols
- **Arena storage** eliminates string allocation overhead
- **SIMD-accelerated base64 decoding** using simdutf for maximum performance
- **Perfect hash table** provides O(1) JSON key lookup via gperf
- **Zero hash collisions** for known JSON tokens eliminates branching

### Build Notes

See [style.md](style.md) for comprehensive build configuration and CMake integration details.
