weaseldb/persistence.md

Persistence Thread Design

Overview

The persistence thread receives commit batches from the main processing pipeline and uploads them to S3. It uses a single-threaded design with connection pooling and batching for optimal performance.

Architecture

Input: Commits arrive via ThreadPipeline interface from upstream processing Output: Batched commits uploaded to S3 persistence backend Transport: Single-threaded TCP client with connection pooling Protocol: Higher layers handle HTTP, authentication, and S3-specific details

Batching Strategy

The persistence thread collects commits into batches using two trigger conditions:

1. Time Trigger: batch_timeout_ms elapsed since batch collection started
2. Size Trigger: batch_size_threshold commits collected (can be exceeded by final commit)

Flow Control: When max_in_flight_requests reached, block until responses received.

Main Processing Loop

1. Batch Collection

No In-Flight Requests:

• Use blocking acquire to get first commit batch
• Process immediately (no batching delay)

With In-Flight Requests:

• Check flow control: if at max_in_flight_requests, block for responses
• Collect commits using non-blocking acquire until trigger condition:
  • Check for available commits (non-blocking)
  • If batch_size_threshold reached → process batch immediately
  • If below threshold → use epoll_wait(batch_timeout_ms) for I/O and timeout
  • On timeout → process collected commits
• If no commits available and no in-flight requests → switch to blocking acquire

2. Connection Management

• Acquire healthy connection from pool
• Create new connections if pool below target_pool_size
• If no healthy connections available, block until one becomes available
• Maintain automatic pool replenishment

3. Data Transmission

• Write batch data to S3 connection using appropriate protocol
• Publish accepted transactions to subscriber system
• Track request as in-flight for flow control

4. I/O Event Processing

• Handle epoll events for all in-flight connections
• Monitor connection health via heartbeats
• Process incoming responses and detect connection failures

5. Response Handling

• Ordered Acknowledgment: Only acknowledge batch after all prior batches are durable
• Release batch via StageGuard destructor (publishes to next pipeline stage)
• Publish durability events to subscriber system
• Return healthy connection to pool

6. Failure Handling

• Remove failed connection from pool
• Retry batch with exponential backoff (up to max_retry_attempts)
• Backoff delays only affect the specific failing batch
• If retries exhausted, abort process or escalate error
• Initiate pool replenishment if below target

Connection Pool

Target Size: target_pool_size connections (recommended: 2x max_in_flight_requests) Replenishment: Automatic creation when below target Health Monitoring: Heartbeat-based connection validation Sizing Rationale: 2x multiplier ensures availability during peak load and connection replacement

Key Design Properties

Batch Ordering: Batches may be retried out-of-order for performance, but acknowledgment to next pipeline stage maintains strict ordering.

Backpressure: Retry delays for failing batches create natural backpressure that eventually blocks the persistence thread when in-flight limits are reached.

Graceful Shutdown: On shutdown signal, drain all in-flight batches to completion before terminating.

Configuration Parameters

Parameter	Default	Description
batch_timeout_ms	1ms	Maximum time to wait collecting commits for batching
batch_size_threshold	1MB	Threshold for triggering batch processing
max_in_flight_requests	50	Maximum concurrent requests to persistence backend
target_pool_size	2x in-flight	Target number of connections to maintain
max_retry_attempts	3	Maximum retries for failed batches before aborting
retry_base_delay_ms	100ms	Base delay for exponential backoff retries

Configuration Validation

Required Constraints:

• batch_size_threshold > 0 (must process at least one commit per batch)
• max_in_flight_requests > 0 (must allow at least one concurrent request)
• max_in_flight_requests <= 1000 (required for single-call recovery guarantee)
• target_pool_size >= max_in_flight_requests (pool must accommodate all in-flight requests)
• batch_timeout_ms > 0 (timeout must be positive)
• max_retry_attempts >= 0 (zero disables retries)
• retry_base_delay_ms > 0 (delay must be positive if retries enabled)

Performance Recommendations:

• target_pool_size <= 2x max_in_flight_requests (optimal for performance)

Recovery and Consistency

Recovery Model

WeaselDB's batched persistence design enables efficient recovery while maintaining strict serializable consistency guarantees.

Batch Ordering and Durability

Ordered Acknowledgment Property: Batches may be retried out-of-order for performance, but acknowledgment to the next pipeline stage maintains strict ordering. This ensures that if batch N is acknowledged as durable, all earlier batches (higher numbers N+1, N+2, etc.) are also guaranteed durable.

Durability Watermark: The system maintains a durable watermark indicating the latest consecutively durable batch (lowest batch number in the consecutive sequence). This watermark advances only when all preceding batches (higher numbers) are confirmed persistent.

Recovery Protocol

WeaselDB uses a sequential batch numbering scheme with S3 atomic operations to provide efficient crash recovery and split-brain prevention without external coordination services.

Batch Numbering Scheme:

• Batch numbers start at 2^64 - 1 and count downward: 18446744073709551615, 18446744073709551614, 18446744073709551613, ...
• Each batch is stored as S3 object batches/{batch_number:020d} with zero-padding
• S3 lexicographic ordering ensures recent batches (higher numbers) appear first in LIST operations

Leadership and Split-Brain Prevention:

• New persistence thread instances scan S3 to find the next available batch number
• Each batch write uses If-None-Match="*" to atomically claim the sequential batch number
• Only one instance can successfully claim each batch number, preventing split-brain scenarios
• Batch object content includes leader_id to identify which leader wrote each batch

Recovery Scenarios:

Clean Shutdown:

• All in-flight batches are drained to completion before termination
• Durability watermark accurately reflects all durable state
• No recovery required on restart

Crash Recovery:

1. S3 Scan with Bounded Cost: List S3 objects with prefix batches/ and limit of 1000 objects
2. Gap Detection: Check for missing sequential batch numbers. WeaselDB never puts more than 1000 batches in flight concurrently, so a limit of 1000 is sufficient.
3. Watermark Reconstruction: Set durability watermark to the latest consecutive batch (lowest batch number in consecutive sequence)
4. Leadership Transition: Begin writing batches starting from next available batch number. Skip past any batch numbers claimed in the durability watermark scan.

Bounded Recovery Guarantee: Since at most 1000 batches can be in-flight during a crash, the durability watermark is guaranteed to be found within the first 1000 objects in S3. This ensures O(1) recovery time regardless of database size, with at most one S3 LIST operation required.

Recovery Protocol Detail: Even with exactly 1000 batches in-flight, recovery works correctly:

• Worst case: earliest batch (highest number) fails while remaining 999 batches succeed
• S3 LIST returns 1000 objects: the 999 successful batches plus one previously written batch
• Gap detection identifies the missing batch and sets watermark to latest consecutive batch (lowest number in sequence)
• Since batches count downward with zero-padded names, lexicographic ordering ensures proper sequence detection

Recovery Performance Limits: To maintain single-call recovery guarantees, max_in_flight_requests is limited to 1000, matching S3's maximum objects per LIST operation. This ensures a single S3 API call is sufficient for recovery.