diff --git a/persistence.md b/persistence.md index 6d0ca66..871a7fe 100644 --- a/persistence.md +++ b/persistence.md @@ -102,7 +102,7 @@ The persistence thread collects commits into batches using two trigger condition **Required Constraints**: - `batch_size_threshold` > 0 (must process at least one commit per batch) - `max_in_flight_requests` > 0 (must allow at least one concurrent request) -- `max_in_flight_requests` < 1000 (required for single-call recovery guarantee) +- `max_in_flight_requests` <= 1000 (required for single-call recovery guarantee) - `target_pool_size` >= `max_in_flight_requests` (pool must accommodate all in-flight requests) - `batch_timeout_ms` > 0 (timeout must be positive) - `max_retry_attempts` >= 0 (zero disables retries) @@ -119,9 +119,9 @@ WeaselDB's batched persistence design enables efficient recovery while maintaini #### **Batch Ordering and Durability** -**Ordered Acknowledgment Property**: Batches may be retried out-of-order for performance, but acknowledgment to the next pipeline stage maintains strict ordering. This ensures that if batch N is acknowledged as durable, all batches 0 through N-1 are also guaranteed durable. +**Ordered Acknowledgment Property**: Batches may be retried out-of-order for performance, but acknowledgment to the next pipeline stage maintains strict ordering. This ensures that if batch N is acknowledged as durable, all earlier batches (higher numbers N+1, N+2, etc.) are also guaranteed durable. -**Durability Watermark**: The system maintains a durable watermark indicating the highest consecutively durable batch ID. This watermark advances only when all preceding batches are confirmed persistent. +**Durability Watermark**: The system maintains a durable watermark indicating the latest consecutively durable batch (lowest batch number in the consecutive sequence). This watermark advances only when all preceding batches (higher numbers) are confirmed persistent. #### **Recovery Protocol** @@ -147,10 +147,16 @@ WeaselDB uses a **sequential batch numbering** scheme with **S3 atomic operation **Crash Recovery**: 1. **S3 Scan with Bounded Cost**: List S3 objects with prefix `batches/` and limit of 1000 objects -2. **Gap Detection**: Check for missing sequential batch numbers. WeaselDB never puts 1000 batches in flight concurrently, so a limit of 1000 is sufficient. -3. **Watermark Reconstruction**: Set durability watermark to highest consecutive batch number found +2. **Gap Detection**: Check for missing sequential batch numbers. WeaselDB never puts more than 1000 batches in flight concurrently, so a limit of 1000 is sufficient. +3. **Watermark Reconstruction**: Set durability watermark to the latest consecutive batch (lowest batch number in consecutive sequence) 4. **Leadership Transition**: Begin writing batches starting from next available batch number. Skip past any batch numbers claimed in the durability watermark scan. -**Bounded Recovery Guarantee**: Since at most 999 batches can be in-flight during a crash, the durability watermark is guaranteed to be found within the first 1000 objects in S3. This ensures **O(1) recovery time** regardless of database size, with at most **one S3 LIST operation** required. +**Bounded Recovery Guarantee**: Since at most 1000 batches can be in-flight during a crash, the durability watermark is guaranteed to be found within the first 1000 objects in S3. This ensures **O(1) recovery time** regardless of database size, with at most **one S3 LIST operation** required. -**Recovery Performance Limits**: To maintain single-call recovery guarantees, `max_in_flight_requests` is limited to **1000**, matching S3's maximum objects per LIST operation. This ensures recovery a single S3 API call is sufficient for recovery. +**Recovery Protocol Detail**: Even with exactly 1000 batches in-flight, recovery works correctly: +- Worst case: earliest batch (highest number) fails while remaining 999 batches succeed +- S3 LIST returns 1000 objects: the 999 successful batches plus one previously written batch +- Gap detection identifies the missing batch and sets watermark to latest consecutive batch (lowest number in sequence) +- Since batches count downward with zero-padded names, lexicographic ordering ensures proper sequence detection + +**Recovery Performance Limits**: To maintain single-call recovery guarantees, `max_in_flight_requests` is limited to **1000**, matching S3's maximum objects per LIST operation. This ensures a single S3 API call is sufficient for recovery.