weaseldb/todo.md

WeaselDB Todo List

📋 Planned Tasks

Core Database Features

• Design commit pipeline architecture with three-stage processing
  • Stage 1: Version assignment and precondition validation thread
  • Stage 2: Transaction persistence and subscriber streaming thread
  • Stage 3: Connection return to server thread
  • Use ThreadPipeline for inter-stage communication
  • Design persistence interface for pluggable storage backends (S3, local disk)
• Integrate https://git.weaselab.dev/weaselab/conflict-set for optimistic concurrency control
• Design and architect the subscription component for change streams

API Endpoints Implementation

• Implement GET /v1/version endpoint to return latest committed version and leader
• Implement POST /v1/commit endpoint for transaction submission with precondition validation
• Implement GET /v1/status endpoint for commit request status lookup by request_id
• Implement GET /v1/subscribe endpoint for Server-Sent Events transaction streaming
• Implement PUT /v1/retention/<policy_id> endpoint for retention policy management
• Implement GET /v1/retention/<policy_id> endpoint for retention policy retrieval
• Implement GET /v1/retention/ endpoint for listing all retention policies
• Implement DELETE /v1/retention/<policy_id> endpoint for retention policy removal

Infrastructure & Tooling

• Implement thread-safe Prometheus metrics library and serve GET /metrics endpoint
• Implement gperf-based HTTP routing for efficient request dispatching
• Implement HTTP client for S3 interactions
• Implement fake in-process S3 service using separate Server instance with S3 ConnectionHandler
  • Use createLocalConnection to get fd for in-process communication
  • Implement ListObjectsV2 API for object enumeration
  • Implement PutObject with chunked encoding support for streaming uploads
  • Add If-None-Match conditional header handling for PutObject
  • Implement GetObject for object retrieval
  • Add byte range support for GetObject (Range header handling)
  • Implement DeleteObjects for batch object deletion

Client Libraries

• Implement high-level Python client library for WeaselDB REST API
  • Wrap /v1/version, /v1/commit, /v1/status endpoints
  • Handle /v1/subscribe SSE streaming with reconnection logic
  • Provide idiomatic error handling and retry logic
• Implement Python administrative/operator library for WeaselDB management
  • Support retention policy management (/v1/retention/* endpoints)
  • Include metrics querying and monitoring tools
  • Provide CLI tooling for database administration

Testing & Validation

• Build out-of-process API test suite using client library over real TCP
  • Test all /v1/version, /v1/commit, /v1/status endpoints
  • Test /v1/subscribe Server-Sent Events streaming
  • Test retention policy endpoints (/v1/retention/*)
  • Test /metrics Prometheus endpoint
  • Test error conditions and edge cases
  • Test concurrent request handling and threading model

✅ Completed Tasks

Most recent completions at the top

• Built streaming JSON parser for commit requests with high-performance parsing
• Implemented HTTP server with multi-threaded networking using multiple epoll instances
• Created threading model with pipeline for serial request processing for optimistic concurrency control
• Designed connection ownership transfer system to enable the serial processing model
• Implemented arena-per-connection memory model for clean memory lifetime management
• Built TOML configuration system for server settings