weaseldb/todo.md

WeaselDB Todo List

📋 Planned Tasks

Core Database Features

• Design commit pipeline architecture with three-stage processing
  • Stage 1: Version assignment and precondition validation thread
  • Stage 2: Transaction persistence and subscriber streaming thread
  • Stage 3: Connection return to server thread
  • Use ThreadPipeline for inter-stage communication
  • Design persistence interface for pluggable storage backends (S3, local disk)
• Integrate https://git.weaselab.dev/weaselab/conflict-set for optimistic concurrency control
• Design and architect the subscription component for change streams

API Endpoints Implementation

• Implement GET /v1/version endpoint to return latest committed version and leader
• Implement POST /v1/commit endpoint for transaction submission with precondition validation
• Implement GET /v1/status endpoint for commit request status lookup by request_id
• Implement GET /v1/subscribe endpoint for Server-Sent Events transaction streaming
• Implement PUT /v1/retention/<policy_id> endpoint for retention policy management
• Implement GET /v1/retention/<policy_id> endpoint for retention policy retrieval
• Implement GET /v1/retention/ endpoint for listing all retention policies
• Implement DELETE /v1/retention/<policy_id> endpoint for retention policy removal

Infrastructure & Tooling

• Implement thread-safe Prometheus metrics library and serve GET /metrics endpoint
• Implement gperf-based HTTP routing for efficient request dispatching
• Implement HTTP client for S3 interactions
  • Design HttpClient class following WeaselDB patterns (factory creation, arena allocation, RAII)
  • Implement connection pool with configurable limits (max connections, idle timeout)
  • Support HTTP/1.1 with Keep-Alive for connection reuse
  • Use epoll-based async I/O model similar to Server architecture
  • Implement request pipeline: HttpRequest -> HttpResponse with arena-backed memory
  • Support streaming request bodies (chunked encoding) for large S3 uploads
  • Support streaming response bodies with callback-based data consumption
  • Add timeout handling (connect timeout, read timeout, total request timeout)
  • Implement retry logic with exponential backoff and jitter
  • Support custom headers and authentication (AWS Signature V4 for S3)
  • Single-threaded event-driven design for integration with persistence pipeline
  • Integration with WeaselDB's ThreadPipeline for request batching
  • Comprehensive error handling with detailed error codes and descriptions
  • Support for HTTP redirects (3xx responses) with redirect limits
  • SSL/TLS support using OpenSSL for HTTPS connections
  • Request/response logging and metrics integration
  • Memory-efficient design with zero-copy where possible
• Implement fake in-process S3 service using separate Server instance with S3 ConnectionHandler
  • Use create_local_connection to get fd for in-process communication
  • Implement ListObjectsV2 API for object enumeration
  • Implement PutObject with chunked encoding support for streaming uploads
  • Add If-None-Match conditional header handling for PutObject
  • Implement GetObject for object retrieval
  • Add byte range support for GetObject (Range header handling)
  • Implement DeleteObjects for batch object deletion

Client Libraries

• Implement high-level Python client library for WeaselDB REST API
  • Wrap /v1/version, /v1/commit, /v1/status endpoints
  • Handle /v1/subscribe SSE streaming with reconnection logic
  • Provide idiomatic error handling and retry logic
• Implement Python administrative/operator library for WeaselDB management
  • Support retention policy management (/v1/retention/* endpoints)
  • Include metrics querying and monitoring tools
  • Provide CLI tooling for database administration

Testing & Validation

• Build out-of-process API test suite using client library over real TCP
  • Test all /v1/version, /v1/commit, /v1/status endpoints
  • Test /v1/subscribe Server-Sent Events streaming
  • Test retention policy endpoints (/v1/retention/*)
  • Test /metrics Prometheus endpoint
  • Test error conditions and edge cases
  • Test single-threaded event-driven processing model

✅ Completed Tasks

Most recent completions at the top

• Built streaming JSON parser for commit requests with high-performance parsing
• Implemented HTTP server with multi-threaded networking using multiple epoll instances
• Created threading model with pipeline for serial request processing for optimistic concurrency control
• Designed connection ownership transfer system to enable the serial processing model
• Implemented arena-per-connection memory model for clean memory lifetime management
• Built TOML configuration system for server settings