# WeaselDB Todo List

## 📋 Planned Tasks

### Core Database Features
- [ ] Design commit pipeline architecture with three-stage processing
  - [ ] Stage 1: Version assignment and precondition validation thread
  - [ ] Stage 2: Transaction persistence and subscriber streaming thread
  - [ ] Stage 3: Connection return to server thread
  - [ ] Use ThreadPipeline for inter-stage communication
  - [ ] Design persistence interface for pluggable storage backends (S3, local disk)
- [ ] Integrate https://git.weaselab.dev/weaselab/conflict-set for optimistic concurrency control
- [ ] Design and architect the subscription component for change streams

### API Endpoints Implementation
- [ ] Implement `GET /v1/version` endpoint to return latest committed version and leader
- [ ] Implement `POST /v1/commit` endpoint for transaction submission with precondition validation
- [ ] Implement `GET /v1/status` endpoint for commit request status lookup by request_id
- [ ] Implement `GET /v1/subscribe` endpoint for Server-Sent Events transaction streaming
- [ ] Implement `PUT /v1/retention/<policy_id>` endpoint for retention policy management
- [ ] Implement `GET /v1/retention/<policy_id>` endpoint for retention policy retrieval
- [ ] Implement `GET /v1/retention/` endpoint for listing all retention policies
- [ ] Implement `DELETE /v1/retention/<policy_id>` endpoint for retention policy removal

### Infrastructure & Tooling
- [ ] Implement thread-safe Prometheus metrics library and serve `GET /metrics` endpoint
- [ ] Implement gperf-based HTTP routing for efficient request dispatching
- [ ] Implement HTTP client for S3 interactions
  - [ ] Design `HttpClient` class following WeaselDB patterns (factory creation, arena allocation, RAII)
  - [ ] Implement connection pool with configurable limits (max connections, idle timeout)
  - [ ] Support HTTP/1.1 with Keep-Alive for connection reuse
  - [ ] Use epoll-based async I/O model similar to Server architecture
  - [ ] Implement request pipeline: `HttpRequest` -> `HttpResponse` with arena-backed memory
  - [ ] Support streaming request bodies (chunked encoding) for large S3 uploads
  - [ ] Support streaming response bodies with callback-based data consumption
  - [ ] Add timeout handling (connect timeout, read timeout, total request timeout)
  - [ ] Implement retry logic with exponential backoff and jitter
  - [ ] Support custom headers and authentication (AWS Signature V4 for S3)
  - [ ] Single-threaded event-driven design for integration with persistence pipeline
  - [ ] Integration with WeaselDB's ThreadPipeline for request batching
  - [ ] Comprehensive error handling with detailed error codes and descriptions
  - [ ] Support for HTTP redirects (3xx responses) with redirect limits
  - [ ] SSL/TLS support using OpenSSL for HTTPS connections
  - [ ] Request/response logging and metrics integration
  - [ ] Memory-efficient design minimizing unnecessary copying
- [ ] Implement fake in-process S3 service using separate Server instance with S3 ConnectionHandler
  - [ ] Use create_local_connection to get fd for in-process communication
  - [ ] Implement `ListObjectsV2` API for object enumeration
  - [ ] Implement `PutObject` with chunked encoding support for streaming uploads
  - [ ] Add `If-None-Match` conditional header handling for `PutObject`
  - [ ] Implement `GetObject` for object retrieval
  - [ ] Add byte range support for `GetObject` (Range header handling)
  - [ ] Implement `DeleteObjects` for batch object deletion

### Client Libraries
- [ ] Implement high-level Python client library for WeaselDB REST API
  - [ ] Wrap `/v1/version`, `/v1/commit`, `/v1/status` endpoints
  - [ ] Handle `/v1/subscribe` SSE streaming with reconnection logic
  - [ ] Provide idiomatic error handling and retry logic
- [ ] Implement Python administrative/operator library for WeaselDB management
  - [ ] Support retention policy management (`/v1/retention/*` endpoints)
  - [ ] Include metrics querying and monitoring tools
  - [ ] Provide CLI tooling for database administration

### Testing & Validation
- [ ] Build out-of-process API test suite using client library over real TCP
  - [ ] Test all `/v1/version`, `/v1/commit`, `/v1/status` endpoints
  - [ ] Test `/v1/subscribe` Server-Sent Events streaming
  - [ ] Test retention policy endpoints (`/v1/retention/*`)
  - [ ] Test `/metrics` Prometheus endpoint
  - [ ] Test error conditions and edge cases
  - [ ] Test single-threaded event-driven processing model

## ✅ Completed Tasks

*Most recent completions at the top*

- [x] Built streaming JSON parser for commit requests with high-performance parsing
- [x] Implemented HTTP server with multi-threaded networking using multiple epoll instances
- [x] Created threading model with pipeline for serial request processing for optimistic concurrency control
- [x] Designed connection ownership transfer system to enable the serial processing model
- [x] Implemented arena-per-connection memory model for clean memory lifetime management
- [x] Built TOML configuration system for server settings