From 7db0e331e4c86599f73bba3c983fee72a0a52fd3 Mon Sep 17 00:00:00 2001
From: Andrew Noyes <andrew@weaselab.dev>
Date: Sat, 23 Aug 2025 18:44:50 -0400
Subject: [PATCH] Add todo.md

---
 todo.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100644 todo.md

diff --git a/todo.md b/todo.md
new file mode 100644
index 0000000..51d363e
--- /dev/null
+++ b/todo.md
@@ -0,0 +1,66 @@
+# WeaselDB Todo List
+
+## 📋 Planned Tasks
+
+### Core Database Features
+- [ ] Design commit pipeline architecture with three-stage processing
+  - [ ] Stage 1: Version assignment and precondition validation thread
+  - [ ] Stage 2: Transaction persistence and subscriber streaming thread
+  - [ ] Stage 3: Connection return to server thread
+  - [ ] Use ThreadPipeline for inter-stage communication
+  - [ ] Design persistence interface for pluggable storage backends (S3, local disk)
+- [ ] Integrate https://git.weaselab.dev/weaselab/conflict-set for optimistic concurrency control
+- [ ] Design and architect the subscription component for change streams
+
+### API Endpoints Implementation
+- [ ] Implement `GET /v1/version` endpoint to return latest committed version and leader
+- [ ] Implement `POST /v1/commit` endpoint for transaction submission with precondition validation
+- [ ] Implement `GET /v1/status` endpoint for commit request status lookup by request_id
+- [ ] Implement `GET /v1/subscribe` endpoint for Server-Sent Events transaction streaming
+- [ ] Implement `PUT /v1/retention/<policy_id>` endpoint for retention policy management
+- [ ] Implement `GET /v1/retention/<policy_id>` endpoint for retention policy retrieval
+- [ ] Implement `GET /v1/retention/` endpoint for listing all retention policies
+- [ ] Implement `DELETE /v1/retention/<policy_id>` endpoint for retention policy removal
+
+### Infrastructure & Tooling
+- [ ] Implement thread-safe Prometheus metrics library and serve `GET /metrics` endpoint
+- [ ] Implement gperf-based HTTP routing for efficient request dispatching
+- [ ] Implement HTTP client for S3 interactions
+- [ ] Implement fake in-process S3 service using separate Server instance with S3 ConnectionHandler
+  - [ ] Use createLocalConnection to get fd for in-process communication
+  - [ ] Implement `ListObjectsV2` API for object enumeration
+  - [ ] Implement `PutObject` with chunked encoding support for streaming uploads
+  - [ ] Add `If-None-Match` conditional header handling for `PutObject`
+  - [ ] Implement `GetObject` for object retrieval
+  - [ ] Add byte range support for `GetObject` (Range header handling)
+  - [ ] Implement `DeleteObjects` for batch object deletion
+
+### Client Libraries
+- [ ] Implement high-level Python client library for WeaselDB REST API
+  - [ ] Wrap `/v1/version`, `/v1/commit`, `/v1/status` endpoints
+  - [ ] Handle `/v1/subscribe` SSE streaming with reconnection logic
+  - [ ] Provide idiomatic error handling and retry logic
+- [ ] Implement Python administrative/operator library for WeaselDB management
+  - [ ] Support retention policy management (`/v1/retention/*` endpoints)
+  - [ ] Include metrics querying and monitoring tools
+  - [ ] Provide CLI tooling for database administration
+
+### Testing & Validation
+- [ ] Build out-of-process API test suite using client library over real TCP
+  - [ ] Test all `/v1/version`, `/v1/commit`, `/v1/status` endpoints
+  - [ ] Test `/v1/subscribe` Server-Sent Events streaming
+  - [ ] Test retention policy endpoints (`/v1/retention/*`)
+  - [ ] Test `/metrics` Prometheus endpoint
+  - [ ] Test error conditions and edge cases
+  - [ ] Test concurrent request handling and threading model
+
+## ✅ Completed Tasks
+
+*Most recent completions at the top*
+
+- [x] Built streaming JSON parser for commit requests with high-performance parsing
+- [x] Implemented HTTP server with multi-threaded networking using multiple epoll instances
+- [x] Created threading model with pipeline for serial request processing for optimistic concurrency control
+- [x] Designed connection ownership transfer system to enable the serial processing model
+- [x] Implemented arena-per-connection memory model for clean memory lifetime management
+- [x] Built TOML configuration system for server settings