85 lines
5.0 KiB
Markdown
85 lines
5.0 KiB
Markdown
# WeaselDB Todo List
|
|
|
|
## 📋 Planned Tasks
|
|
|
|
### Core Database Features
|
|
- [ ] Design commit pipeline architecture with three-stage processing
|
|
- [ ] Stage 1: Version assignment and precondition validation thread
|
|
- [ ] Stage 2: Transaction persistence and subscriber streaming thread
|
|
- [ ] Stage 3: Connection return to server thread
|
|
- [ ] Use ThreadPipeline for inter-stage communication
|
|
- [ ] Design persistence interface for pluggable storage backends (S3, local disk)
|
|
- [ ] Integrate https://git.weaselab.dev/weaselab/conflict-set for optimistic concurrency control
|
|
- [ ] Design and architect the subscription component for change streams
|
|
|
|
### API Endpoints Implementation
|
|
- [ ] Implement `GET /v1/version` endpoint to return latest committed version and leader
|
|
- [ ] Implement `POST /v1/commit` endpoint for transaction submission with precondition validation
|
|
- [ ] Implement `GET /v1/status` endpoint for commit request status lookup by request_id
|
|
- [ ] Implement `GET /v1/subscribe` endpoint for Server-Sent Events transaction streaming
|
|
- [ ] Implement `PUT /v1/retention/<policy_id>` endpoint for retention policy management
|
|
- [ ] Implement `GET /v1/retention/<policy_id>` endpoint for retention policy retrieval
|
|
- [ ] Implement `GET /v1/retention/` endpoint for listing all retention policies
|
|
- [ ] Implement `DELETE /v1/retention/<policy_id>` endpoint for retention policy removal
|
|
|
|
### Infrastructure & Tooling
|
|
- [x] Implement thread-safe Prometheus metrics library and serve `GET /metrics` endpoint
|
|
- [ ] Implement gperf-based HTTP routing for efficient request dispatching
|
|
- [ ] Replace nlohmann/json with simdjson DOM API in parser comparison benchmarks
|
|
- [ ] Implement HTTP client for S3 interactions
|
|
- [ ] Design `HttpClient` class following WeaselDB patterns (factory creation, arena allocation, RAII)
|
|
- [ ] Implement connection pool with configurable limits (max connections, idle timeout)
|
|
- [ ] Support HTTP/1.1 with Keep-Alive for connection reuse
|
|
- [ ] Use epoll-based async I/O model similar to Server architecture
|
|
- [ ] Implement request pipeline: `HttpRequest` -> `HttpResponse` with arena-backed memory
|
|
- [ ] Support streaming request bodies (chunked encoding) for large S3 uploads
|
|
- [ ] Support streaming response bodies with callback-based data consumption
|
|
- [ ] Add timeout handling (connect timeout, read timeout, total request timeout)
|
|
- [ ] Implement retry logic with exponential backoff and jitter
|
|
- [ ] Support custom headers and authentication (AWS Signature V4 for S3)
|
|
- [ ] Single-threaded event-driven design for integration with persistence pipeline
|
|
- [ ] Integration with WeaselDB's ThreadPipeline for request batching
|
|
- [ ] Comprehensive error handling with detailed error codes and descriptions
|
|
- [ ] Support for HTTP redirects (3xx responses) with redirect limits
|
|
- [ ] SSL/TLS support using OpenSSL for HTTPS connections
|
|
- [ ] Request/response logging and metrics integration
|
|
- [ ] Memory-efficient design minimizing unnecessary copying
|
|
- [ ] Implement fake in-process S3 service using separate Server instance with S3 ConnectionHandler
|
|
- [ ] Use create_local_connection to get fd for in-process communication
|
|
- [ ] Implement `ListObjectsV2` API for object enumeration
|
|
- [ ] Implement `PutObject` with chunked encoding support for streaming uploads
|
|
- [ ] Add `If-None-Match` conditional header handling for `PutObject`
|
|
- [ ] Implement `GetObject` for object retrieval
|
|
- [ ] Add byte range support for `GetObject` (Range header handling)
|
|
- [ ] Implement `DeleteObjects` for batch object deletion
|
|
|
|
### Client Libraries
|
|
- [ ] Implement high-level Python client library for WeaselDB REST API
|
|
- [ ] Wrap `/v1/version`, `/v1/commit`, `/v1/status` endpoints
|
|
- [ ] Handle `/v1/subscribe` SSE streaming with reconnection logic
|
|
- [ ] Provide idiomatic error handling and retry logic
|
|
- [ ] Implement Python administrative/operator library for WeaselDB management
|
|
- [ ] Support retention policy management (`/v1/retention/*` endpoints)
|
|
- [ ] Include metrics querying and monitoring tools
|
|
- [ ] Provide CLI tooling for database administration
|
|
|
|
### Testing & Validation
|
|
- [ ] Build out-of-process API test suite using client library over real TCP
|
|
- [ ] Test all `/v1/version`, `/v1/commit`, `/v1/status` endpoints
|
|
- [ ] Test `/v1/subscribe` Server-Sent Events streaming
|
|
- [ ] Test retention policy endpoints (`/v1/retention/*`)
|
|
- [ ] Test `/metrics` Prometheus endpoint
|
|
- [ ] Test error conditions and edge cases
|
|
- [ ] Test single-threaded event-driven processing model
|
|
|
|
## ✅ Completed Tasks
|
|
|
|
*Most recent completions at the top*
|
|
|
|
- [x] Built streaming JSON parser for commit requests with high-performance parsing
|
|
- [x] Implemented HTTP server with multi-threaded networking using multiple epoll instances
|
|
- [x] Created threading model with pipeline for serial request processing for optimistic concurrency control
|
|
- [x] Designed connection ownership transfer system to enable the serial processing model
|
|
- [x] Implemented arena-per-connection memory model for clean memory lifetime management
|
|
- [x] Built TOML configuration system for server settings
|