67 lines
3.7 KiB
Markdown
67 lines
3.7 KiB
Markdown
# WeaselDB Todo List
|
|
|
|
## 📋 Planned Tasks
|
|
|
|
### Core Database Features
|
|
- [ ] Design commit pipeline architecture with three-stage processing
|
|
- [ ] Stage 1: Version assignment and precondition validation thread
|
|
- [ ] Stage 2: Transaction persistence and subscriber streaming thread
|
|
- [ ] Stage 3: Connection return to server thread
|
|
- [ ] Use ThreadPipeline for inter-stage communication
|
|
- [ ] Design persistence interface for pluggable storage backends (S3, local disk)
|
|
- [ ] Integrate https://git.weaselab.dev/weaselab/conflict-set for optimistic concurrency control
|
|
- [ ] Design and architect the subscription component for change streams
|
|
|
|
### API Endpoints Implementation
|
|
- [ ] Implement `GET /v1/version` endpoint to return latest committed version and leader
|
|
- [ ] Implement `POST /v1/commit` endpoint for transaction submission with precondition validation
|
|
- [ ] Implement `GET /v1/status` endpoint for commit request status lookup by request_id
|
|
- [ ] Implement `GET /v1/subscribe` endpoint for Server-Sent Events transaction streaming
|
|
- [ ] Implement `PUT /v1/retention/<policy_id>` endpoint for retention policy management
|
|
- [ ] Implement `GET /v1/retention/<policy_id>` endpoint for retention policy retrieval
|
|
- [ ] Implement `GET /v1/retention/` endpoint for listing all retention policies
|
|
- [ ] Implement `DELETE /v1/retention/<policy_id>` endpoint for retention policy removal
|
|
|
|
### Infrastructure & Tooling
|
|
- [ ] Implement thread-safe Prometheus metrics library and serve `GET /metrics` endpoint
|
|
- [ ] Implement gperf-based HTTP routing for efficient request dispatching
|
|
- [ ] Implement HTTP client for S3 interactions
|
|
- [ ] Implement fake in-process S3 service using separate Server instance with S3 ConnectionHandler
|
|
- [ ] Use createLocalConnection to get fd for in-process communication
|
|
- [ ] Implement `ListObjectsV2` API for object enumeration
|
|
- [ ] Implement `PutObject` with chunked encoding support for streaming uploads
|
|
- [ ] Add `If-None-Match` conditional header handling for `PutObject`
|
|
- [ ] Implement `GetObject` for object retrieval
|
|
- [ ] Add byte range support for `GetObject` (Range header handling)
|
|
- [ ] Implement `DeleteObjects` for batch object deletion
|
|
|
|
### Client Libraries
|
|
- [ ] Implement high-level Python client library for WeaselDB REST API
|
|
- [ ] Wrap `/v1/version`, `/v1/commit`, `/v1/status` endpoints
|
|
- [ ] Handle `/v1/subscribe` SSE streaming with reconnection logic
|
|
- [ ] Provide idiomatic error handling and retry logic
|
|
- [ ] Implement Python administrative/operator library for WeaselDB management
|
|
- [ ] Support retention policy management (`/v1/retention/*` endpoints)
|
|
- [ ] Include metrics querying and monitoring tools
|
|
- [ ] Provide CLI tooling for database administration
|
|
|
|
### Testing & Validation
|
|
- [ ] Build out-of-process API test suite using client library over real TCP
|
|
- [ ] Test all `/v1/version`, `/v1/commit`, `/v1/status` endpoints
|
|
- [ ] Test `/v1/subscribe` Server-Sent Events streaming
|
|
- [ ] Test retention policy endpoints (`/v1/retention/*`)
|
|
- [ ] Test `/metrics` Prometheus endpoint
|
|
- [ ] Test error conditions and edge cases
|
|
- [ ] Test concurrent request handling and threading model
|
|
|
|
## ✅ Completed Tasks
|
|
|
|
*Most recent completions at the top*
|
|
|
|
- [x] Built streaming JSON parser for commit requests with high-performance parsing
|
|
- [x] Implemented HTTP server with multi-threaded networking using multiple epoll instances
|
|
- [x] Created threading model with pipeline for serial request processing for optimistic concurrency control
|
|
- [x] Designed connection ownership transfer system to enable the serial processing model
|
|
- [x] Implemented arena-per-connection memory model for clean memory lifetime management
|
|
- [x] Built TOML configuration system for server settings
|