weaseldb/config.md

WeaselDB Configuration

WeaselDB uses a TOML configuration file to control server behavior and API limits. The configuration is organized into three main sections that correspond to different aspects of the system.

Configuration File Location

By default, WeaselDB looks for config.toml in the current directory. You can specify an alternative path:

./weaseldb /path/to/custom/config.toml

Configuration Sections

Server Configuration ([server])

Controls server networking, threading, and request handling behavior.

Parameter	Type	Default	Description
interfaces	array of objects	TCP on 127.0.0.1:8080	Network interfaces to listen on. Each interface can be TCP or Unix socket
max_request_size_bytes	integer	1048576 (1MB)	Maximum size for incoming requests. Requests exceeding this limit receive a 413 Content Too Large response
io_threads	integer	1	Number of I/O threads for handling connections and network events
epoll_instances	integer	io_threads	Number of epoll instances to reduce kernel contention (max: io_threads). Lower values allow multiple threads per epoll for better load balancing, higher values reduce contention
event_batch_size	integer	32	Number of events to process in each epoll batch
max_connections	integer	50000	Maximum number of concurrent connections (0 = unlimited). Note: Due to race conditions between connection acceptance and cleanup, it's possible to trip this limit without actually having that many concurrent connections, especially under high connection churn.
read_buffer_size	integer	16384 (16KB)	Buffer size for reading from socket connections

Commit Configuration ([commit])

Controls behavior of the /v1/commit endpoint, request ID management, and commit pipeline threading.

Parameter	Type	Default	Description
min_request_id_length	integer	20	Minimum length required for client-provided request_id fields to ensure sufficient entropy for collision avoidance
request_id_retention_hours	integer	24	How long to retain request IDs in memory for /v1/status queries. Longer retention reduces the chance of log_truncated responses
request_id_retention_versions	integer	100000000	Minimum number of versions to retain request IDs for, regardless of time. Provides additional protection against log_truncated responses
pipeline_wait_strategy	string	"WaitIfUpstreamIdle"	Wait strategy for the commit pipeline. "WaitIfStageEmpty" = block when individual stages are empty (safe for shared CPUs), "WaitIfUpstreamIdle" = block only when all upstream stages are idle (requires dedicated cores, highest throughput), "Never" = never block, busy-wait continuously (requires dedicated cores, lowest latency)
pipeline_release_threads	integer	1	Number of threads in the release stage (final stage of commit pipeline). Higher values increase parallelism for connection release and response transmission

Subscription Configuration ([subscription])

Controls behavior of the /v1/subscribe endpoint and SSE streaming.

Parameter	Type	Default	Description
max_buffer_size_bytes	integer	10485760 (10MB)	Maximum amount of unconsumed data to buffer for slow subscribers. Connections are closed if this limit is exceeded
keepalive_interval_seconds	integer	30	Interval between keepalive comments in the Server-Sent Events stream to prevent idle timeouts on network proxies

Benchmark Configuration ([benchmark])

Controls benchmarking and health check behavior.

Parameter	Type	Default	Description
ok_resolve_iterations	integer	4000	CPU-intensive loop iterations for /ok requests in resolve stage. 0 = health check only, 4000 = default benchmark load (~650ns, 1M req/s)

Example Configuration

# WeaselDB Configuration File

[server]
# Network interfaces - can specify multiple TCP and/or Unix socket interfaces
interfaces = [
  { type = "tcp", address = "0.0.0.0", port = 8080 },
  # { type = "unix", path = "weaseldb.sock" },  # Alternative Unix socket
]

# Performance tuning
max_request_size_bytes = 2097152  # 2MB
io_threads = 8
epoll_instances = 8  # Reduce kernel contention (max: io_threads)
event_batch_size = 64
max_connections = 50000
read_buffer_size = 32768  # 32KB

[commit]
min_request_id_length = 32
request_id_retention_hours = 48
request_id_retention_versions = 50000
pipeline_wait_strategy = "WaitIfUpstreamIdle"  # Options: "WaitIfStageEmpty", "WaitIfUpstreamIdle", "Never"
pipeline_release_threads = 4  # Default: 1, increase for higher throughput

[subscription]
max_buffer_size_bytes = 52428800  # 50MB
keepalive_interval_seconds = 15

[benchmark]
ok_resolve_iterations = 10000  # Higher load for performance testing

Configuration Loading

WeaselDB uses the toml11 library for configuration parsing with robust error handling:

• File Loading: Uses ConfigParser::load_from_file(path) to parse TOML files
• Fallback Behavior: If the config file doesn't exist or contains errors, WeaselDB uses default values and logs a warning
• Optional Parameters: All configuration parameters are optional - missing values use the defaults shown above
• Type Safety: Configuration values are parsed with type validation and meaningful error messages
• Validation: All parameters are validated against reasonable bounds before use

API Relationship

These configuration parameters directly affect server and API behavior:

Server Performance:

• io_threads: Controls parallelism for both accepting new connections and I/O processing. Should typically match CPU core count for optimal performance
• event_batch_size: Larger batches reduce syscall overhead but may increase latency under light load
• max_connections: Prevents resource exhaustion by limiting concurrent connections

Request Handling:

• max_request_size_bytes: Determines when /v1/commit returns 413 Content Too Large
• min_request_id_length: Validates request_id fields in /v1/commit requests for sufficient entropy

Request ID Management:

• request_id_retention_*: Affects availability of data for /v1/status queries and likelihood of log_truncated responses

Commit Pipeline Performance:

• pipeline_wait_strategy: Controls CPU usage vs latency tradeoff in commit processing. WaitIfStageEmpty is safest for shared CPUs, WaitIfUpstreamIdle provides highest throughput with dedicated cores, Never provides lowest latency but uses 100% CPU
• pipeline_release_threads: Determines parallelism in the final stage of commit processing. More threads can improve throughput when processing many concurrent requests

Subscription Streaming:

• max_buffer_size_bytes: Controls when /v1/subscribe connections are terminated due to slow consumption
• keepalive_interval_seconds: Frequency of keepalive comments in /v1/subscribe streams

Configuration Validation

The configuration system includes comprehensive validation with specific bounds checking:

Server Configuration Limits

• port: Must be between 1 and 65535
• max_request_size_bytes: Must be > 0 and ≤ 100MB
• io_threads: Must be between 1 and 1000
• event_batch_size: Must be between 1 and 10000
• max_connections: Must be between 0 and 100000 (0 = unlimited)

Commit Configuration Limits

• min_request_id_length: Must be between 8 and 256 characters
• request_id_retention_hours: Must be between 1 and 8760 hours (1 year)
• request_id_retention_versions: Must be > 0
• pipeline_wait_strategy: Must be one of: "WaitIfStageEmpty", "WaitIfUpstreamIdle", or "Never"
• pipeline_release_threads: Must be between 1 and 64

Subscription Configuration Limits

• max_buffer_size_bytes: Must be > 0 and ≤ 1GB
• keepalive_interval_seconds: Must be between 1 and 3600 seconds (1 hour)

Cross-Validation

• Warns if max_request_size_bytes > max_buffer_size_bytes (potential buffering issues)

Configuration Management

Code Integration

• Configuration Structure: Defined in src/config.hpp with structured types
• Parser Implementation: Located in src/config.cpp using template-based parsing
• Default Values: Embedded as struct defaults for compile-time initialization
• Runtime Usage: Configuration passed to server components during initialization

Development Guidelines

• New Parameters: Add to appropriate struct in src/config.hpp
• Validation: Include bounds checking in ConfigParser::validate_config()
• Documentation: Update this file when adding new configuration options
• Testing: Verify both valid and invalid configuration scenarios