Observability & Operations

This area is about running Chronicler in production without flying blind, and operating it without pulling it down. The through-line is that observability and operability are not bolted on afterwards. They are conventions every service already follows, so that a new context is watchable and operable the day it ships.

Watching, without marrying a vendor

Every service method that takes a context opens a trace span, and every store method names its query, and both names derive automatically from the method rather than being typed by hand, so the instrumentation cannot drift from the code. Traces and logs export over OTLP to a vendor-neutral collector, and the application names an OTLP endpoint, never a specific tracing backend. That keeps the choice of monitoring vendor a deployment decision instead of a code dependency.

Logging goes through one logger, fanned out to a readable console, rotating JSON files, and an OTel sink at once. Verbosity is a single named ladder from high-level milestones down to a protocol firehose, so turning up detail widens one stream rather than flipping category switches and hoping you picked the right ones.

Health and scheduled work

Health checks live in a registry, each tagged by how critical it is. Readiness gates only on the critical dependencies, while a separate diagnostic surface reports on everything, so a non-critical hiccup does not take the instance out of rotation. Recurring maintenance jobs run on NATS's own native scheduling rather than an in-process cron loop, which means the schedule survives a restart and does not depend on one process staying up.

Read models you can rebuild and swap under load

Because read models are projected from the event log, they can be rebuilt, and Chronicler makes that a first-class operation. A read-model instance is a named database plus a consumer namespace, projected independently. You can run two, build the new one from the log while the old one serves traffic, and switch over. That is a blue-green cutover. A carry-over manifest copies the few tables that are not derivable from events, and a write-gate freezes writes during the copy and doubles as a general maintenance mode. Which instance is live is decided outside the app, by a proxy or a pinned restart, so the swap is an operational act, not a risky in-process hot-swap.

Operating the instance

The write-gate grows into proper operator controls: several service modes, a scheduled maintenance window, an optional banner, and a switch to close new registrations, each behind its own permission and reachable from a break-glass CLI for when the UI itself is down. The admin console is a capability-gated area backed by the system API, showing each section only to an operator who holds the matching permission, and moderation adds a suspended-account state with its own login gate and audit trail.

A control plane whose real boundary is its connection

The control plane, which drives things like seeding, rides on the same NATS connection that carries the event store. The honest consequence is that the connection, not the individual command, is the security boundary, so that is what gets secured. Trust is same-network only. A network-reachable bind is authenticated with NKey users, encrypted with TLS, and split into a full service user and a control-only operator user, so a leaked operator credential cannot read or write events. A reachable bind refuses to start without authentication at all. A long-running seed reports its progress through a port that publishes to a subject keyed by a run id, so a client can disconnect and reconnect and still watch it finish.

Where this connects

The rebuildable read models are the Foundations projection idea taken to its operational conclusion, and the durable notification inbox depends on the instance model described here.