Lydia Agentic AI Assistant and Event Delivery Platform

Problem

High-volume telemetry questions often required engineers to inspect schemas, write SQL, run analysis, and turn the result into a chart or report. Lydia was built to make that workflow conversational without losing the operational controls needed around long-running agent work.

The first version used a direct agent execution path that worked for a prototype but carried production risks: browser sessions were tightly coupled to long-running work, restarts could strand state, cancel behavior was informal, and there was no durable job lifecycle to inspect or replay.

The production requirement was larger than the inference loop. The platform needed tenant-aware job ownership, idempotent submission, reconnect and resume behavior, formal terminal states, backpressure, and enough observability to explain stuck or slow jobs.

Architecture Decision

I built Lydia as a multi-turn, tool-calling agent for high-volume telemetry analytics and proposed moving its execution path from direct calls to a durable job-delivery model:

• A gateway accepts authenticated requests, validates ownership, and writes job state to MySQL.
• The agent inspects schema context, generates SQL, executes analytical queries over ClickHouse, and produces charts or reports from the result.
• A transactional outbox records the handoff event in the same database transaction, avoiding a database-plus-broker dual-write gap.
• A polling publisher delivers accepted jobs to Kafka/MSK for durable processing by the agent backend.
• Redis Streams carry replayable live events back to connected frontend sessions.
• Redis Pub/Sub is used only as a fast interrupt path for cooperative cancel.
• ClickHouse handles analytical SQL over high-volume telemetry, while richer turn history, tool traces, and audit detail stay outside the hot control plane.

The design deliberately kept the v1 runtime on EC2 and on-prem inference because the immediate problem was not worker scale. The core gap was reliable job ownership and recovery semantics around long-running agent work.

Job Lifecycle

The control plane makes the lifecycle explicit:

received -> queued -> processing -> final
                              \-> error
                              \-> cancelled
                              \-> timed_out

Frontend submit, reconnect, follow-up, and cancel flows all resolve through the durable job record rather than a single live socket. That lets multiple frontend sessions observe the same job and lets the gateway recover state after process restarts.

Shipped & Staging-Validated

Production Runtime & Package Boundaries

I organized the worker runtime as a set of responsibility-scoped packages rather than one service blob. A thin composition root wires everything at startup, a lifecycle layer owns graceful draining, an orchestration layer sequences a job's turns, wire-contract packages hold the shared message shapes, adapters isolate every external system behind an interface, and an observability package centralizes tracing and structured logs. Dependencies only point inward — orchestration and adapters may depend on contracts, never the reverse — so the boundaries are enforceable, not aspirational. Authentication fails closed by default: a request without a verified identity is rejected, not waved through. The runtime processes a single job at a time per worker, which keeps concurrency reasoning trivial and makes draining, cancellation checkpoints, and back-pressure deterministic. This structure has been exercised end-to-end in staging.

Client Contract & Resumable Delivery

The client contract is frozen at v1 so the frontend and backend can evolve independently. Submission is idempotent: retrying the same request returns the existing job instead of forking a duplicate. When a client subscribes, it first receives a snapshot, then a cursor-based replay of buffered events up to a high-water mark, and only past that boundary does it switch to the live tail — so a reconnecting client never sees a gap or a duplicate at the seam. All subscribers to one job share a single live tail rather than each spawning their own. Ownership is validated before any event is delivered, so a client only ever observes its own work. Terminal ordering is deliberate: I drain the final transient events, then synthesize terminal status from the durable store — transient telemetry and durable lifecycle truth are kept strictly separate.

End-to-End Validation

I validated the delivery path end-to-end in staging against the failure modes that actually matter for long-running agent work, not just the happy path. The staging-verified dimensions were:

• Submission — a job is accepted, persisted, and handed off durably under at-least-once delivery, with idempotent execution ensuring a retry or redelivery never produces a second job.
• Replay then live — a subscriber replays buffered events and crosses cleanly into the live tail with no gap or overlap.
• Reconnect recovery — a client that drops and returns resumes from its cursor without losing or repeating events.
• Terminal snapshots — final status is reconstructable from durable state alone.
• Idempotent retry — re-submitting the same request is a no-op that returns the original job.
• Duplicate suppression — repeated or overlapping events collapse to a single observed sequence.

Each dimension has an explicit staging check rather than a manual spot test.

Architecture in Progress

Cooperative Cancellation

Cancellation is designed but not yet shipped. The plan uses a dual-path signal: a durable cancel request written to the control plane as the source of truth, plus a lossy, low-latency hint for fast reaction. The hint is an optimization, never a dependency — the headline invariant is that correctness never relies on the hint path arriving. If the hint is dropped, heartbeat convergence closes the gap: the worker re-reads authoritative state on its regular heartbeat and honors a pending cancel it never received directly. A watchdog handles the harder case where a worker stops heartbeating entirely, reclaiming and finalizing the job so it can't hang in a live state forever. Cancellation resolves only at safe checkpoints — between turns, tool calls, and stream chunks — so a cancelled job always lands in a well-defined terminal state.

Attachment Verification Pipeline

Attachment handling is a designed verification pipeline, not yet in production. Uploads go directly to object storage through a presigned request, so large payloads never stream through the application tier. On arrival, content is verified before it's trusted: size bounds, a checksum match against what the client declared, and real content-type inspection rather than a claimed extension. Verified content is then normalized — re-encoded and stripped of embedded metadata — so nothing incidental rides along with it. Only after an attachment passes readiness gating is it associated with its job, and that association happens in a single transaction so a half-attached record can never exist. Access is authenticated per request rather than handed out as long-lived bearer URLs, so a leaked link is not a standing grant. The design keeps untrusted bytes quarantined until every check has passed.

SLO Thinking

The key architecture decision was to separate delivery latency from model execution time. GPU inference, tools, and retrieval can be slow for legitimate reasons; the platform still needs to prove that accepted jobs move through handoff, queueing, event delivery, and terminal-state updates predictably.

The provisional delivery SLOs focus on measurements such as time to queued, event-to-browser latency, outbox publish lag, broker consumer lag, and terminal-state completion ratio. The cancel-related targets — cancel acknowledgement and cancel completion after checkpoint — are provisional goals for the in-progress cancellation work above, not measurements from a shipped path.

Tradeoffs

This design adds more infrastructure than a direct request/response path. The tradeoff is intentional: a broker and control plane are not necessary for a single happy-path prototype, but they become justified when the product needs durable handoff, replay, backpressure, reconnect, auditability, and restart recovery.

For v1, a polling outbox publisher was the pragmatic choice over a managed CDC connector. It keeps cost low, gives the application direct control over retry and status semantics, and is enough for a single app-owned outbox stream.

The hardest part was not streaming model output. It was deciding which system remained authoritative when the database, the broker, the cache, the worker, and the client could each fail independently.