Dec 2024 - Present
Event-Driven Telemetry Fan-Out for Real-Time Operations
Built Kafka and Redis Pub/Sub fan-out paths for live high-volume telemetry, enabling sub-second operator dashboard updates and horizontally scalable real-time processing.
MQTT Volume
High-volume
Dashboard Freshness
Sub-second
Query Latency Reduction
Substantial
Telemetry Fan-Out Reliability
Normalized operator-facing freshness after separating ingestion, fan-out, and query-serving paths.
Kafka and Redis Pub/Sub separated live operator updates from ingestion pressure while MySQL tuning reduced serving latency.
Telemetry Fan-Out Pipeline
Ingest MQTT
100%Device telemetry enters ECS-backed services
Fan out events
84%Kafka and Redis Pub/Sub decouple consumers
Serve dashboards
82%Operators receive sub-second updates
Tune hot queries
72%Indexes and schema changes cut MySQL latency
Problem
Operations teams needed live telemetry updates without coupling every dashboard and service directly to the ingestion path.
Architecture
I implemented event-driven fan-out over production IoT and backend services.
- Shipped TypeScript and NestJS services on AWS ECS across MySQL, MongoDB, Redis, SNS/SQS, and REST APIs.
- Used Kafka and Redis Pub/Sub to fan out telemetry updates for sub-second operator dashboard refreshes.
- Tuned high-traffic MySQL query paths with composite indexes and schema-level optimization.
Results
The fan-out path supported high-volume MQTT ingestion, enabled horizontally scalable real-time processing, and substantially reduced average MySQL query latency.
Engineering Notes
The key reliability win came from separating ingestion, fan-out, and query-serving responsibilities so operator-facing latency could improve without overloading the write path.