2024 - 2025
Self-Hosted LLM Inference for Lydia
Deployed vLLM and llama.cpp on on-prem GPU infrastructure behind EC2, NGINX, and Route 53, replacing hosted inference and reducing the assistant's serving cost.
Internal Users
Internal eng team
Daily Query Volume
Regular usage
Infra Cost Reduction
Meaningful
Inference Cost Control
Normalized internal usage after moving the assistant's inference to self-hosted GPU infrastructure.
Self-hosted vLLM and llama.cpp serving reduced hosted GPU spend while keeping the assistant usable for daily engineering workflows.
Goal
Move inference for our internal analytics assistant (codename “Lydia”) off hosted GPU infrastructure while keeping latency and operational access practical for internal engineering workflows.
Implementation
I deployed a self-hosted inference engine with vLLM and llama.cpp on an on-prem GPU server behind the existing AWS edge path.
- Put EC2, NGINX, and Route 53 in front of the on-prem GPU server for controlled access.
- Supported both high-throughput serving and lighter local model execution paths with vLLM and llama.cpp.
- Kept the inference runtime separate from the assistant's job-delivery control plane so delivery SLOs and model runtime could be measured independently.
Impact
The stack replaced RunPod, delivered a meaningful reduction in serving cost, and supported regular use of the assistant across the internal engineering team.
Future Extensions
- Add feedback loops for answer-quality scoring.
- Integrate chart generation for trend-heavy questions.
- Support deeper comparative analytics across cohorts.