Self-hosted inference

From a model to a service

Serving an LLM brings familiar backend questions into a different resource environment. Requests compete for limited capacity, longer inputs change execution behavior, and a fast response in a small test does not establish how an application will behave under concurrent use.

I built self-hosted inference using vLLM and llama.cpp and integrated serving with application workflows. My contribution covered getting models running, configuring serving profiles, and evaluating whether the resulting setup was suitable for the work the application actually performed.

How I evaluated it

I benchmarked representative workloads at different levels of concurrency and examined throughput, tail latency, and GPU memory behavior. I also evaluated complete agent workloads, where model calls are only part of the elapsed time a user experiences.

That distinction matters when interpreting a result. An inference benchmark can reveal runtime behavior, while an application benchmark exposes the effect of orchestration and tool use. Each measurement answers a different question. A configuration that looks promising in one test still needs to be evaluated in the context where it will run.

I used the measurements to qualify a serving profile and to make the operating tradeoffs concrete. The work was as much about choosing a useful test and interpreting its limits as it was about adjusting runtime settings.

What I take from it

I prefer performance discussions that begin with a workload, an environment, and a definition of success. Those details make comparisons meaningful and give the next engineer a way to repeat the evaluation.

This experience strengthened my interest in ML infrastructure: applying backend engineering to the systems that make models usable. I like working across the boundary between an inference runtime and the application around it, especially when measurements help explain behavior that a single response-time number would hide.