Five Practical Lessons for Serving Models with Triton Inference Server

Summary

Triton Inference Server has become a popular choice for production model serving, and for good reason: it is fast, flexible, and powerful. That said, using Triton effectively requires understanding where it shines—and where it very much does not. This post collects five practical lessons from running Triton in production that I wish I had internalized earlier.Choose the Right Serving LayerNot all models belong on Triton. Use vLLM for generative models; use Triton for more traditional inference workloads.LLMs are everywhere right now, and Triton offers integrations with both TensorRT-LLM and vLLM. At first glance, this makes Triton look like a one-stop shop for serving everything from image classifiers to large language models.In practice, I’ve found that Triton adds very little on top of a “raw” vLLM deployment. That’s not a knock on Triton—it’s a reflection of how different generative workloads are from classical inference. Many of Triton’s best features simply don’t map cleanly to the way LLMs are served.A few concrete examples make this clear:Dynamic batching → Continuous batching Triton’s dynamic batcher waits briefly to group whole requests and then executes them together. This works extremely well for fixed-shape inference. LLM serving, on the other hand, benefits from continuous batching, where new requests are inserted into an active batch as others finish generating tokens. While this is technically possible through Triton’s vLLM backend, it is neither simple nor obvious to operate.Model packing → Model sharding Triton makes it easy to pack multiple models onto a single GPU to improve utilization. LLMs rarely fit this model. Even modest models tend to consume an entire GPU, and larger ones require sharding across GPUs or even nodes. Triton doesn’t prevent this, but it also doesn’t meaningfully help.Request caching → Prefix caching Triton’s built-in cache works by storing request–response pairs, which is very effective for deterministic workloads. Generative...