The Uncomfortable Reality of Reducing AI Inference Costs
Distillation, caching, routing, and what agents do to your token bill.
Reducing AI inference costs takes several techniques at once. Builders are distilling large models, caching tokens, running evaluations in parallel, and bypassing language models for static knowledge, speakers from Nebius and Sierra said. Yang Liu at Nvidia said contributing to the open-source community and using that work inside Nvidia's NIM microservices gives the best performance.
Guillermo Rauch, Founder & CEO at Vercel, said his team improved the performance of an open-source project by upwards of 90%. One engineer there orchestrated a large number of digital employees in the cloud, and that used a lot of tokens.
Why are agents driving up token use?
Rauch noted that upwards of 70% of page views now come from agents and only 30% from humans.
Noam Salinger, Co-founder & CEO at Impala AI, said AI is moving beyond chat applications that answer questions. It now augments work with 24/7 agents that run while people sleep. He claimed Impala AI built a fully managed inference platform that optimizes every layer for throughput to support autonomous, asynchronous agents.
Which model-level changes cut costs?
Sujee Maniyam, Developer Advocate at Nebius, recalled a note-taking app that went from roughly 100 billion parameters to 2 billion using distillation. Cosine, an enterprise coding agent, started from a 300 billion parameter model and trained smaller models on its own data. Maniyam said the smaller models perform better because they are tuned to that data.
Caching tokens gives speedups of five to 10 times, Maniyam said, though the cache grows memory intensive as more tokens are generated.
Dan Fu, VP, Kernels at Together AI, said FP8 quantization in the KV cache is pretty standard today. The industry does not yet have good formulas for lower precision such as FP4.
How do routing and bypassing save money?
Pol Peiffer, Head of Agent Development at Sierra, said a simple turn of a conversation might make a dozen or more inference calls to different models.
Liu described a split that avoids paying frontier prices for every step. A frontier large language model handles orchestration, and a cheaper open model handles information retrieval. Sierra also uses tools that let agents inject static knowledge into an answer without the language model ever seeing it.
What do infrastructure and hardware change?
Liu ran a test of 200 concurrent requests on a Llama 3.1 8 billion parameter model using one H100 SXM configuration. Without NIM, performance was around 613 tokens per second. With NIM, it jumped to about 1,200 tokens per second.
Fu said the industry's ability to get the most out of GPUs is actually maybe 2x worse than it was a few years ago. Even well-optimized inference stacks see about 5% flop utilization, he noted. Together AI uses a MegaKernel approach that treats the entire model as a single computation, which helped a customer who is sensitive to the time to the first 64 tokens.
Vercel automated upwards of 90% of its support requests with its own support agent, displacing legacy platforms.
How should AI-native scaleups budget for inference?
Greg Shove, CEO at Section AI, predicted that by 2030, super managers will get an annual inference budget to manage for ROI. At his own 40-person company, humans account for 80% of the operating budget, and inference is already the third biggest expense. Shove said inference will soon be the second biggest expense, ahead of all other software combined, as usage climbs.
Sierra bills on outcomes. Customers pay when an agent does a job well, and the company does not charge by the token, which keeps pricing tied to customer results. Rauch added that Vercel's designers are now producing their own software and agents.
As agents take over web traffic, reducing AI inference costs means optimizing every layer of your infrastructure for throughput and parallel execution.
To hear more from the people putting AI into production, see the HumanX 2027 program preview.
Frequently asked questions
What are the most effective ways to cut inference costs for large language models in production?
Sujee Maniyam at Nebius said distillation uses a larger teacher model to train a smaller student model. Pol Peiffer at Sierra added that agents can skip the language model entirely for static knowledge such as a legal disclaimer, and that an account balance can be handled with a template string that is filled in at the end so the number is never wrong in the output tokens.
How do prompt caching and batching specifically impact inference cost at scale?
Caching changes the compute requirement from quadratic to linear, according to Sujee Maniyam at Nebius. He noted that the cache competes for space with model weights and other parameters. Because the cache can be evicted when the GPU needs room for model data, Nebius offloads it to regular memory or even disk.
What trade-offs should CTOs consider when compressing or quantizing models?
Dan Fu at Together AI said FP8 quantization in the KV cache is standard today, but the industry lacks good formulas for lower precision such as FP4. He noted that Google recently published the TurboQuant paper on the topic, and that Together AI is still evaluating whether the approach will work.