Reducing AI inference costs through disaggregation and custom silicon

The unit cost of inference is decreasing, but the total volume of agentic use cases is rising exponentially. Tuhin Srivastava, speaking on a panel for Baseten, reported that it becomes drastically cheaper to serve the same models every six months.

At Nebius, the infrastructure team faced a bottleneck when serving a multi-turn chat application. Mashrur Haider, technical product manager at Nebius, said separating the prefill and decoding steps increased throughput from one request per second to 10 requests per second on a single node.

Why is inference economics dominating AI budgets?

Haider noted that inference is now consuming a larger share of compute that was previously dedicated to training. He said 60% to 70% of compute used to be allocated for training and post-training.

Haider argued that the shift forces engineering teams to manage variable production costs rather than fixed capital investments, and that the fundamental difference is financial. “The way to think about inference versus training compute is training compute is CapEx and inference compute is OpEx.”

Bill Gross of ProRata AI described how Jensen Huang recently stated that he wants all of his $500,000-a-year engineers to spend $250,000 a year on tokens. According to Gross, Huang feels that using at least half their salary on tokens will make those engineers more efficient.

Gross shared another example of this spending pattern at a major technology company. He recalled that Microsoft told a hired engineer they wanted them to spend $1,000 a day on tokens. Gross calculated that this daily rate amounts to $365,000 a year.

How are teams disaggregating architectures for throughput?

Haider found that separating the prefill and decoding phases of inference improved performance for multi-turn chat use cases. He described a chat completion use case running on DeepSeek V3, an older 671 billion parameter model.

Disaggregating these steps directly improved the system’s ability to handle concurrent users. “We were able to get up to 10 requests per second from one node of [B200] for the DeepSeek v3 model, which is quite an achievement in terms of 10x improvement in throughput,” Haider said.

What role does memory tuning play in cost reduction?

Haider explained that tuning the key-value cache reduces the compute required to serve expanding context windows. He observed that median tokens per session grew from 1.5K to over 32K.

To manage the cost of these large inputs, Haider recommended that teams adjust how they store and retrieve previous tokens. Using the cache naively yields a 5 to 10% reduction in needed compute, and up to 30% for some common use cases, according to Haider. “When you tune the KV cache to your own workload, then you can see more than 50 to 60% improvement, and sometimes even 95% improvement in the cache hit rate.” Haider estimated that an optimized hit rate means a team needs 90% less compute to serve the same customer workload.

How much can software frameworks accelerate GPUs?

Yang Liu, senior solutions architect at NVIDIA, claimed that optimized microservices can more than double the throughput of standard hardware configurations. Benchmarking a Llama 3.1 8 billion parameter model on one H100 SXM with 200 concurrent requests showed a baseline performance of 613 tokens per second.

When using NVIDIA NIM, throughput jumped to 1,200 tokens per second, which Liu stated is 2.6 times higher. Liu added that the framework also delivered 13% lower inter-token latency.

Beyond speed, Liu claimed that software optimization impacts output reliability. Using optimized NIM microservices reduces the error rate by 35%, he reported. For specific tasks, performance improved further: Embed QA was three times better, and re-rank QA was 1.6 times better with the framework.

When do builders move beyond GPUs for latency?

Sid Sheth, founder and CEO of d-Matrix, argued that teams adopt custom SRAM-based silicon when the physical architecture of GPUs creates a bottleneck for fast interactivity. GPUs struggle with low latency, he said, because they require data to travel from high bandwidth memory to the compute engine and back.

Sheth described how this trip consumes significant energy and time, creating what is known as the memory wall. “It's not a money problem. It's an architecture problem. GPUs are a 30-year-old compute engine” that were originally designed for parallel computing.

To bypass this constraint, Sheth noted that hardware providers are building accelerators that keep memory closer to compute. He described the d-Matrix Corsair product as an SRAM-based inferencing solution featuring two gigabytes of SRAM on a single 600-watt card.

Running a 120 billion parameter GPT OSS model on a GPU alone resulted in an end-to-end latency of about 24 seconds, Sheth said. When run alongside Corsair, latency dropped to two seconds, which he called almost a 10X speed up.

As Srivastava noted, the unit cost of inference continues to fall even as usage scales. Balancing these infrastructure choices determines the baseline economics of deploying models at scale.

Frequently asked questions

What are emerging trends in inference economics at the frontier labs?

Sid Sheth, Founder and CEO at d-Matrix, reported that “Companies like Anthropic, OpenAI even tried charging more for those fast tokens, and they were very successful.”

To hear more from the teams building and running AI infrastructure, see the HumanX 2027 program preview.