How to build a context engine for LLMs in production
Learn how to build a context engine for LLMs, manage token costs, structure context layers, and address challenges in agent memory and retrieval.
Before you work out how to build a context engine for LLMs, settle the context strategy. Simba Khadder, head of AI product at Redis, said that is the first and most important question in any agent strategy. Vishal Marria, founder and CEO at Quantexa, called the context layer the primary requirement for getting value out of AI.
Khadder said that when an agent fails at a coding problem, the cause is almost always missing context or a flawed environment.
Why does raw context inflate token costs?
Rafael Levi, developer relations at Bright Data, compared the cost of raw web data with structured queries. An HTML page takes about 50,000 tokens. A structured database search takes about 300.
Levi's advice: “You don't want to parse pages with LLM. This is something that you don't want to do because that's going to cost you millions of tokens.” Instead, have the agent build a scraper pipeline. “What you tell your agent is to build you a script that doesn't use any LLM to pull this data into your database.”
Khadder described an insurance company that routed every standard customer greeting in its voice agents through the Opus model. Its leader told him, “I spend a million dollars a year on good morning.”
Where do linear RAG pipelines fail?
Khadder described a standard RAG system asked why a customer's order was late. A linear pipeline answered with a generic list of common delays instead of the actual reason.
Xuedong Huang at Zoom said frontier models need dynamic conversational context, internal knowledge and memory of past relationships to be truly useful in the enterprise.
How do agents affect backend infrastructure?
Pointing agents straight at systems of record can break your databases, Khadder warned: “you'll end up breaking your own DBs because agents will run the same query over and over again and not really think of performance.”
Dana Lawson at Netlify said that skipping information architecture before deploying agents leads to tokens wasted on low-value outputs.
How to build a context engine for LLMs that works across models
Organizations now run multi-models, frontier models and open-source models side by side, Marria said, so “you now need to be agnostic at the context layer to be connecting to multiple models.”
Avani Prabhakar of Atlassian described how the company built a context layer and knowledge graph so its AI searches specific assets instead of sprawling across the whole company. The answers got faster and more accurate, and “So we have seen about 48% token cost savings by using AI in that way.”
Debug the environment before the model
Khadder said agent failures today are rarely about intelligence: “It's almost never a it's not smart enough problem anymore. It's almost always a context or data problem.” He predicted a new job will come out of it: “we're going to start seeing context engineers as a role, like a job.”
The common thread in how to build a context engine for LLMs: treat context as its own infrastructure layer, keep raw data and repeat queries away from the model, and fix the environment before blaming the model.
To hear more from the people putting AI into production, see the HumanX 2027 program preview.
Frequently asked questions
What are the most common failure modes when building context layers for agents?
Omer Primor at Bright Data warned that feeding an agent too much data inflates operational costs. He noted that providing too little context also drives up expenses by causing the model to make incorrect decisions.
How do engineers decide what information goes into an LLM’s context window?
Bianca Anghelina at AILY LABS reported that embedding business context into a semantic layer produced recommendations that were 700 times more accurate.