LLMs Have a New Limit – It Costs More to Think Longer

Modern agents use tools, handle text, images, and audio, remember context, use computers, and delegate tasks to subagents. But remember, each step adds more context, more tokens, and more compute.

For developers, that means model quality still matters, but benchmark scores no longer tell the whole story. As Igor Dmochowski (Developer Relations Manager, Nvidia) said at Infobip Shift 2026:

An agent is far more complex than a 2022-style chatbot: it can use tools, handle multimodal context, retain memory, and spawn subagents.

Hybrid architectures preserve useful state without making every turn more expensive

Longer context windows matter because agents need to remember more: past messages, documents, tool outputs, code, plans, and earlier steps. But as the context gets longer, self-attention becomes more expensive and harder to scale.

Dmochowski’s point was that context length is a real systems constraint. A model may accept a huge prompt, but still be too slow or too memory-hungry for a real application. The question now is what they cost to process:

The compute required to process your context was growing quadratically with regards to the context. This is not scalable.

The approach discussed on stage was a hybrid model that combines Transformer attention with Mamba-style state-space layers. Instead of using full attention for every token, the model uses more efficient layers for most of the work and keeps attention where it matters most.

That matters for agents because long context often contains information with very different levels of importance. “A production system needs enough capacity to preserve useful state without making every turn proportionally more expensive. Hybrid architectures are one attempt to move that curve in a more manageable direction,” Dmochowski said.

Models like MoE, active compute and throughput matter more than total parameter count

The second shift is sparsity.

In a conventional dense model, every token passes through the same large set of parameters. Igor explained that Mixture of Experts, or MoE, adds a routing mechanism that activates only a subset of specialized expert blocks for each token. The total model can contain far more parameters than are actually used during a single forward pass.

According to Dmochowski, this distinction between total parameters and active parameters is becoming much more useful when comparing models. A large parameter count can suggest capacity, but it says much less about serving cost once sparse architectures enter the picture:

We don’t need to put all of the tokens into all of the parameters of the model, we route them to the experts. For deployment, active compute, memory footprint, context length, KV cache behavior and tokens per second can be more informative than the headline parameter number alone.

Dmochowski connected that efficiency directly to capability. His point was that more throughput gives a system more room to reason within the same compute budget.

Agents can do more intermediate work and finish long tasks with lower latency. When reasoning uses thousands of tokens, throughput becomes part of the capability envelope, which is why multi-token prediction matters.

Lower-precision formats can reduce memory and compute requirements as well. Both techniques point toward the same goal: spend less hardware effort per useful token without destroying model quality:

The faster model is going to be the smarter model. There is obviously a trade-off, but producing two or four times more tokens with the same compute gives you room to make that trade-off.

For developers, efficiency is becoming an architectural feature

A model’s behavior depends on where you run it – the same model can act differently through a basic API than it does inside an agent harness with tools, memory, system instructions, and a loop that shows it the results of its actions.

A leaderboard or prompt benchmark only tells part of the story. An agentic app is a full system, not just a model. “Tool schemas, retry logic, context construction, memory strategy and the execution loop all influence the final result. Evaluation should therefore resemble the environment in which the model will actually run,” Dmochowski said:

The environment in which the models are set matters quite a lot. The behavior you get through an API can be completely different from the one you get within a harness.

The same idea applies during training and post-training: models should be trained in environments that simulate agent harnesses, and model choice and system design should be tested together.

Open models can be the better choice when private data has to stay in-house

Dmochowski also made a case for open models in domains where the most valuable data cannot be handed to a frontier model provider. Internal code, proprietary documents, regulated records and specialized workflows may require organizations to adapt models within their own environment:

Open weights, training recipes and reproducible tooling can matter most in exactly those cases, because the competitive advantage is often the private data and workflow rather than the base model itself.

The useful takeaway from Dmochowski’s talk is that the bottlenecks are moving. The early LLM conversation focused on scale and benchmark scores, but agentic systems force developers to think about long context, memory, generation speed, tool use, and the cost of every extra reasoning step. That changes how models should be evaluated: a smaller or sparser model can be the better choice if it keeps quality while lowering latency, and a huge context window only matters if the system can afford to use it.