Why cheaper AI models do not settle the token bill

CNBC — Technology reports that OpenAI introduced GPT-6 Sol and GPT-6 Luna while Anthropic unveiled Claude Opus 5.5 on 22 September, in releases framed around cheaper models. CNBC’s report was independently matched by AWS Machine Learning, OpenAI, Simon Willison, TechCrunch, The Verge and ZDNET.

OpenAI says GPT-6 Sol and Luna offer different balances of capability and cost, while AWS Machine Learning says the models give users options to match intelligence and efficiency to a workload. The immediate question for a team buying or deploying models is not just which model costs less. It is what the system sends to that model, how often it does so, and whether the surrounding design makes a lower price meaningful.

A token bill is a system property

A token is the unit counted when material enters or leaves a language model. In a simple exchange, instructions and supporting material are input; the generated response is output. A model price concerns those units, but an application’s bill depends on the volume of material its design causes the model to process.

That distinction becomes important in retrieval-augmented generation, or RAG. A RAG system retrieves external evidence for a request and supplies it to the model during generation. A preprint on arXiv reports that RAG can improve language models with retrieved evidence, but that processing many long passages is costly and can introduce distracting information.

This is the practical tension behind the phrase token efficiency. More retrieved material can mean more evidence, but it can also mean more material for the model to process and more irrelevant material competing for attention. The lower price of a model and the size of the context passed to it are separate variables. A release can change one without changing the other.

For the engineer maintaining a knowledge assistant, this shifts the unit of analysis from a model call to the whole pipeline. What documents are retrieved? Which passages reach generation? How much of the conversation and tool history is kept? The supplied launch coverage does not answer those questions for GPT-6 Sol, GPT-6 Luna or Claude Opus 5.5. It does show why they matter when vendors compete on cost.

Soft compression changes the representation

Context compression is an attempt to reduce the amount of retrieved material presented at generation time. It is not the same claim as finding better documents. It concerns how the documents already selected are represented to the model.

The AdaMem preprint describes one version, called soft compression. The authors say passages are encoded as compact sequences of continuous memory embeddings before generation. Its title describes adaptive memory-token allocation: rather than treating context size as fixed in every case, the work proposes allocating memory tokens adaptively.

The reason for building such a mechanism follows from the RAG problem. Long retrieved passages create a cost burden, according to the authors, and can add distracting information. A compressed representation is meant to preserve useful information in less context. That is a trade-off, not a free reduction: the system needs enough of the evidence to support an answer while giving the generator less material to handle.

The important technical distinction is that this is a representation method, not a new source of knowledge. Compression cannot supply evidence that retrieval did not find. Nor does the supplied abstract state the allocation rule, the amount of compression applied, or how the method handles a passage whose crucial detail is hard to preserve. AdaMem is a preprint, not peer-reviewed research, so its reported approach should not be read as a settled deployment result.

The saving must survive the agent loop

A shorter context does not settle the cost question either. A separate preprint on arXiv, An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents, starts from a useful warning: public benchmarks may report that aggressive compression preserves task-solving quality, yet that does not imply that compressing file reads saves money in a real multi-turn agent.

The authors describe instrumenting a production compression gateway because the popular inference skips too many steps. A compression method can look satisfactory when judged on task completion, while the operational question is whether it changes total expenditure across an agent workflow. The paper’s subject is not simply a smaller prompt. It is cost attribution in an agent that has multiple turns.

That matters because an agent is a sequence, not a single response. A coding workflow may read files, call tools, receive results and continue. The context management decision sits inside that loop. The relevant test is therefore not merely whether a compressed version remains useful in isolation, but whether the complete workflow uses less costly model processing without creating a new burden elsewhere in the sequence.

The preprint does not establish that compression fails to save money. It says the conclusion cannot be assumed from quality results alone. That is a valuable boundary for procurement and engineering alike: a model’s advertised cost position, a benchmark result and an application’s total cost are three different claims requiring different evidence.

Less context can still be the wrong context

The second failure mode is relevance. A preprint on arXiv reports that a model’s ability to locate and use decisive evidence may degrade as irrelevant or confusable context is added. The authors call this context poisoning and frame it as interference in attention: decisive evidence has to compete with other material that appears highly relevant.

That observation makes compression more than a billing technique. Keeping less context could help with distraction, but only if the system preserves the evidence that decides the answer. Keeping a compact representation of the wrong passage does not solve the retrieval problem. Discarding the decisive qualification from the right passage would not solve it either.

The supplied AdaMem material does not say how its method distinguishes decisive evidence from expendable detail, and it does not report a result on context poisoning. The context-poisoning paper is also a preprint, not peer-reviewed research. Neither item establishes that context compression makes a RAG system reliable; together, they identify the tension an implementation has to test.

This is where a reader should resist a familiar shortcut. More context is not automatically safer, because irrelevant context can interfere. Less context is not automatically cheaper in the total workflow, because an agent may still take further turns. And compressed context is not automatically faithful, because a compact representation still has to carry the evidence the answer depends on.

Price changes the question, not the architecture

The releases of GPT-6 Sol, GPT-6 Luna and Claude Opus 5.5 make model selection more visible, because they put lower-cost positioning alongside high-capability model families. OpenAI says its two new models make different capability-and-cost trade-offs, and AWS Machine Learning says they are available on Amazon Bedrock for matching efficiency to a workload.

Understanding context compression makes that language more concrete. The model choice affects the cost terms for processed tokens. Retrieval, conversation handling and compression affect how much context the application asks that model to process, whether useful evidence survives, and what happens across an agent’s full sequence of work.

No source in the supplied material reports a head-to-head total-cost study of the new OpenAI and Anthropic models in a particular RAG or agent system. It would therefore be wrong to treat this week’s launches as proof that any one workflow will cost less. The useful conclusion is narrower: a cheaper model is an input to system cost, not a replacement for measuring the system that uses it.