Frontier AI Is Pricing Itself Into a Corner. Local Models Are the Way Out.

Executive Summary

In 2023 LLM-based artificial intelligence began showing clear utility and a path to business value across nearly all industries with a significant software component. In the subsequent AI boom, the emphasis was on integration and adoption. In 2026, companies are now feeling the costs of AI integration. In some cases, spending is directly encouraged. Nvidia’s Jensen Huang famously said that he expected his software engineers to spend half of their salary in LLM tokens, and a recent Gartner report projects token costs to match employee salaries by 2028 following current growth trends. Such growth is not likely to be sustainable. At Uber, for example, the company exhausted its 2026 token budget in 4 months, and a major division in Microsoft was instructed to pause its AI use due to excessively high bills (Forbes). 

Many trends in the high-tech industry are cyclical, and we expect a pendulum swing with AI cost appetite. Consumers will demand substantially more economic methods for using AI, which will in turn lead to an overhaul in compute architectures, agentic harnesses, and model choice.

Rising costs can be attributed to one or more of the following factors:

1. Model Size: Larger model sizes correlates with intelligence breadth, but greater model parameter counts require more compute. Models have increased in size from hundreds of millions of parameters to trillions of parameters in only a few years. Whereas model size for all classes has increased by an average of 2.3x per year for over a decade, compute per dollar has only scaled up by 1.4x annually.
2. Inference Scaling and Reasoning: Models can use additional tokens for reasoning, improving accuracy in particular on technical domains. Generated tokens are expensive, however, at 4-10x the cost of input tokens.
3. Agents and Tool Calling: Tools fill in the gaps inherent to LLMs, and agents are used for task deconstruction. Each tool call represents another inference pass, which fills up more model context, with hundreds of tools being called per user input under agentic use. Though cached, interim tokens can often far exceed input and final output.
4. Model Choice and Routing: Providers (and often customers) are incentivized to use the most expensive model for a given task, even if a less expensive model could have done the task equally well. Smart routing can mitigate this to some extent, but each change in model negates all previous KV cache for the interaction.
5. Adoption: More use cases are integrated with AI and/or more users are invoking AI-based solutions.

We expect AI adoption to continuously increase, so we will focus on cost factors 1-4 in the next section, after which we will describe how we lower costs, including by leveraging:

• Smaller models that match frontier performance on specialized tasks,
• Concise models trained to produce fewer tokens,
• Edge-focused agentic harnesses designed hand-in-glove for and with efficient models,
• The right model for the job, and
• The unused compute you already possess.

Our local WarClaw harness saves us $25,000 per project annually, our local code assistants save $9,000 per developer per month, and our local batch inference setup saves us $12,100 over a day of continuous use.

Causes of Accelerating Cost

Cost Factor 1: Model Size

Language models became widely effective once they became “large,” and for AI models of all types, performance increases with model and training set sizes. The correlation between task performance and model size was popularized for many in the GPT-3 paper, in which model size was increased 117x versus GPT-2. Figure 1 shows how accuracy increases for zero-shot, one-shot, and few-shot prompting.

This work heralded a race for increasingly large models, culminating in models now with trillions of parameters, albeit often configured in Mixture of Expert (MoE) architectures to decrease inference latency. Epoch AI, as shown in Figure 2, has assessed model size as increasing 2.3x per year across the deep learning era. This is contrasted with compute per dollar (figure 3), which only increases by 1.4x per year. 

Cost Factor 2: Inference Scaling and Reasoning

The “Chain of Thought” paper introduced the idea of reasoning in LLMs. Before that time, an LLM would always immediately output its final answer. With reasoning enabled, the model can first weigh different options for how to respond, perform multi-step assessments, verify its own thinking, and more. Reasoning substantially increases final task performance, particularly for domains like math and coding, an example of which is shown in Figure 4. 

The number of reasoning tokens are then controlled by the “thinking effort” or other similarly-named settings that can be configured with a given model. In some cases this setting is determined automatically, but more often it is a user-controlled setting. Thinking tokens are generated by the model, and output tokens command a price that is 4-10 times higher than input tokens for major LLM providers.

Cost Factor 3: Agents and Tool Calling

LLMs have a number of inherent weaknesses. They are trained on data from a snapshot in time, meaning that they cannot know about current events or any data otherwise unseen during training. The quadratic nature of self attention and the nondeterminism of neural networks make LLMs ill-suited for many data processing tasks, even ones that are trivial to accomplish with short Python or bash scripts. For these reasons and others, many modern systems provide the LLM with tools it can choose to use, such as web search, bash and Python scripting, document retrieval, and memory management. Similarly, many complex problems must be decomposed into a set of tasks, which can then be accomplished by several AI agents that can be dispatched. Yuan et al. characterize modern agentic workflows, with select results shown in Figure 5.

A few points stand out from their work. Most workflows require tens of agentic turns per user input, sometimes extending to over 100 turns. This is a similar finding to Artificial Analysis’ AA-briefcase, which requires over 40 turns per task for the best-performing models. Model context fills during the workflow, with averages in the tens of thousands of input tokens and can necessitate compaction. Tool calling tokens often dominate the context, even consuming up to 90% of the input. Other interesting results emerge, such as the fact that reasoning models may require fewer tool calls. This then leads to design tensions for which a solution is not always clear, such as whether one accepts the additional token costs for reasoning in order to achieve fewer tool calls and agentic turns.

Cost Factor 4: Model Choice and Routing

AI systems have already surpassed human performance on an increasing number of tasks at various points in the past. Even as far back as 2021, Kiera et al. documented the trend of accelerating benchmark saturation, meaning that the time needed for an AI model to exceed human performance after introduction of a new benchmark is decreasing.

Although such trends are generally interesting, in our context this tells us that older, less expensive models and techniques are already capable of superhuman performance for certain tasks. But here’s the rub: How can it be known beforehand if the system configuration will perform well for an upcoming task? It can be difficult for either the system designer or the final user to choose model size, reasoning settings, agentic settings, or available tools ex ante. At this stage in the macro AI cycle where venture capital is abundant and customer willingness to invest is high, most providers and customers will err toward the possibility of higher accuracy, even though they will then often use a setup that is massive overkill for the task at hand. Anyone who has used Claude Opus to summarize an email or write a simple Python function understands this dynamic.

Cost is Accelerating

Because cost is increasing along all of the aforementioned axes, the overall cost to the end-user is accelerating. As an anecdote, one can see dramatic cost increases over 1.5 years for Artificial Analysis to run their Intelligence Index benchmarks, as shown in Figure 7.

Their index is composed of both direct model tasks and agentic tasks. Although scores on the index are rising, as shown in the parentheses of each model label, costs have risen from tens of dollars to thousands of dollars due to increased model sizes, increased reasoning token budgets, and increased agentic iteration. Given that these tasks are representative of real tasks performed by end customers of AI, a similar increase in cost to end users can be inferred.

Case Studies

Case Study 1: AI-Assisted Development

$9,000 saved per developer per month

As a real-world example, we examined the cost to perform human-in-the-loop code development with Claude Code. We used Claude Code Desktop, code was written in Python, Opus 4.8 and Haiku 4.5 were chosen as subagents, and we used a mix of effort levels from low to ultracode. Though specifically this work was for developing model layer information estimation, the types of tasks are representative of most software development workflows, including asking Claude to develop, to test, to fix, and iterating across all three task types. Wrapping in the tool calls, their context, and agents, we found the all-in cost for 1M final output tokens, to include all interim turns and tools, to be $200. The 30-day total cost is then $9,000 for a single user and one project.

Case Study 2: Advanced Agentic Systems

$25,000 per project saved annually

Using our WarClaw tooling, we created an advanced agentic framework that performs development on multiple codebases, conducts iterative improvements along with human collaborators, ingests realtime signals from the web, and integrates with numerous popular messaging platforms. The system is designed to ingest high level directives from its administrator, decompose that into smaller tasks, write code, write tests, and implement new features as background processes. It possesses memory and can access a host of tools within its sandbox. Besides direct responses to user inputs, the system also monitors our bug reports, kanbans, and feature requests, and it works on these asynchronously in branches of each codebase. The goal is a system that continuously runs and makes improvements.

We designed this system to run with local, edge models. Over the most recent 30 days for one project, the system ingested 1.6B input tokens and generated 22M output tokens. Even assuming a high 70% cache hit rate for input tokens, such usage would translate to $2,100 per project per month for Claude Sonnet 5 and $3,500 per project per month for Claude Opus 4.8.

Case Study 3: Batch Processing

$12,100 saved per day of use

Internally we also utilize on-device compute resources to perform large-scale evaluations and data analysis. In one such case we had a workstation with a single GPU produce 3.7M evaluation records (one per response) over the course of 4 days, yielding over a billion input tokens per day. Reproducing the setup using GPT 5.6 Luna, the cheapest in the latest OpenAI lineup, would cost us an estimated $12,100 per day, which is more than triple the cost of hardware for the local machine, while our local setup consumes $5 per day in electricity.Specialized local setups allow us to keep GPU utilization pinned at >95% by constantly feeding information and managing data movement in predictable patterns, greatly improving compute efficiency and overall throughput, while decreasing overhead and cost. This has allowed us to scale experimentations and insights immensely at a fraction of a percent of the cost.

Saving Money with Local, Edge Systems

At EdgeRunner, we’re tackling each cost factor independently, leading to substantial reductions in overall cost. Besides the converses of the previously-described cost factors, we introduce an additional cost saving factor, namely that we can leverage the unused compute capacity that you already possess.

Cost Saving 1: Smaller Models that Match Frontier Performance on Specialized Tasks

Larger model size is a reliable way to increase model performance, but the broader research community has been improving models of all sizes in parallel. For example, OpenAI’s more recent gpt-oss-20b model, which has 20-billion total parameters and only 3.6B active parameters, can achieve a score of 24.6% on the Natural Questions benchmark without using any reasoning, whereas their older GPT-3 model, in which all 175-billion parameters were active, could only achieve a score of 14.6% in the zero-shot setting. Improvements have been realized through better training data, distillation techniques from larger models, newer training techniques like Reinforcement Learning from Verified Rewards (RLVR), and much more.

In our own work for the military domain, we recently published results showing that our Medium model, which can run on laptops with as little as 16 GB of VRAM, can routinely match and occasionally exceed frontier model performance, as shown in Figure 8.

Even EdgeRunner Light 2.0, which can run on laptops and smartphones with 8 GB of VRAM, sometimes also matches frontier model performance. These achievements are the result of specialization. Rather than building a single model that achieves full state-of-the-art results across all conceivable tasks, we deploy smaller models that are specialized for the specific tasks that our users actually perform.

Cost Saving 2: Concise Models

Though with less investment than in other areas, researchers are exploring methods for increasing token efficiency. This includes techniques such as adaptive reasoning, in which the model dynamically adjusts how much reasoning it performs based on its own perception of the problem’s difficultly, training the model to be more efficient, assessing if the model already knows the answer before calling tools, and even end-user mitigations like the caveman plugin.

We are actively working toward increased token efficiency along several vectors. As part of our model training, we curate our training data to prefer denser responses, and we have begun integrating length penalties into our reinforcement learning pipelines. In our evaluations, we have performed a number of analyses to arrive at a set of model judges that don’t bias toward longer responses. Token efficiency is important not only for cost savings, but also for tactical realities. Many military situations are time-sensitive, and the AI must be as succinct as possible while retaining a fully accurate response. Although EdgeRunner Light 2 was not heavily tuned for conciseness, Figure 9 shows that it used 17% fewer tokens versus GPT 5.5 on our military benchmarks. Lower token counts have the added bonus of higher information density, which improves accuracy for subsequent turns. Our goal is to lower our token usage even further as time progresses.

Cost Saving 3: Edge-Focused Agentic Harnesses

We continue to develop EdgeRunner WarClaw, our secure agentic harness designed for use on edge devices, and focus on the following facets: First, we strive to keep a given model in memory as long as possible and to prevent cache invalidation. KV caching increases real-world inference speeds by 5x or more. Second, we are developing toolsets with the highest possible information density. For example, we have invested heavily in tuning our information retrieval system in the EdgeRunner platform to minimize the returned tokens while still capturing the crucial points of knowledge from the grounding documents. Third, aggressively limit turn counts, both through model and harness configurations. In net, we can deploy local, agentic systems that are still fast enough for interactive use.

Cost Saving 4: The Right Model for the Job

It’s only possible to deploy the right model for a given task by deeply understanding the end user. Our research team invests heavily in creating benchmarks that are aligned to our end users. Our gold sets are designed and created by veterans with decades of experience in their respective fields. Our platinum data, which comes from live systems and deployments, is carefully curated to remove noise and extract clear signals about use cases and our models’ performance. By building statistically-significant test sets with high amounts of representation to end user behaviors, we can then quantitatively assess model candidates and deploy those which best align to the end user needs while requiring the least amount of compute or memory. Besides single-model assessments, this concept extends to full agentic assessments, including changes to tool availability, the harness types, and more.

Once we have a proper understanding of a particular use case, then we develop specialized, highly efficient models that can accomplish that use case at frontier accuracies. For EdgeRunner this often translates to MOS-specific or AFSC-specific models, but the concept also applies to our other work developing document processing models, structured data agents, and more.

Cost Saving 5: The Unused Compute You Already Possess

We build models across various tiers to target varying hardware capabilities. Support ranges from smartphone and commodity laptops to powerful workstations. As volunteer computing enthusiasts (e.g. SETI) know, personal computers and devices can be idle for up to 80% of the day. Even when active, most users will rarely max out their machine’s hardware for common office work. This idle compute and memory can be used in one of two ways. First, it can be used synchronously, such as with our core EdgeRunner application, in which models are loaded into memory and invoked in real time to perform tasks. Synchronous use requires a sufficient margin of compute and memory, but that margin is often present. Alternatively, a system like WarClaw can be used to perform recurring tasks and data processing asynchronously, including when the user is not actively using the machine. For example, the system can read and categorize incoming emails and messages, assess data streams on the fly (news, stock prices, sports), detect anomalies, generate candidate artifacts that can later be reviewed by the user, and more.

Conclusion

AI costs are not only increasing, but the cost increase is accelerating. Should current trends hold, AI would only be accessible to the most well-funded companies and laboratories. The good news is that there are concurrent trends to make practical AI both less expensive and more accessible. Whether it’s by using smaller, tuned models, by designing efficient agentic and tool-calling harnesses, or by training models to be more efficient, our mission at EdgeRunner is to keep powerful AI within arms reach, all while ensuring each user’s data remains fully private. With EdgeRunner, you own your AI.

In closing, we invite you to review the dramatic incremental cost savings we observe when using our own benchmarking suite. Across our suite of 44 military evaluations, we can generate over ten million tokens on our laptops for about 20 cents of electricity, which would in turn cost about $440 with GPT 5.5. These cost savings are achieved while still achieving frontier model accuracy on the tasks our users care about.

About the Authors

Jack FitzGerald is Chief Science Officer at EdgeRunner AI. He is a graduate of the US Air Force Academy and the Air Force Institute of Technology. Before joining EdgeRunner, he was a nuclear physicist and Air Force officer, after which he was a principal scientist at Amazon, where he worked on Amazon’s first LLMs.

Vincent Lu is Chief Technology Officer at EdgeRunner AI. A graduate of UCLA with a background in deep learning, he leads the development work for running models at the edge. Prior to joining EdgeRunner, Vincent worked R&D under Boeing Space, Intelligence and Weapon Systems, leading AI/ML efforts within the team.

Sean Bailey is Head of Prototyping at EdgeRunner AI. A graduate of UMBC, he brings architectural principles based on industry experience to bring new ways of running inference in edge situations ranging from desktop, mobile, and decentralized inference. Prior to joining EdgeRunner, Sean was the Solutions Engineer at Stability AI, Solutions Architect at AWS, and Solutions Architect for McKinsey & Co.

Dylan Bates is a Research Engineer at EdgeRunner AI. He received his Ph.D. in Applied Mathematics from North Carolina State University in 2021, specializing in machine learning. A former mathematics professor, he is currently a research engineer at EdgeRunner AI, where he leads the model benchmarking program and develops military-specific large language models for deployment in edge and resource-constrained environments.

Contact us: research@edgerunnerai.com