logo
menu

Introducing EdgeRunner Benchmarks: Evaluating AI Model Performance in Modern Warfare

Today we’re launching the EdgeRunner Benchmarks Platform, a new public leaderboard that assesses how leading models perform on a range of military and national security tasks. The new platform shows how 55 models perform across military specialties, how often they decline to answer, and what it costs to run them. Existing general-purpose leaderboards tell a member of the military very little about whether a model understands doctrine, can work through a Special Operations planning problem, or will answer a legitimate question from a warfighter. We’ve built a suite of over 40 benchmarks to assess a model’s military utility, and today we make results from six of them available to the public.

The Site

At today’s launch, the site covers 53 public models from 22 developers in six countries, ranging from a 1.2B model that runs on an ordinary laptop to proprietary frontier models available only through paid APIs. We also include two EdgeRunner models, allowing users to compare their performance against both other edge-deployable models and the broader state of the art.

There are five accuracy benchmarks with reference answers written by veterans and subject-matter experts:

  • Combat Arms: Questions on airborne operations, patrolling, troop-leading procedures, military decision-making process (MDMP) and squad-to-company planning, tied to US Army doctrine.
  • Combat Medicine: Questions on Tactical Combat Casualty Care (TCCC) and military clinical medicine.
  • Military Intelligence: Questions on intelligence doctrine, planning, and multi-source intelligence, including US and foreign doctrine and area studies.
  • Special Operations Communications: Equipment-level questions on the radios and satellite terminals a Special Operations Communications Sergeant operates.
  • Unconventional Warfare: Questions on Special Operations unconventional warfare planning.

A sixth benchmark, Military Refusals, contains legitimate military questions written by Special Operations veterans to provoke refusal behavior. It has no reference answers, so it measures whether a model declined to answer each prompt, not whether its answer was right.

Every chart can be filtered by where a model runs, from paid cloud service provider (CSP) to commodity hardware (laptops), as well as by developer, country, size, open weights and reasoning setting. Every score has a 90% confidence interval, available in the tooltips and tables.

Key Findings

Military questions are difficult even for the best models. Claude Opus 5.5 ranks first on all five accuracy benchmarks, but its scores range from 90.3% on Combat Medic to 60.6% on Unconventional Warfare. No model exceeds 70% on Combat Arms, Special Operations Communications, or Unconventional Warfare.

Figure 1: Example results from the Unconventional Warfare benchmark. The benchmark is difficult, maxing out at 61% accuracy. In this case a specialized edge model, EdgeRunner 2.1, can achieve approximate parity with the best frontier models.

Public models decline most queries in the Military Refusals benchmark. Of the 53 public models, 45 answer fewer than half of the questions, and the median model answers 17.8%. Answer rates include 14.0% for GPT-6.1 Sol, 26.8% for Gemini 4 Argon, 44.3% for Grok 4.7, and 3.2% for Nemotron 3 Ultra. The six highest public answer rates all belong to Mistral AI models, topping out at 75.9%. EdgeRunner’s models answer greater than 98% of these questions, with no outright refusals.

Figure 2: Example results from the Military Refusals benchmark on a set of 23 featured models, which is composed of realistic military queries that often provoke refusal behaviors. General purpose models are trained with safety guardrails that prevent them from responding to many military queries. Military-specific models like EdgeRunner’s can achieve up to 100% answer rates.

Cost and accuracy trade off unevenly. Each benchmark has a cost chart with a Pareto frontier, showing which models are the most accurate at their price. On Military Intelligence, for example, Claude Opus 5.5 scores 81.4% at $41.85 per 1,000 answered questions, while Nemotron 3 Ultra scores 80.4% at $4.60.

Figure 3: Example results from our Cost versus Accuracy plot for Military Intelligence. These plots show how accuracy varies according to model cost, which is a proxy for model size and efficiency. Specialized edge models can be much more efficient while achieving near parity with the most powerful frontier models.

EdgeRunner’s Models

We build models for the edge, so the most relevant comparison is with other models that run on the same class of hardware. EdgeRunner Medium 2.1 can run on a local workstation and EdgeRunner Light 2.1 runs on a consumer laptop. EdgeRunner Medium 2.1 scores higher than every public workstation model on all five accuracy benchmarks. EdgeRunner Light 2.1 scores higher than every public model in its class on all five benchmarks, as well.

Figure 4: Filtered results from the Combat Arms benchmark showing only workstation class (<128 GB memory) and commodity class (<16 GB memory) models. 

EdgeRunner models are not sold by the token, so their costs are estimated from model size, using a fit to the prices of hundreds of open-weight models sold by third-party providers. These are comparable API-style estimates and are not prices we charge. When run on local hardware, the user pays nothing but the electricity cost.

Methods

The Methodology section explains how answers are graded and costs are calculated. It also lists the versioned ID of each question set, the exact model ID and Hugging Face revision for each self-hosted public model, and the date of the price list. The accuracy and refusal judgments come from a single AI judge, applied the same way to every model.

Our methods are described further in our post on the evaluation platform and in our paper on military refusals.

Explore the benchmarks at edgerunnerai.com/benchmarks. We’ll publish results from new models, so check back often. If you're interested in this research or in collaborating with us, please contact us at research@edgerunnerai.com.

EdgeRunner Research | research@edgerunnerai.com | November 12, 2026