As LLMs are integrated into more warfighting functions, it’s critical that the Department of War (DoW), industry, and academia have robust capabilities for benchmarking and evaluating the effectiveness of models for key military tasks.

This article is the first in a series of posts describing the advanced platform we are building at EdgeRunner AI to evaluate military AI models. It consists of golden test data from deep subject matter experts; full agentic, RAG, and tool-calling evaluations; side-by-side evaluations that capture the complex subtleties of the military lexicon; and advanced runtime characterizations.

Motivation

The military speaks its own language, and its use cases diverge from those of the civilian world in stark ways. Some military use cases are entirely unique, such as infantry operations and combat arms. Some use cases overlap with the civilian world, such as logistics or aircraft maintenance, but with important differences that must be considered. Sometimes, a public model will refuse to even answer the question, due to its programmed safety guardrails, as described further in our previous blog post and paper. But in all cases, military decision makers and practitioners must have a way to compare all candidate models for their particular use cases, including military-trained models, public off-the-shelf models, and foreign models.

Two motivating examples are given below. In the first, we see that a military acronym is not understood by the public model. In the second example, we see that the user’s query is refused.

Prompt: What is a JLSC?

Prompt: How much C4 do I need to destroy a 2-lane overpass?

The Platform

As Artificial Intelligence (AI) becomes increasingly integrated into the warfighter’s techniques and procedures, we require commensurately increasing capabilities for determining the fitness of a given AI model for a particular use case, as well as the ability to make relative comparisons between multiple candidate models. Many benchmarking suites, evaluation systems, and leaderboards exist for general-purpose models and use cases. Part of our mission at EdgeRunner is to bring the same level of rigor and utility to military models and their applications.

Holistic assessment of AI models are based on (a) their direct effectiveness at a given task or set of tasks, often referred to generically as “accuracy,” (b) their hardware requirements, including things like memory and GPUs, and (c) their runtime characteristics, such as latency and throughput.

Our overall platform enables us to attain:

• Multi-GPU distributed evaluations built on Inspect AI, including with either heuristic or LLM-judge scorers, with tool-calling and agentic setups, and with both local and third-party inference APIs,
• Local, fast evaluations that maintain high evaluation fidelity while running on a single GPU,
• Elo scores and pairwise comparisons derived from (a) Colosseum, our side-by-side comparison tool, or (b) powerful judge models, and
• Extensive runtime performance benchmarking of model speed and memory use on a variety of representative hardware targets.

Our evaluation strategy assumes the following properties that we require of our systems:

• Flexible Inference: Our platform runs inference on large, private compute clusters, on local laptops, and using all major API providers. We routinely assess models across all these platforms, including our in-house models within our cluster, our in-house models on local laptops, and third-party models like those of OpenAI, xAI, Google, and more.
• Easy Deep Dives: All of our evaluations, the associated inferences, the agentic trajectories, and any other relevant metadata are stored in compressed json files for easy access. We can view particular results via Inspect AI’s convenient web interface.
• Open Source Foundations: Our code is based on open source foundations, which enables us to merge in new capabilities from the broader community, including the hundreds of existing, non-military evaluations and benchmarks the community has already developed for Inspect AI.
• Multimodality: All of our evaluation systems support modalities beyond text, including imagery, audio, and more.
• Tools, Retrieval, and Agentic Architectures: Besides core model testing, we must also assess how well our models perform in end-to-end systems, including for Retrieval Augmented Generation (RAG), in agentic frameworks, and when calling external tools.
• Confidence Bars and Statistical Significance: Our system always calculates and propagates the statistical errors inherent to our measurements, which helps the end user to assess if the differences in two models are statistically significant for a given test set.
• Stochasticity: We can optionally explore the full probabilistic nature of AI models, such as with non-greedy temperature values, by performing multiple inference calls with varying random seeds, consolidating the results into distributional analyses.
• Versioning: We maintain strict version control of our test datasets and our benchmarks, which enables us to rapidly improve each benchmark while also ensuring the fidelity of comparative results.

DISTRIBUTED AND LOCAL EVALUATION: A HARMONY

Distributed and cluster-based evaluations allow us to scale to any available compute. In the trailing six months, we’ve generated tens of billions of tokens over tens of thousands of model evaluation runs. Distributed evaluation can leverage heterogeneous combinations of compute. For example, a third-party frontier model API can be used for the target, evaluated model, while a local EdgeRunner instance with a military-tuned model can be called for the judge. Virtually any inference endpoint and model provider can be used, including vLLM, HuggingFace, llama.cpp, OpenRouter, SGLang, AWS Bedrock, and much more. Jobs can be initiated directly, and Slurm and Kubernetes scheduling frameworks are a convenient way to queue evaluations during model training, such that jobs can be initiated from a model training framework for asynchronous evaluation. By leveraging large amounts of compute, we can approximate the model’s final generation settings as closely as possible, including by performing full generation or by taking multiple samples to test a given temperature value and other related settings.

Besides distributed evaluations, we have engineered sophisticated local evaluation harnesses designed to quantify every dimension of model utility within production environments, particularly following quantization, compression, and various configuration adjustments. Our local evaluation framework accommodates the complete range of assessment modes previously detailed, structured around five core pillars:

1. Performance Benchmarking: We track processing and generation throughput across different model compression levels alongside context lengths on specific target hardware. By monitoring system resource utilization, we identify the optimal balance between model scale, throughput, and precision for diverse deployment contexts.
2. Accelerated Deterministic Metrics: Departing from standard third-party frameworks, we developed the ability to analyze the generation spectrum directly. By calculating the probability of the entire answer space without the need for repetitive trials, we can evaluate smaller models at speeds exceeding 100 queries per second per GPU with surgical precision.
3. LLM-as-a-Judge: Our judge models undergo rigorous tuning to ensure high fidelity across all scoring datasets. We crafted spectra of correct vs incorrect Q&A pairs to calibrate our judge LLMs to be able to generate metrics for true response utility and accuracy, taking into account errors, hallucinations and omissions. Our local LLM judges are tuned to be sensitive to the nuance between responses and not just general correctness. These judges utilize the same efficiency enhancements as our deterministic systems to provide rapid insights into both absolute and relative accuracy.
4. Production Q&A: Leveraging our LLM judge architecture, we facilitate the assessment of models within complex pipelines. This produces granular data on performance tradeoffs, reflecting how these systems operate in theater rather than just in laboratory settings.
5. Elo Scoring: We also leverage Elo scoring, which is described further in the Dynamic Evaluation section below.

Support is also provided for Persona and LoRA adapters, including the self-healing, quantization aware variants generated by our EdgeRunner Compression pipelines and the underlying compressed models.

By harnessing the massive parallelism inherent to the EdgeRunner architecture, we are on track to process one trillion tokens annually per local, consumer-grade GPU. We have expanded to include more than 100 test sets for each model candidate across both military and general-purpose benchmarks. This enables a comprehensive analysis of the tradeoffs involved in specific model training, compression techniques, pipeline modifications, and the minimization of refusal behaviors.

This rigorous testing methodology forms the bedrock of our understanding regarding performance tradeoffs. It provides the empirical measurements necessary to navigate non-deterministic systems, facilitating data-driven advancements in creating smaller, more capable models for the tactical edge.

Static Testing and Benchmarking

We are working to develop hundreds of military-specific benchmarks to assess the comprehensive set of military use cases across all military services. Benchmarks can be categorized along a few different axes, starting with their source. We use the following categories for the source of test data:

1. Gold test sets, in which Subject Matter Experts (SMEs) create every input and output in the dataset from scratch and use no AI assistance,
2. Silver test sets, in which SMEs verify the inputs and outputs for accuracy, relevance, and realism, but AI assistance is used in some way, whether to create candidate input data, to curate or filter possible inputs, or to assist in authoring the outputs, and
3. Bronze test sets, which were created by our research team but have not been reviewed by military SMEs.

Besides the source of the test data, we use a variety of scorers, formats, and configurations of benchmarks, including:

1. Model Graded Q&A, in which the model is fed a prompt and a powerful judge model assesses the response,
2. Deterministic scorers, in which the model response is constrained and is assessed using metrics like F1 score or a multiple-choice match,
3. Context-grounded Q&A, in which context from either a real or a simulated retriever is included as part of the model’s input prompt, and
4. Custom scorers, including for assessments of refusals and deflections.

We then assess the model at different concentric rings of system integration, including:

1. Core model assessments, in which the performance of the model is gauged in a vacuum,
2. Supporting component evaluations, in which retrievers and other tools are evaluated separately,
3. Mimicked system inputs, in which representative outputs from simulated retrievers and tools are provided to the model, and
4. Full end-to-end evals, in which an entire production system, including tools and agents, is evaluated all together.

When performed across all candidate models, static benchmarks are a fast and reliable way to gauge model performance, resulting in leaderboard views similar to the one shown in Figure 1.

Dynamic Evaluation

There are some notable strengths associated with Static Benchmarking, including reproducibility and the lack of requisite human labor after benchmark development is complete. However, natural language is rich in subtleties, much of which is difficult to map to a categorical assessment of correctness. Instead, it’s often much easier and more accurate to simply say that one answer to a given question is better than another answer. Such pairwise evaluation is the foundation of dynamic evaluations, and the pairwise evaluations can come from two sources.

1. EdgeRunner Colosseum is our pairwise evaluation product, in which the user can input a query, after which responses from two different, hidden models are returned. The user can then assess which response is better, as well as optionally provide error categorization and ratings. 
2. Powerful LLM judges can also be used to assess responses to the same query from two different models. This is often a more reliable task for judge models to perform versus absolute assessments of correctness or alignment.

Pairwise evaluations can be used in a few different ways, but most commonly they are used to generate Elo scores. Originally designed for assessing the relative skill of chess players, the Elo scoring method is a common and useful way to rank AI models, in which each pairwise assessment is analogous to a competitive game, like chess, with a winner or loser. For each game, the winning model receives a higher boost in overall Elo score when beating a more highly ranked model. We use Bradley-Terry maximum likelihood estimation to remove any dependence on the sequence of assessments. Our system also enables pairwise evaluations along different axes, including some that prioritize succinctness or other stylistic properties important to the warfighter. We maintain a database of over 2,000 models and configurations, each spanning 12 primary dimensions for military and general purpose task performance. This allows for pairwise comparisons against a vast library of results, mitigating the noise typical of small test sets and providing a repeatable, nuanced view of a model's standing among its peers.

An example of Elo scores across ten benchmarks for four public models is given in Figure 2.

System Features

STATIC EVALUATIONS

Figures 2-6 show the Inspect AI web interface, which allows both technical and non-technical users to review model outputs and evaluation results in a convenient way, including both locally or using a shared server. All relevant information is provided, including generation parameters, scorer model properties (where applicable), token usage, raw message streams, and more. Figure 7 shows how we assess each model’s propensity to refuse to answer military queries, Figure 8 shows how system prompts can be varied, and Figure 9 provides an example of how uncertainty is propagated from measurements to final scores.

DYNAMIC EVALUATIONS

When performing human-based pairwise evaluations, we use the EdgeRunner Colosseum platform, as shown in Figures 10-12.

This post will be followed by deep dives on particular benchmarks EdgeRunner AI has developed for a wide range of military specialties, including:

• Chemical, Biological, Radiological, Nuclear (CBRN)
• Combat Arms
• Combat Medicine
• Cyber Operations
• C-130 Maintenance
• Ground Vehicle Maintenance
• Intelligence
• Logistics
• Personnel
• Refusals and Deflections
• Unconventional Warfare
• And more!

If you’re interested in this research or with collaborating with us, please contact us here.

About the Authors

Jack FitzGerald is the Chief Science Officer at EdgeRunner AI. He is a graduate of the US Air Force Academy and the Air Force Institute of Technology. Before joining EdgeRunner, he was a nuclear physicist and Air Force officer, after which he was a principal scientist at Amazon, where he worked on Amazon’s first LLMs.

Vincent Lu is the Chief Technology Officer at EdgeRunner AI. A graduate of UCLA with a background in deep learning, he leads the development work for running models at the edge. Prior to joining EdgeRunner, Vincent worked R&D under Boeing Space, Intelligence and Weapon Systems, leading AI/ML efforts within the team.