VigilSAR is a defense-ISR software product that has published an LLM benchmark focused on the reasoning, reporting, and restraint an analyst actually needs—not general trivia. Its goal is to assess which language models can be trusted with intelligence-surveillance-reconnaissance work.
The evaluation covers 14 models and 300 tasks, with scores dated 2026-07-17. Aggregate results appear on the public leaderboard, while the task set remains private by design.
Keeping those tasks private is an anti-training-contamination measure: models cannot train on the evaluation material. A private held-out set exists on top, and each model’s gap between public and held-out scores is published to flag memorization.
The headline result is the debut of Moonshot’s Kimi K3 at #3 with a score of 64.65 in Band B. That places the new entry ahead of every GPT and Gemini row on the board.
At the top, claude-fable-5 leads with 67.77 in Band A and is pinned. The GPT-5.x family fills Bands C-D, while the Gemini rows sit in Bands E-F.
The bands matter more than the apparent order because confidence intervals inside a band overlap. By publishing bands instead of pseudo-precise ranks, along with confidence intervals and a pinned reference row, the benchmark makes uncertainty visible rather than cosmetic.
Deployment constraints also count: one locally runnable open model is scored as “sovereign-deployable”, because deployment reality is part of the score. The board also publishes per-model cost-per-correct-answer economics.
The operators’ premise is blunt: “Vendor claims are not evidence.” They built the evaluation to decide which models get anywhere near their own product and to rank the models they themselves use. They say they are not paid by any vendor and summarize the philosophy this way: “we would rather be measured than believed.”
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A RAND Analysis Tool for Intelligence, Surveillance, and Reconnaissance: The Collections Operations Model (Technical Report (RAND))
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Strategic Benchmarking Reloaded with Six Sigma: Improving Your Company's Performance Using Global Best Practice
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Large Language Models: The Hard Parts: Open Source AI Solutions for Common Pitfalls
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
