Defense Software Firm Publishes Public AI Model Leaderboard, New Chinese Entry Debuts at #3

VigilSAR Defense LLM Benchmark — The public benchmark page — aggregate results public, task set private. Source: vigilsar.com

VIGILSAR (https://vigilsar.com/) has taken a bold step in the AI world by releasing a public leaderboard that ranks various language models based on their ability to handle intelligence, surveillance, and reconnaissance tasks. Unlike typical AI rankings, this one is specifically designed to measure models’ trustworthiness and restraint in sensitive scenarios, rather than general trivia or broad capabilities.

The current setup includes 14 models evaluated across 300 tasks, with results scored on July 17, 2022. Importantly, the task set is private; the models are tested against a secret collection, ensuring they cannot simply memorize answers. A public leaderboard displays aggregate results, while a hidden, held-out set keeps the true performance of each model under wraps, with the difference between the two scores serving as a flag for memorization issues.

Leading the pack is Claude Fable-5, holding a score of 67.77 and firmly in Band A. A notable newcomer, Kimi K3 from Moonshot, debuted at #3 with a score of 64.65, placing it in Band B. Remarkably, Kimi K3 outperformed all GPT variants and Gemini models on this test, highlighting the rapid rise of Chinese-developed AI in this space.

The rankings are based on confidence intervals and bands rather than precise ranks, emphasizing the uncertainty inherent in AI evaluation. The site also publishes data on the cost-per-correct-answer and whether models are deployed in real-world settings, reflecting the practical considerations of model usability and safety.

According to the site, “vendor claims are not evidence,” underscoring that the evaluation was built by independent operators who aim to compare models on a fair and transparent basis. They stress that their goal is to see which models can truly meet the rigorous demands of intelligence work, not to promote any specific vendor.

For readers interested in the details, the public leaderboard offers a snapshot of the current standings, while the full scoring methodology and model economics are available on the Vigilsar site. This ranking is a rare peek into how AI models are measured for real-world, high-stakes applications, with an emphasis on honesty, transparency, and practical deployment considerations.