VyroAI

Loading...

The Model Benchmark

How the public Model Benchmark ranks AI models on the same plugin and bot tasks with the same grading, and how to read the score, value and speed cards.

What it is

The Benchmark page is a public ranking of the AI models VyroAI offers. Every model gets exactly the same assignments and exactly the same grading, so the numbers are comparable. We run the rounds and we pay for them — you only see the result.

How to read it

Three cards sit above the table:

The table below them ranks every model by score, and shows how long each one took on average.

The columns on the leaderboard

Each model gets a row with:

Checks used to be shown as a count like "3/8" and was widely misread as a model giving up part-way. It never meant that: every model answers every assignment. It is the share of requirements ticked, so a high overall score with a middling Checks percentage simply means good answers that left small things out.

How a model is scored

Each assignment is a real job — build this plugin, make it do that — and the answer is marked out of 100 in two parts:

So the ranking is not a compile test: nothing is executed, and the larger half of every score is one model's judgement of another model's work.

The assignments are not worth the same

This is the part that changes what you should conclude from the table. VyroAI sells plugins and bots, not short essays, so the exam is weighted like the product:

On the standard eight-assignment set that makes build work about 46% of a model's final score. A model that writes beautiful short answers and mediocre plugins cannot climb this table.

A panel of judges, not one judge

Grading is done by a panel — one model per company (Anthropic, OpenAI, Google, as many as are configured), and the median grade is the one that counts. For the checklist, anything a computer can count is measured in code, and the panel's verdict on it is ignored — word limits, bullet counts, valid JSON, code fences, the keys in plugin.yml, balanced tags in pom.xml, and the author name, package and versions an assignment tells the model to use. That has been the rule since 30 July 2026. Every other requirement counts as met when the majority of the panel says it was.

This replaced a single judge on 29 July 2026, for a blunt reason: in the round before it, the single judge was also a contestant, and it awarded itself a perfect 10,0 on all eight assignments while giving no other model a single 10. Making a different model the judge would only have moved the bias, so now no company holds the pen alone.

Judges are drawn from the contestants — but never for their own paper. Every company on the panel grades every answer, in parallel; the one exception is that a company never marks its own model's work — its seat passes to its next-best model for that paper, or sits empty if it has no other. So nothing marks its own work, and no single judge decides anything.

Since 5 August 2026, a company's seat on the panel goes to whichever of its models writes the best code, rather than to whichever one happened to be selected elsewhere in the app. And when the panel splits down the middle on a requirement, that requirement is dropped from the marking instead of being counted as a fail — a tie is doubt, not a verdict, and one dissenting judge used to be able to cost a model up to 20 points on its own.

Everyone sits the same exam

Occasionally a provider's safety filter refuses a perfectly harmless assignment, and that model ends up with one fewer answer than the rest. When that happens the assignment is simply asked again a few minutes later. If it still cannot be graded, it stops counting for every model in the round — better a fair exam of seven questions than a table where one model answered seven and another eight.

This rule and the judging panel above it were both introduced on 29 July 2026 and apply from the first round run after that date. Anything published before it was scored the old way — a single judge, and no equalising — so compare an old round with a new one carefully.

Open a row to see the marking

The table is a summary — the interesting part is underneath it. Click a model's row and it expands into one card per assignment, showing:

That is where a ranking becomes useful: you can see whether a model lost points on something you care about or on something you never do.

The three marks on a checklist, and what struck-through really means

This one catches people out, because a line through text normally means failed. Here it does not:

So a card with several struck-through lines is not a bad result — it is a shorter exam. A tie among the judges is doubt, not a verdict, and one dissenting judge used to be able to cost a model up to 20 points on its own.

Missing models

A model that could not be reached during a round is left off the page entirely rather than shown at zero. A broken connection on our side is not a verdict on the model, and showing it as a zero would read like one.

Use this with Choosing an AI Model: the ranking tells you which model is strongest, that article tells you where to switch to it.

Also in Power User

4 of 7 in Power User · Last checked 11 September 2026 · Previous: Choosing an AI Model