“Video understanding” is the next big thing in AI vision. Our two video tests show a split: a few models handle it very well, and most cannot do it at all.
In one test the model counts items passing along a conveyor. In the other it follows a hidden object while containers are shuffled, the old shell game. Video is given to the model as a series of frames from a short clip. Each video test has an exact answer.
Of the 36 models, 34 could take both tests. (Two cheap models, Perceptron Mk1.5 and Mistral Small 2603, accept too few pictures per question.) Here is the average of their two video scores next to their average on every still-picture test, for the leaders and a few well-known names.
| Model | Video | Still pictures |
|---|---|---|
| Claude Fable 5.2 | 99 | 80 |
| GPT-6 Astra | 98 | 83 |
| GPT-6.1 Sol | 95 | 80 |
| Claude Opus 5.5 | 90 | 75 |
| Claude Fable 5.1 | 90 | 67 |
| Kimi K3.1 Pro* | 83 | 69 |
| Gemini 4 Argon | 68 | 78 |
| Qwen3.8 Max* | 66 | 71 |
| Claude Sonnet 5.5 | 62 | 71 |
| Qwen3.8 Flash | 57 | 57 |
| Claude Haiku 5.5 | 57 | 62 |
| Gemini 3.8 Flash | 49 | 63 |
| GPT-6 Luna | 40 | 62 |
* Tested on a smaller sample, so the margin of error is wider.
Ordered by the average of the two. “Not supported” means the model accepts fewer pictures per question than a video test sends.
Six models do better on video
Claude Fable 5.2, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and Claude Fable 5.1 score 15 to 23 points higher on video than on stills. Kimi K3.1 Pro, tested on a smaller sample, is 14 points higher. One more is about level: Qwen3.8 Flash from Alibaba. Claude Fable 5.2 and GPT-6 Astra are level at the top. Astra counts the conveyor perfectly, with a score of 100, and gets every difficulty level of it right. On the shell game it scores 96.6. Fable 5.2 is the mirror image: 100 on the shell game, every level right, and 97.5 on the conveyor.
Everyone else does worse
The other 27 models are lower on video, mostly by 10 to 25 points. Fifteen of the 34 score under 25 on video, against three on still pictures. Google’s Gemma 4 E4B scores 0 on both video tests. Claude Haiku 5.5 does fine on counting the conveyor (78) but scores 35 on following the hidden object. Gemini 4 Argon is in the top five overall, but its shell game score is 52. Qwen3.8 Max from Alibaba, in the top ten overall and tested on a smaller sample, is 5 points lower on video.
The result is a very wide spread. On stills, the best model scores 83 and the median 50. On video, the best scores 99 and the median is 28. The gap between the leader and the middle of the pack is 71 points on video and 33 on stills.
What this means
If your product needs to count or follow things across frames, the choice of model matters more than for any other task we tested. A cheap model will not cope, and some expensive ones do not either. Pick from the top five or six, and test on your own footage. One caution: our clips are short and clean. Real surveillance or sports footage is messier.
How we tested
Each video test gives the model a series of frames in order, with a question. The conveyor test asks how many items went by. The shell game asks where the hidden object ended up. Both get harder in steps. Answers are exact. Scores are corrected for guessing: 0 means no better than a guess, 100 means always right. Video means these two tests only, so this is a first look rather than the last word on video. Twelve models were tested on a smaller sample, so their margins of error are wider; the leaderboard shows them. The clips are never published.
Tested between 22 October and 10 November 2026. Dates for every model are on the leaderboard.