Cheap models are the ones people actually put in products. They look at thousands of pictures a day, so the price per answer matters. “Cheap” hides a huge range of quality.
Under a dollar, a five-fold spread
Four of the 36 models cost under a dollar per 1,000 answers at standard prices. This is what they scored:
| Model | Maker | Cost per 1,000 answers | Score |
|---|---|---|---|
| Perceptron Mk1.5 | Perceptron | $0.20 | 28.8 |
| Mistral Small 2603 | Mistral | $0.26 | 12.5 |
| Gemini 3.5 Flash-Lite | $0.34 | 40.3 | |
| GPT-6 Luna | OpenAI | $0.74 | 60.7 |
The cheapest of the four, Perceptron Mk1.5, is not the best. Mistral Small 2603 costs 26 cents and scores 12.5, close to a guess. Gemini 3.5 Flash-Lite costs 34 cents and scores 40.3. GPT-6 Luna costs about twice as much as Flash-Lite and scores 20 points higher. Perceptron and Mistral could not take our two video tests, so their scores average the tests they could take.
US labsChinese labsOpen models via APIOther labs
Bars are the overall score out of 100; the whisker is the 95% interval. The figure at the end of each bar is the cost per 1,000 answers.
GPT-6 Luna beats nine of the 11 Chinese-lab models
Luna scores 60.7. Nine of the 11 models from Chinese labs score between 37.9 and 56.4, and they cost between 2 and 17 times as much as Luna: Qwen3.8 Flash $3.04 (56.4), GLM-5V-Turbo $12.50 (46.8), DeepSeek V4.1 Flash $1.43 (44.4), GLM-5.3 Flash $1.40 (43.2), MiniMax M3 $3.53 (40.9), Step-5 (preview) from StepFun $12.50 (40.7), MiMo v2.6 Pro $4.60 (40.3), Seed 2.1 Turbo from ByteDance $10.85 (38.3) and Hunyuan HY4 from Tencent $5.71 (37.9). Luna beats every one of those models at about half the price of the cheapest of them. It also matches Gemini 3.8 Flash (61.9, $12.80) at about a seventeenth of the price.
Two Chinese models do beat it, both with 70.6. Qwen3.8 Max from Alibaba costs $27.40, about 37 times Luna’s price. Kimi K3.1 Pro from Moonshot costs $71.79, about 97 times.
Alibaba’s open model Qwen3.8-27B, run through an API, scores 59.0 for $7.75. That is level with Luna within the margin of error, at about ten times the price. It was tested on a smaller sample, so its margin is wider. Google’s open Gemma 4 31B costs $1.94 and scores 38.8.
When to pay a little more
Claude Haiku 5.5, just over the dollar at $1.01, scores 63.5, slightly above Luna’s 60.7, at about 1.4 times the price. The real step up is GPT-6.1 Sol at $7.94 (81.3) or Gemini 4 Argon at $6.52 (77.2). That is roughly nine to eleven times Luna’s price for 16 to 21 more points. Beyond that, returns fade: GPT-6 Astra scores 83.7 for $38.32.
What a cheap model is good and bad at
Luna scores 99 on saying “I can’t tell” when a picture hides the answer, 92 on a parking-rules question and 91 on checking an invoice. It is weak at counting: its Counting category score is 35, against 76 for GPT-6.1 Sol. It also has trouble telling AI-made pictures from real ones (24). Flash-Lite does best on “can’t tell” questions (89) and absent objects (86), and is middling to poor elsewhere. A cheap model is fine for a yes-or-no check and risky for anything that needs an exact count.
On the “Not being fooled” category, Luna scores 73. GPT-6 Astra, which costs 52 times as much, scores 71. In that category, paying more bought nothing.
How we tested
Each model answered the same questions about the same private pictures, in the same format. Answers are scored automatically against an exact key, then corrected for guessing, so 0 means no better than a guess and 100 means always right. Cost is the standard list price of the tokens used for 1,000 answers. Real test items are never published.
Tested: GPT-6 Luna 5–6 Nov, Gemini 3.5 Flash-Lite, Perceptron Mk1.5, Mistral Small 2603, Claude Haiku 5.5, DeepSeek V4.1 Flash and GLM-5.3 Flash 6 Nov, Qwen3.8 Flash 6–7 Nov, GLM-5V-Turbo and MiniMax M3 6 Nov, MiMo v2.6 Pro 6–7 Nov, Kimi K3.1 Pro and Qwen3.8 Max 7–8 Nov, Qwen3.8-27B, Step-5 (preview), Seed 2.1 Turbo, Hunyuan HY4 and Gemma 4 31B 7 Nov. See the leaderboard.