# VLM Model Benchmark Report

**Date**: 2026-04-04
**Task**: VRT diff heatmap analysis (dashboard redesign, 1.7% pixel diff)
**Prompt**: Identify CSS property changes, affected elements, severity

> Regenerate: `OPENROUTER_API_KEY=... just vlm-bench --md <model1> <model2> ...`

## Results

| Model | Latency | Cost | Tokens | Response | Quality | Notes |
|-------|---------|------|--------|----------|---------|-------|
| **qwen/qwen3-vl-8b-instruct** | 3.8s | FREE | 806 | 812ch | ⭐⭐ best | 12 changes correctly identified, perfect format |
| **qwen/qwen3-vl-32b-instruct** | 5.5s | ~FREE | 1037 | 1411ch | ⭐ detailed | Verbose but accurate analysis |
| bytedance/ui-tars-1.5-7b | 1.1s | ~FREE | 805 | 238ch | ○ adequate | Fast, brief but correct |
| amazon/nova-lite-v1 | 1.4s | ~FREE | 1837 | 118ch | △ brief | Very short response |
| google/gemini-2.0-flash-001 | 1.9s | ~FREE | 1573 | 318ch | ○ adequate | Concise, some detail |
| meta-llama/llama-3.2-11b-vision-instruct | 4.2-8.2s | FREE | 6794 | 1462ch | ✗ hallucination | Long but incoherent, unusable |
| nvidia/nemotron-nano-12b-v2-vl:free | 6.3s | FREE | 3870 | 243ch | △ brief | Slow for its output quality |
| google/gemma-3-27b-it:free | 19.7s | FREE | 377 | 1003ch | ○ adequate | Very slow, confuses heatmap red with CSS color |
| google/gemma-3-12b-it:free | rate limited | — | — | — | — | Free tier rate limit |

## Recommendations

### Best overall: qwen/qwen3-vl-8b-instruct

- **Latency**: 3.8s (acceptable for interactive use)
- **Cost**: Effectively free ($0.00008/1K prompt, $0.0005/1K completion)
- **Quality**: 12/12 changes identified, correct format, no hallucination
- **Use case**: Default model for VRT reasoning

### Fastest: bytedance/ui-tars-1.5-7b

- **Latency**: 1.1s
- **Cost**: ~FREE
- **Quality**: Brief but correct
- **Use case**: Quick pre-screening, CI pipelines where speed matters

### Avoid

- **llama-3.2-11b-vision**: Severe hallucination with heatmap images
- **nemotron-nano-12b**: Slow for its quality level
- **gemma-3-*:free**: Rate limited, slow when available

## Cost analysis (per 1000 VRT diff analyses)

| Model | Cost/analysis | Cost/1000 |
|-------|-------------|-----------|
| qwen3-vl-8b | $0.00008 | **$0.08** |
| gemini-2.0-flash | $0.00016 | $0.16 |
| qwen3-vl-32b | $0.00043 | $0.43 |
| Free models | $0 | $0 |

At $0.08 per 1000 analyses, qwen3-vl-8b can process **12,500 diffs per dollar**.

*Generated by `just vlm-bench --md`*
