Work in progress Private preview. This page documents an unfinished benchmark — scores, prompts and methodology are still changing.
CW/004 — LLM Visual Bench · Automated grading for one-shot visual codegen Local-first · No API keys

Does it run? Does it look right?

LLM Visual Bench asks models for the thing they're actually asked to do in a one-shot chat — produce a single self-contained HTML file that draws an animation, a playable game, an animated page — then grades the result automatically in a real browser and publishes the scorecard.

Explore prompt tests ↗ What it measures ↓
Scroll ↓
About

A small, honest benchmark for one-shot visual codegen — graded by a real browser, not by vibes.

Every run starts from the same fixed prompt. The model answers with one HTML file; the harness boots it in headless Chromium and probes what a screenshot can't show — that the moons are children of the gas giant, that adjacent gears counter-rotate, that the renderer stays under its draw-call budget. The output is a scorecard per run, a percentage per category, a methodology write-up, and a leaderboard.

It's built for the local-LLM scene: 4-bit quantisations of 27B–35B models on a laptop, with prompt tiers short enough that a 30B Q4 model can finish — because a benchmark where everything scores 2/10 ranks nothing. It runs offline, needs no GPU and no API key, and stores every artifact on the filesystem, because benchmark artifacts have to outlive the tool that made them.

Scores

What it measures

Five numbers, kept separate on purpose — craft never contaminates correctness.

01

Correctness

Weighted fraction of all machine-checked requirements that pass. Hard-fail items — a prompt's non-negotiables — are listed separately rather than zeroing the run: "70% correct but missed the point" is a real result.

Machine-checked
02

Functional

Correctness excluding the performance category — the plain "does the thing work" number.

Machine-checked
03

Performance

Frame rate, draw-call and object budgets on their own — measured while the page actually animates.

Machine-checked
04

Craft

How it looks — ranked by blind pairwise voting (Bradley–Terry) between anonymised screenshots of the same prompt, because absolute beauty scores drift between judges and sessions; forced choice doesn't.

Human / VLM vote
05

Composite

One blended number (default 60/25/15 functional/craft/performance) for the leaderboard — with the components always shown beside it.

Blended
Pipeline

How a run works

01

One fixed prompt

Nineteen public prompts in three tiers — short t1 probes a 27–35B 4-bit model can finish, t2 moderate builds, t3 stress prompts where frontier models separate from local ones. Each exists in a one-shot and an agentic variant, never averaged together.

19 prompts · 3 tiers
02

One file out

The model replies with a single self-contained HTML file — nothing else in context, sampling held fixed across models so runs stay comparable.

One-shot or agentic
03

Graded in a real browser

Headless Chromium boots the file against a pinned, vendored Three.js — no CDN version lottery — while executable per-prompt checklists probe scene structure, motion and frame budgets.

Playwright + probes
04

Scorecard & leaderboard

Every run keeps its results, report, scorecard and screenshots on disk. Leaderboard rows average over prompts and attempts, so re-rolling can't lift a score — best-of-N is reported as a ceiling, never ranked on. Rows missing prompts are flagged incomplete, not ranked.

Bootstrap CIs
Tracks

Two tracks,
one contamination check

01

Visual track

Can the model build a scene? Animations, games and animated pages, graded by the probes above.

Rendering
02

Coding track

Can the model make code correct? 18 tasks — repair, implement, multifile — answered with hidden tests copied in after the model's files, so rewriting a test changes nothing. Ranked on its own board, never mixed with visual runs.

Hidden tests
03

Holdout set

Every public prompt is on GitHub, so a good score can't be told apart from recall. Holdout prompts live only on the grading machine — same probes, separate score, and the gap against the public score reported as the contamination check.

Unpublished

Currently work in progress — the grader, dashboard and prompt set run locally; the public leaderboard ships when the methodology stops moving. The prompt test lab stands on its own — its one-off prompts will be enhanced by the visual bench set once it ships.

LLM Prompt Tests → AI Tierlist →