How the animation coding benchmark works

The shared task

Each model creates a twelve-second scene: a Pomeranian runs through a backyard, jumps over a puddle, lands, enters its doghouse, retrieves a bone, backs out and faces the camera. Models choose their own layout, rendering style and lighting. No reference artwork or scene template is supplied.

Seven things to look for

1. Pomeranian identity & anatomy
2. Grounded running and reverse gait
3. Puddle clearance, landing & water reflection
4. Doghouse entry & occlusion
5. Bone grip & object permanence
6. Coherent front-facing turn
7. Fur, light & final presentation

Fair settings and recorded evidence

Models accessed through Microsoft Foundry, Google Vertex AI and OpenRouter receive the same initial prompt. OpenRouter additions fix a named model and upstream provider with fallbacks disabled. The shared output allowance is 32,768 tokens, including reasoning, subject to lower provider ceilings. One execution repair is available for empty, truncated or unexecutable code. The final response is kept, not a best-of selection.

Gemini 4 Argon also received a disclosed framing correction with visibility feedback. Both responses are counted. No organizer manually edits model code. Browser checks cover execution, movement, repeatability and offline operation; they do not grade anatomy, visual realism or narrative success.

Tokens and known cost

An offline browser captures 360 frames at 960 × 540, at 30 fps. The site plays those recorded films. Provider-reported output includes reasoning once; known cost uses recorded input and output at dated API rates, or the terminal provider-reported charge for OpenRouter results. Missing receipts or prices remain unavailable, not zero.