DreamWorks Animation, through Amdocs Studios
Two candidate systems, 73 searches, and an automated judge nobody had checked against an animator. This page shows what the blind grading found, and the call I made.
Pick a grade. An animator would ask one thing: would I keep this result? Grades of 3 and 4 count as yes.
<1 sec
to search video in the AI pilot, down from more than 10 seconds.
81%
voluntary adoption of the asset platform among more than 1,000 artists.
Both are results from the same engagement. The grading below is how I made the search decision behind the first one.
The first comparison produced numbers that looked like progress. I told leadership they could not support a pick, then replaced the automated judge with people who animate for a living.
1,460results graded
A first evaluation package ran 73 searches through two candidate systems: 65 with a good answer in the library and 8 built to have none. One automated judge graded every result.
Both systems returned a useful result in the top three for most searches. The numbers were easy to trust and nobody had checked the judge against an animator.
Not yeton picking a system
I wrote to DreamWorks product and technology leadership that the evidence did not support selecting any candidate. The two systems had not been tested against a library proven to hold the same files. The judge had never been compared with animators. One candidate was absent and a build-your-own prototype had not run under the same test.
I proposed a small, bounded pass instead: a handful of real animator searches, playable clips, and two animators grading without knowing which system returned what.
57clips, graded blind by two animators
Eight searches, six with a good answer and two with none. Each card showed a playable clip and a search. It carried no tool name, no rank and no machine grade. Two staff animators graded every clip from 0 to 4.
An animator's question is whether to keep a result. So the line that matters is useful or not useful, grades 3 and 4 against 0, 1 and 2. Exact scores were looser.
On useful or not, they matched on 78.9% of clips. On the exact 0 to 4 score they matched on 54.4%. Fourteen clips still need a third animator, and the way those land moves the answer between 66.3% and 90.0%.
Where the two animators split on useful against not useful, or marked a grade uncertain, a third pass is needed. Until then the labels sit just under the bar I wrote for calling them settled.
Against the main animator it agreed on useful or not in 75.0% of 80 comparisons, and on the exact score in 47.5%. Its misses are lopsided, which matters for how it gets used.
Tap a segment. 80 comparisons of the judge against one animator's useful or not useful call.
I did not name a winner. The findings stay labeled directional, and the 73-search results stay that way until the open clips are resolved.
The grading is more useful for making one tool better than for picking between two. It showed concrete things to fix and to check on every change: results that repeat, and tools that never answer "no match" when nothing fits. A script can catch both with no animator time.