Sam Zoloth / Work / Asset search

Asking for all the chairs

A modeler dressing a set doesn't want the ten chairs that rank highest. They want every chair, and nothing that isn't one. I designed and built an asset search prototype around that, starting with what you type.

Recorded on the public demo with free sample models.

Type a sentence, get a set

Modelers build the props a film is dressed with, and they search a shared library for what already exists. Here you type a sentence and get back a set with a count, laid out the way you'd sort it by hand.

Read as

    Layout it wrote 

    
              
    
              
    
            
    
            
    
              
    
              
    
            
    
          
    
        
    
      
    
      
    Replayed from the running prototype on 521 free sample models. Pick a search to watch it again.
    
    
    

    
    
      
    
        
    The model decides, code computes
    
        
    Each search goes to a small model that answers a few fixed questions, each with a confidence. Is it one thing or every match? How should it sort? Should it line up to scale? When it's unsure, under 0.6, the page asks instead of guessing. Everything after that is code I can test.
    
        
    A second model writes the layout as a short program, and it can only use four components I wrote. It names assets by their number in the list it was given, so every card it places is one the search found. If the program doesn't parse, the plain grid stays.
    
      
    
    
    
  
    

    


    
  
    
    
    
      
    A ranked list isn't a set
    
      
    Embedding search ranks the whole library by how close each asset is to your words. Ask for all the chairs and you get chairs, then stools, then a sofa, then a lamp. There's no line where the chairs end.
    
      
    So the prototype draws one. A model scores each of the top 60 on whether it really is what you asked for, and anything under 0.5 drops out. I picked 0.5 by labeling 720 results across 12 searches and counting mistakes at every cutoff. Drag it.
    
    
    
    
    
    
    The thumbnails are free sample models from Poly Haven. A red ring is in the set but isn't a chair. A dashed ring is a chair the cutoff dropped.
    
  
    

  
    
    
    
      
    Say the next thing
    
      
    A follow-up works on what's already on screen. "Only the leather ones" narrows the chairs instead of starting a new search, and the model read 28 of 30 scripted follow-ups the way I would. The rest of the tool is small pieces like these, each doing one job. The last three come from search I built the same way for the previs and crowds departments.
    
    
    
    
    
    
    Each piece plays by itself. Point at one to take over.
    
  
    


  
    
    
    
      
    Fast enough to type at
    
      
    A set search runs four steps in a row. The retrieval model runs on the laptop's GPU, and after a second idle its first encode takes three to five times longer. The model reading the query is busy for a couple hundred milliseconds anyway, so the server warms the encoder in that gap.
    
    
    
    
    
    
    Stage times on my laptop. The read and membership checks vary with the query, so those bars show a typical value.
    
  
    

  
    
    
    The library is 521 free CC0 models standing in for a studio library. Back to the work