Making a music video 15 seconds at a time

“Halting Problem” (track 6 of Wavefunction Collapse) is getting a music video. Mostly I wanted to see what the new video models are capable of, and the honest answer so far is: the results run the gamut between hilariously bad and actually pretty good, sometimes within the same generation.

The song is 6:36. Video models top out around 5 to 15 seconds per generation, and this one only accepts specific frame counts, so you don’t get to pretend you’re directing continuous footage. You get to direct 33 tiny films and make them agree with each other.

The pipeline

Everything hangs off two files. Whisper aligns the lyrics against the mp3 and writes per-line timestamps, and storyboard.yaml is the source of truth: one entry per segment with start and end (snapped to the model’s frame grid at 24 fps), the lyric lines it covers, the prompt, the reference images, and the chosen take.

The 33 segments group into ten scenes. Each scene opens with an audio-synced generation that gets the character reference images, then continues by chaining: extract the last frame of a segment, feed it in as the first frame of the next, no refs, no audio conditioning. Hard cuts only happen at scene boundaries, so within a scene the footage flows like one shot. The original mp3 gets overdubbed at assembly with frame-exact trimming; whatever audio the model hallucinates is stripped.

Consistency is where most of the effort goes. The video has three visual registers: a clinical white void where the narrator is a translucent body of light particles (the room’s light dips on every kick drum, like the world is sidechained to the mix), grainy 16mm memory footage shot as first-person POV, and deep space with a black hole doing the metaphorical heavy lifting. Character identity comes from reference stills, which have their own little pipeline with swappable image backends.

Local or hosted

The model is MiniMax H3. The open-weights base runs on my two 3090s through ComfyUI, and the same model is hosted by fal, Segmind, WaveSpeedAI, and friends at $0.05 to $0.26 per second of output. So I benchmarked.

At my electricity rates, local comes out roughly 4.5x cheaper on marginal cost. Hosted is about 20x faster wall-clock: a full draft pass that would take the rig around 19 hours finishes in under an hour when you fan it out against an API. And local tops out at 768p, so if I ever want a 2K master, that pass has to go hosted no matter what my feelings are.

The conclusion I landed on: generate locally by default and spend on the API when what I need is time rather than savings. When I did buy time, it worked: a persistent submission engine ran a complete parallel pass through WaveSpeed, 189 generations for about twenty bucks, without a single failure. Drafts render at 864x480 and only the keepers get regenerated at higher resolution.

The gamut

Some segments came back beautiful: the light-particle narrator drifting in the void looks like money I did not spend. Others, less so. One take rendered the protagonist without pants, which the storyboard did not call for. Faces wander between takes, hands do what AI hands do, and physics is more of a suggestion. That’s what the take system is for: generate several per segment, rate everything 0 to 5, delete the zeros, and pin the winner in the storyboard.

Editing with an agent driving

For the cut I tried the browser editors (OpenReel, OpenChatCut) and then admitted I needed a real NLE and installed DaVinci Resolve, which I had never opened in my life. An agent drives it over MCP while I art-direct: it built parallel timelines from the different render passes, ran the triage pass with me, and is now assembling a multi-track edit that samples between up to four alternate takes per shot, with the cuts landing on beat.

Draft cuts exist and they’re already fun to watch, partly for the good parts and partly for the failures. When it’s done it goes up with the track.