The Second Half of Video Generation: Learning the Process Behind the Pixels.

“Seedance can generate remarkable footage, but that’s still a long way from making a film.”

That’s what filmmakers told me last month at an AI film studio, where I'd been working as a technical supervisor after my PhD in AI.

“Sometimes it gives us something better than we imagined. But it rarely gives us exactly what we want. We still have to work out what to keep, what to change, and how it all fits together.”

They didn’t want an AI that directs the film for them and hands back a result. They wanted a partner that could understand why a take wasn’t working and remember what they had already tried. The real filmmaking happened in the back-and-forth: rewatching takes, comparing versions, playing the cut to decide what to try next. The model was never part of it. Each time, it saw only the next prompt, with no idea what came before.

The first half: learning from finished work

Today’s AI is great at generating footage because footage is all it has seen. It can’t help with the creative process because it has never seen one.

Learning from finished footage isn’t a mistake. It’s easy to collect at scale, which is why the field has moved so fast [1]. Over the past few years, my research has been part of that progress, helping bring image and video generation to where it is today [2].

It started early in my PhD at Carnegie Mellon, with one piece of advice from my advisor, Deva Ramanan (who built COCO [3], one of the most widely used datasets in computer vision): if you want to understand visual generation, talk to the people who make visuals.

I took that advice seriously. I went on to work with more than 300 creators, including filmmakers, game designers, and visual artists. The challenges they faced in their work shaped my research in generative AI.

For example, the artists I worked with wanted AI to make what they had in mind. To measure how close it came, I trained generative models on real images to judge generated ones. This led to VQAScore [4], which outperforms OpenAI’s CLIP [5] (cited in Google DeepMind’s Imagen 3 [6]), and GenAI-Bench [7], the sole image generation benchmark used in Nano Banana’s evaluations [8].

With cinematographers, we went further. We built video models that learn the language of film from real films: how the camera moves, whether it looks up at a character or down on them, whose eyes we see the scene through, and when focus pulls from one face to another [9][10].

Today, this research is used by more than 100 AI labs pushing the frontier of image and video generation, including Google DeepMind, SpaceXAI, and ByteDance.

You can see where that frontier stands in models like Seedance 2.5 [11], which can already generate convincing 30-second clips. Soon, models may generate entire films from a single prompt. But that is only the first half of video generation [12].

The second half: learning how films get made

Ambitious creators want to shape a film, not just receive one.Sometimes a happy accident in one take points to a better idea. Sometimes the rough cut shows you what the script missed. You often only discover what you want by working on it.

The second half of video generation is about that process, and it calls for a creative partner, not a better generator. With professionals, it learns their taste and follows their lead. With everyday creators, it guides and inspires. And it remembers every choice along the way, so no one has to explain themselves twice.

In “The Second Half” [12], Shunyu Yao argues that AI must now be measured on real work, which is interactive and builds over time. Creative work is one of the hardest but most valuable tests.

So the second half starts with recording the process. We can’t evaluate or improve what we’ve never recorded.

Toward Recursive Creative Intelligence

There’s a lot of talk about recursive self-improvement (RSI) [13]: AI learning to improve itself. Mathematics has written proofs [14], while software development has GitHub histories of code changes, discussions, and reviews [15]. In both, the process is recorded alongside the result. That record teaches AI how the work is done, and how to do it better.

Content creation has no such record. The internet is full of finished work, from images and videos to music and books, but almost none of the process behind it. Drafts get deleted, and the reasons for changing direction are rarely written down. Collecting more finished work won’t fill that gap.

Creative work needs a comparable foundation.

What makes this possible now is that content creation is moving into digital environments, with AI involved from start to finish. In traditional filmmaking, much of the process happened on set and in conversations no one recorded. Now it can happen in one place, often in days rather than months. We have an opportunity to support that work while keeping the ideas, attempts, and feedback connected.

That’s why I built Moodio: a multimodal environment where people and agents create together, and where agents learn from every round of that work and get better. I call this loop Recursive Creative Intelligence (RCI).

Filmmaking is one of the richest places to begin. It brings writing, imagery, sound, editing, and collaboration into one of our most demanding creative processes. An agent that learns to contribute across that process could eventually help with far more than film, from games to entire interactive worlds.

If we want AI to learn from creators, we first have to build something they want to create with. As our advisor Darcy Antonellis, Warner Bros.’ first female CTO, reminded us, that starts with understanding their world. And their process should stay theirs. Agents should learn from creators to work for them, not to replace them.

We’ve kept paintings on walls and films that outlive the people who made them. But we’ve rarely kept how they were made. Now, for the first time, we can.

The first half of video generation taught AI to make footage. The second half will teach it to make films with us.

That’s why I’m building Moodio.

Acknowledgements

Thanks to the Moodio team—Ryan Rao, Yanru Li, Derek Mu, Jerry Yang, Koi Li, and Thomas Tang—for their editorial support.

References

[1] T. Henighan et al. Scaling Laws for Autoregressive Generative Modeling. arXiv, 2020.

[2] Z. Lin. Scalable Oversight Across Generative Visual AI: Toward Visual Storytelling for Everyone. PhD thesis, Carnegie Mellon University, 2026.

[3] T.-Y. Lin et al. Microsoft COCO: Common Objects in Context. ECCV, 2014.

[4] Z. Lin et al. Evaluating Text-to-Visual Generation with Image-to-Text Generation. ECCV, 2024.

[5] A. Radford et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.

[6] Imagen Team, Google. Imagen 3. arXiv, 2024.

[7] B. Li et al. GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation. CVPR SynData Workshop, 2024. Best Short Paper.

[8] Google DeepMind. Gemini 2.5 Flash Model Card. 2025.

[9] Z. Lin et al. Towards Understanding Camera Motions in Any Video. NeurIPS, 2025. Spotlight.

[10] Z. Lin et al. Building a Precise Video Language with Human–AI Oversight. CVPR, 2026. Highlight.

[11] ByteDance Seed. Seedance 2.5. 2026.

[12] S. Yao. The Second Half. Blog post, 2025.

[13] I. J. Good. Speculations Concerning the First Ultraintelligent Machine. Advances in Computers, vol. 6, 1965.

[14] The mathlib Community. The Lean Mathematical Library. CPP, 2020.

[15] C. E. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR, 2024.