Machine learning (ML) in VFX? It sounds like the beginning of a Sci-Fi flick where robots take over the world, but in reality, it’s more like giving artists superpowers to tackle creative challenges that were once impossible. Let’s dive into how Rising Sun Pictures (RSP) made this happen based on details from Autodesk’s informative YouTube video.
From Sci-Fi to Reality: RSP’s ML Journey
RSP had been collaborating with the Australian Institute of Machine Learning since the early 2000s. But things got real in 2018 when they set up an in-house space for ML researchers, John Bastion and Ben Ward, to work their magic. Their first big ML project? None other than “Shang-Chi and the Legend of the Ten Rings.”
The success of “Shang-Chi” led to the rapid growth of RSP’s ML team, which now boasts 10 developers.
Thor Love and Thunder
RSP used reference for Baby Thor (the grandson of former Disney CEO Bob Chapek) to a CG baby., they were able to “Sync” their digital baby to perform specific gestures and exhibit emotions required by the script.
“One of those challenges involved ‘Baby Thor,’ introduced in an extended flashback montage. The baby is a wholly-digital character, based on the real-life infant of a Marvel executive, created through animation and artificial intelligence,” Rising Sun says on its website.
The ML Toolkit: Revise
At the heart of RSP’s ML arsenal is a suite of software called Revise. it can swap and modify entire heads, tweak hair, and even alter performances using both artist-driven and synthetic data. And we’re not talking about simple face swaps here; the team tackles highly dynamic shots plagued by motion blur, occlusion, object rotation, and ever-changing lighting conditions.
Some pretty epic projects have benefited from Revise, including “The Matrix 4,” “Black Panther: Wakanda Forever,” and “Avatar: The Way of Water.”
Overcoming the Creative Loop
Initially, the integration of ML wasn’t all smooth sailing. The ML team’s creative outputs required artists to loop back to the start of the process after every review, which led to inefficiencies and longer turnaround times—a bit like running on a creative treadmill.
To fix this, RSP implemented a smarter approach: exporting ML models and handing them over to downstream departments. This not only kept the tools in the artists’ hands but also allowed for creative iteration and stress-testing of the models. It was a win-win for efficiency and quality.
Limitless Creative Control
ML in VFX isn’t just about speeding things up or cutting costs. It’s about unlocking new creative possibilities. Need to shift an actor’s gaze? Want to tweak an eyebrow arch mid-scene? Easy. But here’s the catch: integrating ML into the existing pipeline is like threading a needle—everything needs to align perfectly with production schedules and resources, which isn’t always easy when ML models can be unpredictable.
A Case Study: “Furiosa Mad Max Saga”
RSP tackled with ML was “Furiosa Mad Max Saga.” Over a year, they worked on 300 shots, 150 of which were ML-based. This required a team of five ML developers, 25 compositors, and 21 dedicated Nvidia GPUs, with 50 TB of training data to bring the younger and older versions of Furiosa seamlessly to life.
The challenges were real: motion blur, occlusion, lighting changes, and alignment issues were daily hurdles. But the ML team, a mix of technical and artistic talent, tackled them head-on. Their success wasn’t just about cool tech; it was about problem-solving, collaboration, and a relentless focus on creative output.
During the press interviews around the film’s release Taylor-Joy has frequently praised the face replacement VFX work. On ‘The Kelly Clarkson Show’, the actor said, “It is strange to see your eyes and mouth on some one else face but George Miller had clear vision. The audience was already getting used to a new Furiosa. He wanted the transition from both actors playing her to be seamless.”
Looking Ahead
Machine learning is changing the VFX landscape, bringing with it endless possibilities and new challenges. In the end, integrating ML into a VFX studio isn’t just about plugging in some algorithms—it’s about reimagining what’s possible and pushing the boundaries of storytelling. And with ML in the mix, those boundaries are expanding faster than ever.
Watch the full Video here :
