Rotoscoping is one of the oldest visual effects techniques. It is used to trace and create silhouette (matte) around an object or character in a scene. That can be used to extract an object from a scene for use in a different background. Rotoscoping can also be used to create a special visual effect guided by the matte. A classic example of this use are original Star Wars movies where rotoscoping was used to create the glowing lightsaber effect. The nature of rotoscoping is straightforward and repetitive. An average rotoscope artist can make 15 frames a day with clean up and tweaks. Rotoscoping is a perfect candidate to be replaced at least partially with AI machine learning.
With the emergence of technology, a service called RunwayML offers extracting objects out of the background using machine learning. Although the results are pretty nice in most cases, the edges around the extracted object are soft and not accurate enough. Hair or fast-moving complex objects are too complicated for AI to be extracted well. However, this service can be a perfect option for small or low budget projects.
In 2019 The Foundry accompanied with visual effects company DNEG and the University of Bath to create SmartROTO – rotoscoping AI tool with artist assisted machine learning. The idea was that artists would create a set of shapes and a small set of keyframes, and SmartROTO would speed up the process of setting intermediate keyframes across the sequences. The visual effects studio DNEG provided its massive library of shots with rotoscoping to train the AI. Unfortunately, after two years of research, the SmartROTO is not near to being published or finished. Foundry keeps working to create an industry-ready tool for AI rotoscoping.
Ben Kent said in his post about SmartROTO : “The main lesson we’ve learned is that rotoscoping is extremely hard. Machines certainly aren’t replacing artists any time soon. And even though we’ve got something that works with SmartROTO, it’s not currently robust enough for an artist to rely on—and artist experience is paramount for us.”
Ben continues: “The overall approach we’ve ended up taking is pretty similar to what we planned to do at the beginning. It’s still a machine learning-based tracking and shape consistency model; we’re still imagining the user sets up their initial shapes and a few initial keyframes. With SmartROTO, we’re then using this model to predict in-between keyframes, better than interpolated or tracked keyframes, to try and reduce the amount of time they spend having to finesse in the middle.”
Sources:
https://www.foundry.com/insights/machine-learning/smartroto-enabling-rotoscoping