Rotoscoping Software Review

Learn about the different rotoscoping softwares and there features.

Summary of Roto Tools

After Effects

After Effects was the first tool to bring professional compositing motion graphics and effects functionality to the desktop. After Effects was originally developed by CoSA, then acquired by Aldus, which in turn was acquired by Adobe. After Effects had very limited rotoscoping tools in earlier versions, with only one rotospline and no paint tools, but this is slowly changing. Version 4 added multiple rotosplines for cutting mattes, version 5 added vector paint, and version 6.5 has added cloning tools and tracker advancements.

Today, AE is one of the most popular tools in the visual effects world. Known as a swiss army knife of tools, it is able to both generate rotos and import splines from programs such as Mocha. (see below)

In AE CS5/5.5, the RotoBrush tool allows user to draw a rough brush stroke to create a fairly accurate outline of the shape that needs to be extracted. It can track the shape over a range of frames defined by the user. The matte can be further tweaked by adjusting the values in the Roto Brush settings.

Here is an example. (sidenote: ironically the image here is the original optical printer at ILM – used for combining hold out mattes for many of the greatest optical vfx films ever)

AE has clip tools not just frame based like P/shop

AE is now rightly one of the most popular programs worldwide


Autodesk’s Advanced System, which includes Flame and Smoke, runs on Linux and MacOSX workstations and ranges in price from $60,000 to over $300,000. These products offer a complete post-production solution, including powerful rotoscoping tools. The painting and cloning tools include point and object tracking but no planar tracking. The rotosplining functionality is good, though not quite up to par with Mocha or Silhouette. In the most recent releases these tools have been expanded to include stereo workflow. But Flame and Smoke use roto to support other functions such as keying and will often have roto work done on cheaper systems and then loaded into Flame/Smoke. If a facility already has Flame software, for example, then they may add Autodesk’s Flare (software only) to do roto and pre-build Batch and the Action 3D compositing setups.

Top view is Flame’S answer to planar tracks, below roto is controlling lighting

The newest version of Flame also provides Autodesk’s answer to planar tracking by allowing perspective planes to be corner tracked and thus provide a correct 3D plane in Flame’s 3D compositing environment. This allows for rotos on that plane to achieve a similar effect to say Mocha’s planar track. (As seen in the fxguidetv preview of flame 2012×1 Ep115)

Flame’s roto shapes or GMask shapes have advanced now to a point that they can be used to control ‘volumetric’ lighting effects i side Flame’s 3D compositing environment.

Digital Fusion

Eyeon’s Digital Fusion started in Sydney and moved to Toronto, Canada. At one stage a version of Fusion was provided with Alias 3D, but today eyeon has gained one of the strongest positions in NT/Windows desktop compositing solutions. Eyeon has multiple products including Digital Fusion.

Digital Fusion 6 is eyeon’s flagship product. Fusion is a popular choice amongst sections of the stereo conversion roto community. The tools are direct and can be applied to the problems of depth map and shape isolation for stereo conversion.

Fusion 6.2 (build 897)

Fusion roto tools include two mask roto types, the Bspline and polyline masks. The roto tool inside Fusion is actually called the polygon tool.

What most users like about Fusion is the workflow, and the speed it affords users. The UI is thought to be very intuitive and fast in practice. It should also be noted that Fusion – like some other products – also works well with a Mocha copy/paste workflow.

Eyeon also offers Rotation which is designed specifically to work with Fusion for film pipelines. Roation’s tools has been defined for the exacting demands of rotoscoping, keying and retouching, and recently expanded for stereo conversion projects.  The integrated scripting in Rotation allows larger film facilities to use this to create roto mattes and clean plates as part of a film workflow.

Eyeon also announced at IBC 2011 ‘Eyeon Dimension’, which offers additional specialist stereo tools.

Fusion, like Nuke and Flame, is a powerful 3D compositing environment. This allows the use of camera mapping and camera projection techniques to remove items that once would have required multiple frames of roto, by rotoing/painting a single frame and then adding that clean ‘card’. By piping this clean plate into a tracked Fusion camera solution, this single roto/paint fix can solve the problem for perhaps the entire clip.

These types of solutions normally involve a 3D camera solution and the can be extremely effective when the lighting is not changing dramatically, even for shots with camera motion blur or hand held shots.



As discussed above, The Foundry led the pack in developing roto tools for dealing with stereo shot material.Avatar and other stereo sourced films lead to great advances in The Foundry’s toolsets and both ILM and Weta used Ocula in their stereo pipelines.

Nuke’s base roto tools however have not been as cutting edge, until recently, and this has lead to pipelines with specialist programs such as Mocha feeding Nuke compatible splines. Roto is done on a specialist roto station and then the splines are exported and can then be adjusted and modified further easily in Nuke.

Nuke does however have the depth of film handling tools and major keying infrastructure needed to solve the most complex film compositing problems in the world.


Prior to version 6 of Nuke the roto tools left a little to be desired. Since the introduction of the new RotoPaint tool in Nuke version 6, the application now has a fully featured Roto and Vector Paint solution, including the ability to use both Beziers and B-spline shapes. The  new version of the application supports full graph editor (f-Curve) representation of all aspects of each individual point, which means that you can drill down to tweak any animation curve for any point on a shape. Additionally, trackers can be linked to either points, shapes or layers containing multiple shapes. Crucially, every tool supports stereo pairs or SXR sequences and allows the user to dictate whether they are working on left, right or both eyes per shape.

In Nuke 6, The Foundry completely rewrote its Roto and Paint toolset, combining the two tools into one RotoPaint node. For speed, a separate Roto node was also made available.  The new tools enabled multiple splines and paint strokes to be combined under a single node along with the ability to add per-point feathering, per-point tracker control, spline edge colouring, clone/offset/time-offset capabilities, spline or stroke life time controls and motion blur. Paint strokes and Roto splines can also now be grouped into Layers enabling a more parent – clild relationship, ideal for rotoing characters, etc.

The output of the RotoPaint / Roto node can be fed directly into Nuke’s multi-channel system enabling mattes to be stored for later use in the main composite instead of overwriting the existing Alpha channel.  The Roto splines along with Paint stokes can also store RGB information making them useful for not just matte creation but also patches, vector graphics and the like.

The Foundry have listened to user requests and have included both a Dope Sheet and a Planar Tracker in recent builds of Nuke. These are a welcome addition to anyone doing any serious roto in the application. As the Dope Sheet often dramatically simplifies making timing changes to complex shapes of groups of shapes. The Planar Tracker is also a very helpful addition to the Roto toolkit in Nuke. Nuke’s planar tracker is entirely tied in to the Roto node, relying on Rotoshapes to define areas to be masked or tracked and then outputting Planar transforms to the associate Layer in the Roto shapes list. One really nice new feature is the ability to link Planar tracker maticies to other Rotoshapes layer or Spline or Grid Warpers which can prove very helpful in complex warping and morphing.

In practice Nuke’s Roto feels somewhere between either Silhouette or say Autodesk’s old Combustion, which is not a bad thing at all, but it can tend to bog down on really large scripts, according to a film pipeline specialist we spoke to, who works on major Hollywood Nuke production pipelines.



Silhouette is a specialist tool that has recently had a make over. The product aims to produce a valid matte output from any method or combination of technologies but it is very much written as a pipeline tool.

Power Matte, from a 5D Epic clip fxguide shot.

Power Matte is a newer tool in Silhouette v4 that aims to reduce the need for spline based hand done rotos.  It is an easy to use interactive matting tool capable of extracting almost any object in an image, even when dealing with fine hair detail, smoke, or reflections. The process does require some manual input for object identification. This extraction process creates a matte including soft transparency. Once a matte is extracted, the foreground object can be seamlessly composed onto a new background, but the matte is pixel not vector based. Vector roto shapes can be imported or combined with the Power Matte shapes, allowing for important garbage matting of say a face, if the Power Matte is automatically producing a hair matte.

New Planar tracker in 4.1

If manual tracking of hair is required, open shapes or lines effectively can now be roto’d. Most shape programs require a spline to be close, making long waving thin lines extremely hard to roto, as the splines fold and produce huge exceeding erroneous bubbles or loops all too easily. This tool (v4.1) is relatively new and thus still rough, it shows great promise and hopefully will improve in coming releases.

Also new in the v4.1 release is a new re-written planar tracker.



Even organic objects can be helped by planar tracking

Key to the fast way that mocha, from Imagineer systems, works is the notion of the planar track. A planar track that moves in perspective can be tracked and then attached to that planar perspective move the roto can be applied. With Mocha, the roto does not need to be point tracked or shape tracked, but the plane that the roto sits on is tracked and then the roto inherits this movement in 3 space. So a roundish object that would not respond to an object track directly, does work when its roto plane is moved in perspective, a perspective moved derived perhaps from the object it is attached to. In a sense, you know a lot about how the shapes in a room will move if you know the perspective planar shifts of the walls or floor. If a wall is seen in perspective for example, the door close to camera may make an excellent source of planar tracking data, but the actual movement of the wall would not apply well to a person leaning on the wall (further away in perspective down the wall). A literal track of the roto linked to the close door would move the person’s roto too much as the door has more parallax movement, but the roto of the person will move with the same perspective also seen in the wall and closer doorway. Thus the roto will move correctly in the scene based on the planar track, and the artist is free to also animate the outline shape of the person as they move or talk etc.

Object tracking is much more valuable to a roto artist that point tracking, point tracking produces high frequency jitter and often erroneous results that make the roto little better than hand painting per frame. This jitter is both wrong and highly distracting. An error or inaccurate edge handled smoothly, will be far less likely to be an issue that a boiling or jittering edge. Shape tracking also handles partial occlusion, and minor foreground issues like hairs, smoke or dust.

For all images here: click on the image for larger version

mocha exports easily to AE and Nuke, with the shapes and splines rather than just the rendered shapes being able to then be used in Nuke or AE directly. This is a key feature to mocha working in a modern pipeline and is a major selling point for this popular solution.


The product is available in several forms frommocha Pro to mocha and mocha AE.



The most ubiquitous graphics application in the world was probably the first digital rotoscoping tool to be used in film and video post production. Though Photoshop was initially intended for still images, it can work with motion by importing frames one at a time or importing filmstrip files from video applications. Photoshop’s brush engine is the benchmark everyone else strives for, and gives excellent control when using pressure sensitive Wacom tablets.

The biggest drawback is a lack of a real time preview of sequential frames. Photoshop lacks temporial tools, in short it is great on a frame and poor when working with a clip. You will not know how well your shapes are working out until you play back your clip in realtime at full resolution. After painting numerous frames in Photoshop, the sequence must be brought back into an editing or compositing application such as Final Cut Pro to see realtime playback and you cant export splines from Phtotoshop into say Nuke’s native spline format. So the result is only a ‘rendered’ output and this is a painfully slow way of working. And since it isn’t intended for film, it lacks advanced travelling matte capabilities, LUT & viewing tools and motion or object tracking.


Other older discontinued products



Shake has three options for roto; Quickpaint, Quickshape, and Rotoshape. Quickpaint is a procedural paint package inside Shake. You can paint frame by frame and then view in realtime or paint with interpolation. As all the paint elements can be animated over time it is a reasonable roto tool. Quickshape is a basic roto tool, somewhat now completely over shadowed by Rotoshape.

Rotoshape allows variable edge softness and logical operations between roto shapes. The rotos in Rotoshapes are classic spline shapes with complex parent child relationships – and velocity based motion blur. For complex rotoscoping this gives very accurate results. Both Rotoshape and quickpaint can use shakes 2D trackers. It is worth noting that given Shake is a node workflow model it is possible to paint or roto through a track or image transform.

Shake is not a completely dead product as the source code was bought by several large facilities with some of those extending the product’s code base and thus its working life. But Nuke has been taking over from Shake in nearly all respects.

Curious gFx Pro

gFx was a relatively new product that showed great promise on the Mac OSX. Unlike other paint programs it is designed around a strong user interface that fully embraces moving footage, as such it could import, composite, track, or stablise footage easily. The spline shapes could not be exported and the product does did not fully import Photoshop files, yet Adobe on April 17, 2005 announced an agreement to license, develop and distribute the rotoscoping technologies of Curious Software. Adobe said then that it intended to use the technology to expand Adobe’s professional capabilities in its  own software. To date these tools do not appear to have explicitly reappeared in any Adobe products, the most likely candidate being AE. One of Curious’s founders was the man behind Parrallax, and it shows in the the depth of tools it launched with: 16bit raster paint with an excellent brush engine, and b-spline rotosplines with an excellent transform points UI, motion blur on splines, grouping splines, selective edge feathering (ie. advanced gradient), and more. The rest of gFx was sold, as best we can understand, to VizRT. (


In 1997, Discreet acquired Paint and Effect from Denim Software. Paint offered a vector based painting and cloning system for Mac and PC, while Effect offered compositing capabilities. Discreet re-designed the interfaces to make the applications more Discreet like, and merged the two applications into Combustion. Along the way, they also replaced some of the core functionality like Keying, Color Correction, and Tracking with the same tool set found in Discreet’s Advanced Systems. Combustion 2.0 added additional Advanced Systems features, including the same rotosplines found in Flame. Combustion 3.0 took the product even further with an edit operator, flash output and much more, most significantly a flow diagram UI feature that many users feel more comfortable working with. Combustion roto spline files can be opened directly in the larger Inferno/flame/flint products.


Developed by Industrial Light and Magic Visual Effects Supervisor Scott Squires, Commotion was used for years at ILM before Scott formed Puffin Designs and released it to the public. Commotion, then called Flipbook, was often sighted at ILM and mistakenly referred to as the “secret ILM motion version of Photoshop”. Though Commotion looked very similar to Photoshop in some respects, Commotion’s interface and tools were designed for moving images, and was the first tool on the desktop to offer realtime RAM based playback. This realtime core functionality was the foundation for all of the roto tools added as the product developed.

Importantly, Commotion curves could be exported and imported into After Effects, a move years ahead of its time. Advanced roto tools included raster based paint, spatial and temporal cloning, wire removal tools, auto-paint, unlimited bezier and natural cubic b-splines, motion blur on rotosplines, and a very fast and accurate motion tracker. Commotion quickly became the de-facto roto tool in the industry, replacing Matador in most post facilities. Puffin Designs was acquired by Pinnacle Systems in 2000, but sadly development stopped. The group was then sold to Avid and again the code sat on the shelf, sadly to a point that one doubts there would be any use in revisiting it.  It is understood that Scott Squires did investigate a few years back trying to revive the product but sadly nothing came of it.


Matador was originally developed by Brittish developer Parralax, and acquired by Avid along with Parralax’s compositing application Illusion. Available only on the SGI platform and priced around $15,000, Matador was one of the first digital rotoscoping tools which gained a wide acceptance in the film post production pipeline. Matador started as a tool made for editing still images, so many of the tools used for motion work were not well thought out. Matador provides excellent matte creation tools including b-splines, motion tracking, and a full set of painting and cloning tools, with full 16bit/channel support. Avid stopped development of Matador in the late 90’s. The original developers tried to spin it off into a new company called Blue, but that never took off.


NewTek is mostly known for their 3D application LightWave. Aura was a stand-alone paint application designed for film and video. It wasn’t widely accepted in the industry, and was mostly just used by LightWave users to finesse 3D renders. Some advanced features included a 16bit/channel paint engine, and auto-paint. NewTek   stopped supporting the program and as of June 2003.

Roto DV

Originally developed as a product named “Roto” by a failed start-up company called Post Digital, Roto DV was aquired by Radius, which later turned its name into Digital Origin, and then was aquired by Media100. Though it was called Roto, it actually didn’t have very sophisticated roto tools, and the ones that were actually pretty cool never made it into the shipping product. Media100 has no information on their website about this product, so we assume it is no longer developed or supported.