In our continuing and popular "Art of..." series, we tackle wire and rig removal. Far from being the most glamourous of visual effects, in reality wire removal is extremely challenging. We audit the current approaches and techniques of the craft. In this week's podcast, we speak with Aaron Rhode from The Orphanage about the complex 2D and 3D solutions for scenes with wires.
Techniques for General Wire Removal
1. Wire Removal By Painting Frame By Frame
The primary comment made about wire removal is "we'll just paint it out in post." But painting out on a per frame basis is extremely difficult when done over a series of frames, potentially causing the images to seem as if they are boiling. While a clone tool works well on a single frame, the lack of frame to frame cohesion means the 'fix' boils and becomes very visible once the clip is played. Of course there are times that wires are only catching the light for a moment,  and thus an isolated frame is still very effectively fixed by a manual painting, either by cloning or delicate painting. Cloning is much more effective, as it clones not just colour but grain or noise that is also almost always  present.
2. Rig Removal By Patching Over The Top
One solution using this method is based on finding a clean frame from some other point in the clip and  pasting it over the top of the offensive rig or wire. This works best on static cameras, and does not work on people in movement. Since the fill for the patch is static, considerations need to be made for grain and noise. Various tricks can used to solve this, including compounding or averaging several clean frames, reducing the grain/noise so it can be reintroduced at a later stage.
Another approach is to patch the wire and then use this new clip as a "reveal back to" clip in a manual painting environment, or roto the patch back in with finer attention to detail than the main patch. This has the advantage of replacing the minimum amount of the overall image but can suffer if there are other light changes that the 'patch' is not matching and tracking in terms of colour.
At its most extreme implementation, the patch approach becomes a 2D environment replacement which we discuss later.
3. Stabilise and Paint Back
One approach to solving a moving shot is to stabilize the image, fix it, and then invert the stabilization. If a wire is moving relative to the background, this approach can be very effective. Once the background is stable, the artist can paint back to a previous frame where the line or wire isn't seen. So unlike painting back to a still as in the patch version discussed previously, the artist paints back to moving clip offset in space to the foreground. This 'moving reveal' can be slightly noticeable when done at this stage but when the original camera motion is reintroduced it can seem flawless.
Problems arise with motion blur. A stablised image still has the blur contained inside that frame from any original camera movement. If this is offset in time - the foreground may have a different camera motion blur than the offset clip below at any point in time. Reveling the 'offset background' will then revel the wrong amount of blur, which can be a problem.
4. Roto Clone Tools/Source Nodes
To solve the problems in approach one (manual painting), there are faster computer-assisted approaches to cloning and painting. A common approach is to create an accurate matte of the wire or rig via rotoscoping. This matte is used to cut a hole in the iamage. To fill the hole, the same frame is slid or shifted to reaveal a part of the frame that is wire-free. In flame this is done by using source nodes, which move the matte or foreground independently of each other. There are many similar approaches available in other applications.
The reason this approach works so well is that it matches motion blur or light changes since each frame draws from the same frame for the fix. It can be thought of as a dynamically updating patch approach. The disadvantage is the issue with sharp edge transitions. Since the entire wire matte region is replaced with the same scale patch, what works on camera right may not align camera left or vice versa. This means that some shots work amazingly well, but a very similar shot with different structural solid lines at a differents angle may produce very poor results.
5. Automated Tools: Matador to Furnace
Matador
This early paint product had specialized roto, paint and wire removal. Matador was originally developed by British developer Parralax, and was acquired by Avid along with Parralax’s compositing application Illusion. Available only on the SGI platform and priced around $15,000, Matador was one of the first digital rotoscoping tools which gained a wide acceptance in the film post production pipeline. Matador started as a tool made for editing still images, so many of the tools used for motion work were not well thought out. Matador provides excellent matte creation tools including b-splines, motion tracking, and a full set of painting and cloning tools, with full 16bit/channel support. Avid stopped development of Matador in the late 90’s.
The original developers tried to spin it off into a new company called “Blueâ€, but that never took off. Matador had strong wire tools that allowed for median painting that would remove a wire from a frame via a brush that performed median filtering.
Commotion
Developed by Scott Squires,  at the time an Industrial Light and Magic visual effects supervisor, Commotion was used for years at ILM before Squires formed Puffin Designs and released it to the public. Commotion, then called Flipbook, was sighted often at ILM and mistakenly referred to as the “secret ILM motion version of Photoshopâ€. Though Commotion looked very similar to Photoshop in some respects, Commotion’s interface and tools were designed for moving images, and was the first tool on the desktop to offer realtime ram based playback. This realtime core functionality was the foundation for all of the wire removal tools added as the product developed.
Advanced wire removal tools include raster based paint, spatial and temporal cloning, clone to center wire removal tools (which painted from the outside of the brush stoke to the center of the same brush stroke), auto-paint, unlimited bezier and natural cubic b-splines, motion blur on rotosplines, and a very fast and accurate motion tracker. Commotion quickly became the de-facto roto tool in the industry, replacing Matador in most post facilities. Commotion curves could even be exported and imported into AfterEffects. Puffin Designs was acquired by Pinnacle Systems in 2000, but sadly development has stopped on the product. All the original developers left and no new work ever been done on the product.
Most paint work done in the wire removal process is used for touching up film or video footage. This includes removing wires and rigs, removing logos, dust busting, scratch removal, etc. Clearly, roto plays a key role in solutions beyond hand painting. Roto-based approaches have the huge advantage in that they are repeatable, or able to be rendered, so a shot can be reworked without requiring starting from scratch. Roto is therefore often  central to any wire removal process. In these circumstances, the roto tool must provide a procedural temporal and/or spatial cloning. Spatial cloning is a type of cloning which takes pixels from one position of the frame, and paints the source onto another position on the frame. Photoshop’s rubber stamp tool is an example of spatial cloning, but it is not easily automated. Temporal cloning allows one to paint pixels from one frame in a sequence to another frame. Commotion’s Super Clone tool is an example of temporal cloning.
A good wire/roto tool should provide both of these options so users can offset position and frame number together. Other cloning tools include wire removal tools which allow you to draw a line to zip out a wire. Typically, wire removal tools clone pixels from a specified value on either side of the line, then smear the outside pixels together to cover up the wire or scratch. More advanced wire removal tools will add advanced cloning techniques to the wire removal process. For example, Commotion looks at a specified number of pixels on either side of the line, flips those pixel values then cross dissolves to cover up the wire.
Furnace
One of the few companies to aggressively try and produce specific wire removal tools is The Foundry in London. Released initially as plugins for Flame, The Foundry now produces a large variety of solutions for a range of applications such as Shake and Nuke.
The Foundry's Furnace tools are based on producing a rotoscoped or animated spline path that can then have a variety of techniques deployed. The Foundry's spline defines the shape of the wire (straight or curved) and the width. It will then use one of four techniques to clone, average, clone to center or temporally remove the wire. In so doing it attempts to not remove film grain and instead just replace the wire. The techniques increase in computational complexity, and are generally agreed to be currently some of the best implementations anywhere, "best in class".
It is openly agreed - even by The Foundry, - that the tools are not one button press fixes that always work perfectly. Rather, the logic is that the wire removal tools can successfully remove a large amount of unwanted wire, but the last 20 percent will need manual or human intervention. As you can hear in this week's fxpodcast, nearly all professional wire removers use Furnace for this reason, and universally agree that getting the job 80% done is invaluable. However, there is a significant gap between any computer solution and an acceptable feature film final shot.
Mokey
As the task of wire removal extends to rig removal, The Foundry and companies such as Imagineer Systems also produce planar trackers. These trackers are widely used to solve wire and rig removal but are not solely aimed at these tasks. Imagineering produces a set of very impressive planar tracker solutions such as Mocha which will track a region of a shot. These regional tracking approaches allow much better "patch' style solutions since the 'patch' can now move in three dimensions and much more seamlessly blend with the original plate.
Imagineering also produces automated roto tools such as Mokey which can aid in separating an object from the background. The combination of this and rotoscoping allow the 2D and 3D environment techniques discussed next to work well. Products such as Mocha and Mokey are not wire removal tools in and of themselves but impressive and valuable tools that aid more advanced solutions which would otherwise be impractical.
7. 2D Background Replacement
Some of the most advanced solutions do not try and minimize the region of the frame which is being painted or retouched. Rather, they remove the entire background and replace it with a clean background tracked in 2D to look real. An example would be wire work done for a martial arts film. In these films extensive wire work is often required. Chinese senior VFX artist August Zhuang gave the example of a fight on wires staged in a small clearing, in front of bamboo. The complex nature of the vertical bamboo made many techniques fails and the amount of complex wires and rigs needed to be removed meant the shot was always going to be very complex. Rather than tackle the wires individually, Zhuang and his team roto-ed the actors off the background and then replaced it.
Using multiple frames from the sequence which revealed clean sections of the background, the team was able to piece together a single long panorama of the bamboo clearing. They would then track this matte painting back in to every shot, replacing the original background with a nearly identical flat 2D cyclorama without any wires. They were careful to create accurate 2D tracks and apply the correct amount of motion blur on each shot. This approach proved more effective and less time consuming than wire removing 20 individual wires one at a time.
8. 3D Environment Replacement
The logical extension of 2D set replacement is full 3D set replacement. Some productions choose to start this way and film the original plate on greenscreen, while others seek more realistic lighting by filming without greenscreen and manually rotoscoping all the key elements such as actors, and then dropping in 3D behind them. In cases where primary plate photography is not greenscreen, 2D background replacement techniques such as the ones listed above can be used and the matte painting is projected or mapped over 3D geometry to allow more complex 3d spatial camera moves. 3D environment replacement has grown in popularity in large part to the advances in 3D tracking and 3D automated camera software from companies such as Pixel Farm, 2d3, Syntheyes, Realviz and 3d Equaliser.
For Die Hard 4, the elevator shaft behind Bruce Willis was extended and a virtual 3D set added to build out from the real set. The set extension had to be three dimensional to work with the complex camera moves that the team at the Orphanage was presented with. The shot works due to the quality of the wire removal on the foreground, the accuracy of the roto, and the skill of the Orphanage's 3D team in matching the lighting and textures so accurately.
In this week's podcast we speak to Aaron Rhodes, the Roto/Paint Supervisor at the Orphanage. We discuss the approaches below in terms of the real world of feature films and the Orphanage's work on Die Hard 4 in particular.
luladunsmoor (78)
hankeston (56)
hytham sameer (33)
