Trevor Hogg chats with visual effects supervisors Janek Sirrs, Jeff White, Bryan Grill, Simon Maddison, Dan Rosen, and Alessandro Cioffi as well as creative director Steve Viola about their work on The Avengers…
“Marvel tends to overlap their film projects – their current M.O. seems to be to have three pictures on the go at any one time, staggered between pre-production, principal photography, and post-production,” explains Visual Effects Supervisor Janek Sirrs (The Matrix) as to how he got involved in making the comic book movie epic The Avengers (2012). “It timed out such that I had conveniently finished up Iron Man 2 [2010] a few months earlier, with enough downtime to recover my sanity in-between. My work on IM2 meant that I was at least the resident expert on 1/6th of the Avengers characters!” Writer-Director Joss Whedon (Serenity) had a major concern which needed to be addressed. “It was key to Joss that the VFX support the very character-driven nature of the story, rather than simply existing for their own sake. For him, it was the interplay of the characters that provided the backbone of the project, and the VFX, quite rightly, couldn’t distract from that…despite the over-the-top, fantastic nature of some of the imagery; before we started storyboarding, or previs’ing any given sequence, we were always sure to discuss and clarify the point/goals of the sequence, and how it would impact the characters.” The script served as a creative influence. “Once we actually started boarding/previs’ing, there was often a good deal of back and forth between the visuals and the written page… one inspiring the other, and so on… which allowed sequences to develop in a more organic manner.”
“There are probably around 2200 ‘real’ VFX shots in the final picture,” states Janek Sirrs. “But the stereo conversion also meant that we had to pre-render all the traditional optical [digital these days] effects – dissolves, re-times – as well, so they also ended going through the VFX pipeline. Ultimately, it felt like the majority of the movie had some sort of VFX component to it.” The cinematic predecessors featuring Avenger team members Iron Man, Thor, Captain America, and Black Widow influenced the look of the epic comic book movie. “It was more important that the characters had some sort of continuity with those seen in the previous movies than the VFX work per se. Marvel has very definite plans for their superheroes, both in their own individual movies, and in the ensemble pieces such as The Avengers that we need to make sure that we’re supporting. That said it was invaluable being able to use the earlier movies as glorified ‘test material’ for The Avengers, and being able to discuss pros and cons of various techniques with their respective supervisors; this meant we didn’t waste precious time and resources reinventing the wheel, or choosing approaches that ultimately wouldn’t prove fruitful. For myself, my time on Iron Man 2 was a great help in understanding stylistically what could be achieved successfully with the Iron Man suit.”
“The important prep work was more about designing the key elements – Hulk, the alien horde, the new Mark VII Iron Man suit, the Helicarrier, [and] the Quinjet, than visual research,” reveals Janek Sirrs. “Many of these were things that would potentially carry forward to future Marvel projects so we wanted to put the time and energy into getting them right first time around. Production Designer James Chinlund [The Fountain] elected to design the hero vehicles from the ground up digitally, which was a great time saver as it made the transition from concepts to assets usable in shots much smoother. Marvel has its own internal design team who are responsible for creating the key superhero looks. They knew that Hulk was always going to be major challenge, given the general dissatisfaction with his appearance in the previous two movies, and the design process began as soon as Mark Ruffalo [You Can Count on Me] was confirmed for the role of Dr. Bruce Banner.”
“Design-wise, Hulk was all about finding the right balance between Mark Ruffalo and a combination of the Alex Ross comic book illustrations, and the rage monster from Ultimates that everybody at Marvel loved,” remarks Janek Sirrs. “He had to be human enough that we could see his character actually develop throughout the picture, rather than just being pure beast. We had to be prepared to create a 100% digital Banner/Mark in extreme close up, not just Hulk, as we had no idea what the final shots would actually entail at that point we started building. If you were to compare Mark and Hulk side by side, you’d find that they actually have the exact same wrinkles, moles, scars, teeth, and so on – the modelers and texture painters put a massive amount of effort into translating as many details as possible. As much of Hulk’s performance as possible was based upon Mark. For shots where is Hulk is standing around…. thinking… acting basically, as opposed to leaping and smashing things, we had Mark play the role on set in a motion capture suit, often on an elevated platform to put him at the correct eye line height. This may have created more paint out and clean up work, but it helped sell the feeling of Hulk being in the scene with the other characters. Once the action got crazier, Hulk’s body performance became pure animation, but we still endeavored to incorporate Mark’s facial expressions and animation from the motion capture sessions.”
“The Iron Man suit seen at the start of Avengers is simply the Mark VI seen at the end of Iron Man 2,” notes Janek Sirrs. “There’s an established handover of the latest and greatest suit from one Marvel movie to the next. But toward the end of Avengers, the Mark VI is replaced by the new Mark VII version. Marvel’s in-house design team was responsible for the look of the Mark VII, but we incorporated some specific features/capabilities at Joss’ request. For example, Joss wanted Iron Man to have the ability to hover, and engage in aerial combat, without his hands being part of the flight system…. basically, the ability to shoot and fly and the same time. The trick was to preserve the iconic look/profile of the established Iron Man suit while making the new suit feel like a logical update from the previous version. For example, the new suit also incorporates some aspects of the portable ‘suitcase suit’ seen in the Monaco sequence in Iron Man 2.”
“Probably the biggest logistical headache was the acquisition of material and subsequent creation of the fully digital portions of NY seen in the various flying shots during the finale battle,” recalls Janek Sirrs. “Having chosen to stage the battle amid prime real estate centered on Grand Central Station, we then spent months assessing the areas to capture that would give us the most bang for buck. Even on a show the size of The Avengers, there’s still a finite amount of time and money, so we had to be very selective about the areas to cover. After determining the optimal coverage, a team of dedicated stills photographers spent close to six weeks, fighting bad weather and the elements in general, to capture high resolution HDR panoramas from a variety of street corners, windows, balconies, rooftops, and 120’ condor cranes. Just getting the necessary location permits alone proved to be a major ordeal. We captured all the NY stills material very early in the schedule so that we could start building the environments in time for shot production during post. Creating those environments was something of an endless task. Once the stills were mapped onto building geometry to create the basic facades and streets, a massive amount of additional work was still needed to add life to the environments – proper window reflections, room interiors seen through windows, vehicles, pedestrians, street ‘furniture’, and so on all had to be added.”
“There’s a great deal of practical pyro work in the ground-based NY street battle scenes that was handled by Dan Sudick [Cowboys & Aliens] and his SPFX team,” notes Janek Sirrs. “Dan likes to make sure that he can over-deliver if needed so most of the time the pyro played in camera without augmentation. I seem to recall him blowing up an entire Cleveland street, complete with flipping cars, in and around fleeing stunt players at one point! When Robert Downey Jr. [Wonder Boys] is seen wearing the Iron Man suit [when the face plate is open, or the helmet is removed], he’s often wearing a practical ½ suit from the waist up so that closer coverage could be achieved in camera without resorting to digital work. In wider framings, we’d often end up augmenting the practical suit with digital legs before heading into 100% digital territory once the helmet was closed. The practical ½ suit was built by Legacy FX, and was designed to be extremely modular with quick release fittings. If a shot didn’t see RDJ’s arms, for example, we could simply unclip them and remove them to make life more comfortable. Elsewhere in the picture, there are many moments where there are handoffs to and from digital doubles from practical stunt players.”
“The alien creatures were designs specific to Avengers, rather than basing them on races drawn from the comic books, such as the Skulls,” explains Janek Sirrs. “Joss always wanted a sense of a proud, regal army so we incorporated some level of armor and ornamental adornment in all of the alien creatures and their vehicles/weaponry. We originally intended to have additional classes of aliens and vehicles but had to scale back to just the ‘foot soldiers’ and the’ jumbos’ as we dubbed them, to keep at least a little bit of a lid on things.” As for the alien artefact that enables the galactic invasion, Sirrs states, “The Tesseract had already been clearly established in Captain America [2011], so we were really just mimicking what went before. We did, however, see the Cosmic Cube in greater detail this time around, so the quality/nature of its internal ‘life’ had to be boosted to compensate. This internal ‘life’ was generated with a series of animating digital fluid simulations, and image treatments to create the sense that the cube was hot.”
“We inherited Thor’s character and abilities from the movie Thor [2011], so the foundation was already nicely prepared for Avengers,” says Janek Sirrs. “The hardest part about Thor was probably his flying….. although ‘flying’ is a bit of strong term as he’s just really dragged through the air by Mjolnir – think of throwing a ball through the air but then being dragged immediately behind it, along its parabolic arc. I don’t feel there’s any way to convincingly simulate free flying shots with stunt wire work unless you’re talking about extreme close up shots [where it’s hard to tell what’s going on, to be honest]. However, you still need some degree of wire work to help out with the take off and landing moments. In Avengers, we opted to use a Thor digital double for all the flying action to keeping things looking more natural. However, this did present some challenging transition work, to and from Chris Hemsworth [Snow White and the Huntsman], for those take offs and landings.”
In regards to high tech imagery featured in the New York City penthouse of Tony Stark, Janek Sirrs says, “If anything, we opted to simplify the holographic whole room idea, seen in Tony’s workshop in Iron Man 2, to something more akin to discreet holographic workstations. The idea being that Tony would take conventional 2D monitors to the next technological level. There actually are now real-world prototype holographic devices that are capable of displaying true three-dimensional images in thin air. They work by using multiple laser beams to ionize the air where they intersect, creating a visible hot spot. By rapidly moving the point of intersection, imagery can then be drawn out in 3D space. Currently, the imagery is still very crude but you can definitely see that’s headed in a Stark Industries direction.”
“ILM was the logical choice to handle the Iron Man related aspects of the show given their involvement with the previous Iron Man pictures,” explains Janek Sirrs as to why he chose the Californian VFX facility to be a key contributor to The Avengers. “Additionally, I knew that they already had a well-established pipeline for creating digital city environments from re-projected stills material, something that would be crucial in creating a great deal of the final battle sequence. However, it quickly became apparent that the show had more material than could sensibly be handled by a single facility, and that it would be have to be divided up between multiple vendors just to get it all completed in time. In pre-production we did a round of testing among several facilities, and had them produce their own Iron Man shots, using the digital assets from the last movie, so that we could assess their capabilities. Weta ended up being our second heavy hitter, and they handled all of the Iron Man work in the middle act action sequence set on the Helicarrier. Other, smaller vendors were often companies that Marvel had worked with in the past, and had forged a good relationship with.” Accompanying ILM and Weta Digital on the $220 million Hollywood production were Scanline VFX, Fuel VFX, Evil Eye Pictures, Trixter and Method Design.
“Joss [Whedon] was an integral part of guiding the visual effects work,” states ILM VFX Supervisor Jeff White. “We would do regular cinesync’s with Joss and the team at Marvel to review the shot work; he was great at identifying the big picture items a shot needed and letting us work through the details. Joss’ skill in working with actors extended very naturally to working with the animators to get the performances he was looking for. We had to keep Google search close at hand, however, as Joss is a walking film library and was throwing out film and artist references all the time that we’d have to look up!” The previous contributions on Iron Man and Iron Man 2 helped when constructing the Iron Man Mark VII outfit. “We were lucky enough to have Bruce Holcolmb as our ILM model supervisor; he built the suits for the first two films and is a walking encyclopedia on all of the detail and subtleties that need to be part of any Iron Man suit. Bruce came up with some great new gadgets and weapons for the Mark VII.”
“The biggest challenge was making sure the Mark VII held up to the high standards that have been established for Iron Man in the previous two films, while incorporating some new features,” states Jeff White. “We spent a significant amount of time tuning the colour and materials to get the right look for his metals. We also created several levels of damage for both the Mark VI and the Mark VII. We wanted the Mark VI damage to look like scraped away paint and gouges from his run in with the Helicarrier engine, while the Mark VII damage was battle oriented. Because it was important to have a progression to the damage, we broke the damage into five layers that we could dial on as Iron Man encountered specific battles.” Creating the cinematic moment where Tony Stark is falling out of his penthouse and Mark VII catches up with him was a complicated sequence to produce. “It ended up being quite difficult. We had to figure out the pacing of the suit attaching to Tony in each shot so that he finishes in the nick of time, but that it’s still believable that it could all happen before he hits the ground. Once we had a layout for the sequence, we started into the suit deployment animation. It was all hand key-framed to get the level of complexity we needed. The backgrounds are constructed from photography of New York, while Stark Tower was a fully CG asset. We added CG legs with cloth sim, cheek wiffle on Tony and lots of moving atmosphere to help sell the speed at which he’s falling.”
“Starting with a captured performance from Mark Ruffalo was key,” answers Jeff White when asked on how ILM ensured that the acting of Ruffalo was not overshadowed by the CGI. “Joss could direct and make selects on the performance he was looking for. Mark brought so much to the table both in terms of overall performance as well as subtle movements and gestures that we were able to incorporate into the Hulk. There was a lot of really incredible key-frame animation to augment that performance to get us the rest of the way there. Having the Hulk look like Banner let us use Mark’s eyes as a guide for adjusting our CG Hulk eyes, helping us overcome one of the more difficult problems in CG.” Several different techniques were utilized to blend the CG Hulk into the various environments and to make him move realistically. “Movement was a combination of motion capture and key-frame animation. Hulk had to do many jumping and smashing actions where mocap couldn’t be used and it was great key-frame animation that got us there, supervised by Marc Chu. Our Hulk character TD, John Doublestein, also created a system with three layers of muscle and skin simulation to get the right secondary jiggle and slide to the skin. The final pass was detailed hand modeled shapes to preserve the correct relationships in the anatomy. Because we used virtual environments for many of our New York City shots, we were able to utilize those as lighting environments for the Hulk. We would also comp additional fires into the scene and use those as accent lights on Hulk to get good definition on his forms. Joss had also picked a very de-saturated green colour for the Hulk which worked nicely as a skin tone and kept him from popping out too much when standing side by side with the other Avengers.”
The final battle sequence, which leads to a devastated New York City, required extensive visual research. “It was by far our biggest photo shoot in New York and took a few months of planning by our Digital Matte Supervisor Richard Bluff and Janek Sirrs,” remarks Jeff White. “We had four guys shooting for over eight weeks to capture all of the material. That meant getting permits for lane closures, access to building rooftops it was a complicated undertaking, but we had a great locations and productions team to work with at Marvel.” As for integrating practical effects such as pyro and debris, White remarks, “A key to getting the environments to look good was adding back in the movement that’s lost in the photography. With the end of the movie being in a raging battle, we played up drifting smokes, ground fires and particulates in the air to add texture. For most of the alien destruction of the city we used CG fire, smoke, dust and glass but we’d always mix in real practical elements to help where the CG was lacking detail.”
When questioned about the continuous shot which begins at the street level fighting and concludes with Hawkeye shooting an arrow from a rooftop, White replies, “The idea was to have a shot showing all of the Avengers working as a team and the challenge was to link them all together without it feeling disjointed. Detailed pre-viz was done by the Third Floor and aided the plate shoots that were done for Black Widow, Hawkeye and Thor. We then spent months working out the camera layout and animation so that it maintained the right pace and energy throughout the shot. The backgrounds were 100% CG, using our photography from New York as a base. We had to splice together several streets to create a run that was long enough to sustain the final shot. The backgrounds were then used as lighting environments for the CG characters. We played up the characters moving through slashes of sunlight and caustics to help sell forward travel. The backgrounds were augmented with 2D smoke, Plume effects simulations for explosions, as well as ash, embers and other debris in the air to dirty up the scene.”
“There were two different types of shots we had to deal with for The Avengers,” states Visual Effects Supervisor Bryan Grill who works for Scanline VFX. “First were the Helicarrier shots which depict a large aircraft carrier lift out of the ocean and start to fly. The next was the Helicarrier cloaking so people from the ground were unable to see this behemoth flying around. We started with some concept images based on real world hover craft and aircraft such as helicopters and harrier jets over the water. We studied the scale of white water wakes around real aircraft carriers for wave height and white water detail around such a big ship. With all that in mind we extrapolated those visual references and started to do a wedge of water simulations until we had the water surface reacting around the aircraft carrier and didn’t question its size or scale. It took a few more times to refine the amount of detail we needed which gave us a base look for all the shots. The next step was the dripping water coming off the carrier; it needed to feel like if was falling off a very large and tall ship. The dripping water started off thicker from where it was cascading off and eventually had a misty quality by the time it hit the water surface below.”
Visual Effects Supervisor Stephan Trojansky collaborated with his Scanline VFX colleague on the project. “Since he is the principal architect of our proprietary fluid simulation software Flowline, Stephan’s knowledge of the software is indispensable,” states Bryan Grill, “but his ability to reproduce water and its behavior does not stop at the simulations. He has an amazing eye for detail and pushes his vision throughout the VFX process; as my background is more 2D and compositing, it made perfect sense for Stephan to spearhead the water shots and for me to cover the rest of the complicated set reconstruction and extension work. Both of us supervised the shots and provided creative and aesthetic directions throughout the whole process.” The signature software program invented by Scanline VFX was an indispensable creative tool. “Flowline was a huge asset in creating the Helicarrier scenes. The pipeline at Scanline is set up to maximize the iterative process of simulations and rendering. The multiple and fast iterations, together with sizable disc storage and processing power, really helped us from a technical standpoint. But as always it is the artistry of the artists that pushes the aesthetics. We have some amazing people here at Scanline that compliment the software’s ability in creating the final look for the filmmakers.”
“The biggest challenge in creating the introduction was always the scale of the Helicarrier,” reveals Bryan Grill. “Making sure it was always looking big and impressive. Starting with a model of the Helicarrier we got from ILM we were tasked to continue to build on the model to make it work for our group of shots. We kept increasing the amount of detail that we saw on the ship. The lighting and rendering of the Helicarrier was done in 3ds Max and rendered in Vray. It was very important that our CG Helicarrier was photo realistic even before we started with all the water simulations. Adding motion captured people and animating moving vehicles helped immensely. The cloaking effect was an especially daunting task. We all have seen the cloaking of spaceships from science fiction movies and television shows but the science we had to figure out needed to be realistic for our times. We scoured the Internet to learn of the latest LED, plasma, and LCD technologies. What we came up with was what if there was some sort of material on the bottom of the carrier that lit up like a big JumboTron you would see at any concert or sporting event; not just one JumboTron but hundreds of JumboTrons all being fed a video signal of the environment above the Helicarrier. Once the basic concept was in place it became the process of iterating different looks and timings until we came up with the look that film makers were happy with. This led to agreeing to do the other two cloaking shots in the movie and passing along our Nuke scripts to Weta so they could have the ability to tweak the parameters in their composites.”
“We’ve developed a great relationship with Marvel [Iron Man 2, Captain America and Thor] which included working with Janek Sirrs on Iron Man 2,” states VFX Supervisor Simon Maddison who works for Fuel VFX. “It’s not uncommon on Marvel films to be asked to come up with looks for things that aren’t seen everyday and Janek is great at helping direct you down what avenues are the best ones to explore.” The Australian based company was responsible for developing the holograms activated by Tony Stark while in his penthouse. “Obviously there is a design precedent that has been set in the Iron Man films. We spent a lot of time discussing what was relevant in those holograms and where we could push things this time. Certainly the look of the light hanging in air is something that required a lot of very subtle CGI; if we pushed how much it ‘flickered’ for example, it looked fake; push it too far the other way and it was hard to read, or kind of ugly to look at. The other trick was to find the balance between making something look cool and conveying all the parts of the story that Joss and Janek needed to tell. When Tony throws out the dossiers on the other characters and the Tesseract for example, there are over 70 different bits of information in those holograms, all on the screen at the one time. Deciding which bits to give the right amount of real estate to was tricky. It had to all be relevant information and feel like there was a lot of it, but at the same time the audience needed to take away only a fraction of that information. Which parts we wanted them to focus on and how to make them do that in such a dense frame was a bit of a tap dance.”
Designing any kind of hologram that needs to sit in on top of live-action was always going to be tricky,” says Simon Maddison. “First of all there’s the fact that no one really knew what they were going to look like when they shot it – how the holographic components would be composed and what would need to sit where in the frame to tell the right story. So we found we were fighting a little bit with letting the audience know what the hologram was about and at the same time not cover the actor’s face with anything that was going to distract from their performance. Add to that the fact that what worked for an element in one shot, might put that same thing right over the actor’s face in another and we had some pretty tricky compositional problems. The only way to solve it was to have all the 3D cameras set up for each shot, design something basic for one of them, then populate all the shots with the same stuff to see where you were at. We did it in Nuke, so luckily we could move things around the 3D space pretty much in real time. So after a few back and forths between shots and with a little cheating in each one, we ended up with something that cut well and conveyed the story that the same hologram was being viewed from different angles. But most importantly everyone got to look at what they really went to the cinemas for – to see the characters.”
Incorporating the sweeping views of New York City into the glass windows of the penthouse was required for night and day scenes. “They both presented interesting challenges but I’d have to say the daytime was a little trickier,” notes Simon Maddison. “First of all, you naturally saw a lot more in the daytime. Also, it’s important to give a matte painting some kind of movement, an indication of life, otherwise it looks like what it is – a still image. With the night time stuff is was easy because we could add bright defocused city lights that danced around because they were being affected by the atmosphere. We also added some distant planes in the sky and moving traffic with headlights. In the daytime cyc, it was way harder to read any of that movement so we had to add much more of it and apply it in different ways. Moving clouds, steam from rooftop vents, moving reflections on the water, some very slight rolling highlights on the Chrysler Tower when the camera moved – lots of subtle stuff that hopefully added up to a seamless and believable shots.” Maddison adds, “There was really only minimal set extension for the inside of the penthouse. We needed to continue the glass windows all the way to the ceiling and add a little bit to the balcony outside. The hardest part was probably retaining the right amount of reflection from the photography in the blue screen windows.”
The experience of altering the glass reflections in scene where Tom Cruise (The Last Samurai) climbs the Burj Khalifa in Mission: Impossible – Ghost Protocol (2011) came in handy during the Loki-Thor Fight Sequence. “Having been through dealing with creating a reflective building in CG was advantageous, and we faced the same issue on both films where the partial set-piece that was built for in-camera use didn’t exactly fit with the CG designed version,” states Simon Maddison. “We certainly needed to replace more of the live-action set than we originally intended. So there was lots of massaging of the CG Stark Tower, and because of our shots are peppered within other shots that were being done by ILM, there was a lot of back and forward with their supervisors and digital matte painting departments to make sure that we were all sitting in the same place. Then we also had to make sure both Fuel’s and ILM’s comp work could cut together seamlessly which was much harder than it sounds as ILM had fully CG shots while we were working with live-action filmed on a blue screen stage. For example, the exact colour of the sun flares needed to match across the shots; Thor’s hair which was isolated in the grade, needed to maintain continuity. The biggest one was the exact colour and brightness of the sky – this took a lot of work to get right. Janek had a great attention to detail about all this and we looked at a lot of contact sheets together that contained the sequence so we could all get it right.”
“Getting practical sets to line up with CG versions of the same thing for set extensions can be tricky sometimes, especially when you have to make small tweaks to the design along the way,” observes Simon Maddison who had to add the cityscape of New York City in a seamless fashion. “Lots of movement in the digital matte painting and lots of subtle colour grading went into it. Also, getting the right level of brightness was important. Obviously you want to be able to see the city outside the window but you also need make it bright enough so that it looks like it’s all been exposed in the same image as the foreground. If you really tried to shoot that with a camera you would probably find that the background would blow right out in the daytime. We came to a balance with Janek by pulling it down slightly from where it should probably sit in reality and using the idea that the glass in the windows was slightly polarized. If you look carefully at the balcony outside the apartment, you can see that the more levels of glass you see through, the darker the background appears to be. Other less obvious things helped as well. How much the light wrapped around the edges of the foreground plate, especially the actors hair, help sell it as real.”
“The trickiest part of the CG chariots was continuity with the work ILM were doing on the larger fight scenes,” says Simon Maddison. “As for the scepter, it wasn’t so much the FX animation from its blast but the scepter itself. Because it was a fight scene, Loki had a rubber stunt scepter that had an incomplete rubber blade and wobbled around a little bit. So we had to replace this with a CG scepter. In many cases we were able to replace the top section of it but it certainly got tricky when we needed to get it behind Thor’s hair – especially in his close-ups.” A number of shots were shared by the various VFX facilities to create the Loki-Thor Fight Sequence. “Janek has a great overview of what’s required and we received a lot of contact sheets showing how each vendor’s work was comparing to others. We also exchanged DPX files of work in progress shots so we could see it in the right colour space and resolution as our own work. Any time it got particularly tricky we just jumped on the phone with Janek and the other vendors and discussed the best way to move forward.”
“Our sequences covered many different times of day,” remarks Dan Rosen, “ranging from night, to early morning and a twilight shot that all had cloudy skies. Janek directed us to balance the early morning and twilight shots to have enough blue shadows and slight sunny yellows and pinks without being over the top. The Avengers achieved a nice balance of dramatic skies that felt cinematic, but weren’t distracting from the action or the plot. We used helicopter plates shot by Janek for reference and actual source imagery that went into shots. VFX Co-Supervisor and Evil Eye Partner, Matt McDonald, headed up the work on all of the sky environments, which freed me up a great deal to concentrate on all the other issues. For the energy FX in Loki’s scepter and a small one off shot we looked to previous Marvel movies and a lot of other FX heavy films. We also tapped Internet video of plasma energy based on direction from Janek for the one-off shot where Banner is tweaking a piece of equipment in the Wishbone Lab.”
“With multiple vendors working on similar Helicarrier and daytime sky environments we had to keep in step with changes being made across shots at other VFX facilities,” states Dan Rosen. “We wind up working on a lot of big features with multiple post-houses, so we are used to sharing images and assets. Evil Eye led the look on the night-time atrium, so that work provided different challenges from the daytime shots. We ran a lot of tests to get the atrium lighting to feel like it was the correct colour temperature and raking across the structure in the way that it was in the foreground live-action. At night we were also seeing into the atrium hallways given lower reflection values on the glass. We had to model and texture more detail into the hallways. We also created a digital doubles system that underwent more scrutiny in those lighting conditions. We augmented closer proximity shots with live-action doubles shot on blue-screen by Janek.”
“As compared to Captain America, our work on The Avengers covered a wide range of effects, from classic green screen compositing to more vast set extension, to CG atmospherics, fire simulations and effects in general, to the re-design and implementation of some digital props,” states Trixter Visual Effects Supervisor Alessandro Cioffi. “Since the first meeting — which took place virtually, via Skype and Cinesync — VFX Supervisor Janek Sirrs always conveyed a great confidence and encouraged us to propose ideas that would represent visual and creative solutions, emphasizing the storytelling aspect of any given sequence. In subsequent meetings, which occurred with frequency and regularity during the whole process, we had the opportunity to receive constant and direct feedback to refine our work. It is worth mentioning that one of the first tasks assigned to us was to re-design and create Loki’s ‘eye-extractor’ tool and the corresponding hologram coming from Hawkeye’s device. On that particular occasion, Janek left us much room for interpretation in terms of the look of the hologram and the device itself, and we took the opportunity to present some solutions which were well received and subsequently approved.”
“As previously mentioned, we had to create a holographic image, and conceive of a way for it to ‘build up,’ so to speak,” replies Cioffi when asked about the visual research conducted for the project. “We analyzed various solutions, knowing that we wished to present something unique, with a look that had been not necessarily been done dozens of times in the past. It was also important to us that we ‘tell the story’ of the object, in this case the eye of the poor victim, Schäfer, that despite being a virtual, intangible hologram, could possess the characteristics of ‘materiality’ which might fool a detection device reading the information in the retina. We are happy with the result, and most importantly, the effect yielded approval from our customers. Another type of research was required when we were given the task of recreating the aliens’ tracer fires. In that case, we first studied the work done by ILM, which we used as a visual guideline, before recreating the essence of their work, but in a different form. Another example is when we were tasked with creating the look of the weapon used by Coulson against Loki, before Coulson dies. We strived to create a look that would represent the energy derived and inspired by the Destroyer, which previously appeared in Thor.”
“The biggest challenge for us has been in playing together with the big companies, such as ILM and Weta, and staying consistent with the quality and complexity of their images,” remarks Alessandro Cioffi. “For the aforementioned tracer fires in the New York City Attack Sequence, some of our shots intercut with ILM shots. In addition to finishing the shots, the side challenge was to make sure our work didn’t somehow stand out or not integrate with the incredible scenes that ILM has created. The main sequences we worked on included the New York City attack, Loki’s intrusion at Schäfer’s Labs in Stuttgart, and a few sequences around the Hulk Helicarrier Chamber, as well as several more sparse shots in various sequences.” Trixter had other responsibilities. “We worked on a number of set extensions and digital props, including Thor’s hammer and Hawkeye’s quiver, and more generally, some picture enhancement. Some of the matte paintings used to extend the sets was then projected onto some 3D geometry in order to be used on more than one occasion or shot. For instance, in the sequence where Coulson tries to stop Loki with the alien gun, the subsequent wall destruction and scorch marks are actually a matte painting, which was used in several subsequent shots.”
“The big goal was to make it look like that there are real things at stake,” states Method Design Creative Director Steve Viola who worked on the Main and End Title Sequences. “The other big thing was group chemistry and Joss wanted to do some exploration focusing on how the characters were thrust together.” The creative experience was comparable to one encountered on another Marvel production, Captain America which was helmed by Joe Johnston (The Rocketeer). “Both directors had a very clear vision of what it was they were looking for and wanted to have a good collaboration from start to finish. We were still going back and forth and getting everything Joss wanted in there up until the very end.” A significant change was how the Method Design team was incorporated into the production of the blockbuster movie. “Most main title projects I’ve worked on in the past you’re treated more as a design entity; the revisions, the reviews, and the way you are delivering it’s much more of an organic process. On this one because of how visual effects heavy it was, and the entire film was a digital delivery system that Marvel had setup, they looped us in with visual effects.” Viola explains, “When they would have their visual effects reviews that’s when we would go over our materials, our naming conventions, the file type, and colour; everything needed to go through the same workflow and pipeline they had setup for all of the visual effects vendors as well.”
“A lot of it was based around the original footage that they showed us a vignette of all the different moments where you meet the various characters,” states Steve Viola. “We noticed that all of the scenery, the lighting, and the mood were based around each character. When you go to meet Bruce Banner for the first time it’s very green and there’s lot of atmosphere; it’s organic in nature. Whereas when you meet Iron Man it’s on top of his building in New York City. A lot of the areas and colour tonality of the different pieces was based around each character in the same way that they had done in the film. A lot of the locations were based around where they were on the Helicarrier. For instance, Iron Man suit was stored in the cargo hold; we were trying to in a lot of ways match some of that environment so it’s all grounded in reality.”
“When it comes to the superheroes the weapons are also part of them,” notes Steve Viola. “In the case of Black Widow and Hawkeye, they’re master assassins. With Black Widow she’s got her outfit and has the stinger on her wrist which you only see her use a couple of times in the film. There’s Hawkeye who has the quiver with the arrows which he uses a lot. Then you have things like the shield, and for Nick Fury there’s his gun; they’re like the tools or instruments of those characters who don’t have things like the Iron Man suit. Joss’ big thing was that he didn’t want the characters to be there. We had to figure out how do you show the Hulk without showing the Hulk? For him, we setup in the laboratory, which is the home base for Bruce Banner in the film, and had the torn clothing and the broken glasses to allude to his transformation to the Hulk. Later on we revisit the laboratory with one of our transitions where we pull through one of the lab monitors and there’s a frozen image of him in mid-transformation with his hands in there as well. You get a little hint of him without seeing the Hulk himself in the environment.”
“You have to weigh all of the various aspects of trying to shoot something practically over doing all CGI,” says Steve Viola who chose to go the complete CGI route. “In the end having so many assets have already been built for the feature by the various other visual effects companies gave us a good starting point.” The initial concept of assembling the different imagery changed. “Originally, it was all cuts but we did one transition originally and everyone at Marvel including Joss liked the integration and the juxtaposition of how you would get from one character to another. We started doing that a lot and ended up with a number of transitions in the end.” Viola gives some examples. “We have a reflective transition coming out of the lens of the scope of a targeting system on Hawkeye’s bow. We also pulled through the broken glass of one of the monitors, and then rack out of focus on the background which was a pretty good transition. We used lightning to transition from Thor to Black Widow. In the beginning we had a whole bunch. We transitioned more through black with a connected camera move going from the shield to Black Widow. Joss was extremely happy.”
“We were using a lot of rack focuses not just as a photographic tool but also to keep all of the titles at the right length,” states Steve Viola. “It helped to keep the piece flowing nicely because not only do we have the camera moving but the depths are always changing, taking you from one place to another.” Method Design created a couple of type faces. “In the end it was very elegant and simplistic because we were trying to focus on the visuals of our characters while still making it a title sequence. We did tie it into The Avengers brand and also did some custom modifications to the main title.” The end title sequence segues into the final shot that hints of a future adversary for the team of superheroes. “We transition from our shot of the R.T. on Iron Man’s chest to the shot of the moon which was done by Digital Domain. There were a couple of meetings where we both got together with Marvel and talked about what assets we needed. They were getting notes for their shots while we were still working on ours; we kept getting updated camera rigs from them and trying to figure out how to smoothly go from our 50mm to a 20mm. It was a good process and worked well.”
“In terms of basic image quality – resolution, bit depth, etc. – all the digital production footage was put through a single processing pipeline,” states Academy Award-winning Visual Effects Supervisor Janek Sirrs when explaining how he made sure there was a uniformed look maintained by the various VFX vendors. “That way, we could ensure that the ARRI RAW files from our ALEXA cameras had a 100% consistent look between facilities. We actually spent several weeks testing de-bayering algorithms before settling on the solution that all our vendors would end up using. Additionally, all the VFX-related material was pre-colour timed with the DP and the show’s colourist so that facilities could be confident that they were dealing with plates that were actually representative of what the final movie would look like. The final DI session actually ended up being a relatively quick process as we’d done all the heavy lifting earlier. To keep digital assets [such as digital doubles and vehicles] consistent, we elected to have a single vendor build ‘master’ versions of the assets, rather than build have multiple facilities build their own versions. Admittedly, that meant overspec’ing the builds to create very flexible, neatly packaged assets, but it paid off in the long run as we could simply hand over, say, everything to do with Iron Man to a new vendor and have them up and running in no time. We tried to avoid sharing shots as much as possible, just to keep things easier to manage, and relied more on sharing the ‘master’ digital assets instead. But on a show the size of Avengers, there’s no way to get away without some degree of shot sharing, painful though it is.”
“We did 70 shots as full stereo comps and those came with their own challenges because we can’t use quite as many cheats,” states ILM Visual Effects Supervisor Jeff White. “Because so much of the world was being created in CG, however, it was pretty straight forward to maintain a nice continuous depth. For shots that are converted, we broke them into layers to help the conversion house get the best result possible. Joss was always looking for better ways to stage the composition or bring something at camera to utilize in 3D.” Visual Effects Supervisor Bryan Grill of Scanline VFX notes, “The 3D and IMAX didn’t really complicate our lives much. Most of our shots were full CG so that makes it easier to setup and render in stereo. We did have a few plate shots that required us to pay special attention to the fact they were going to be dimensionalized. We worked closely with the client to make sure we were giving the studio that was doing the stereo work everything they needed in a neat and organized manner. As you know working in stereo makes it very complicated when dealing with plate photography. Tracking, roto, paint and all those cool comp tricks we have learned over the years give way to a whole new set of challenges.”
“The production schedule really didn’t facilitate coming up with new technology,” says Janek Sirrs. “We had to hit the ground running just to get through the sheer volume of material in time! The trick was more about researching which facilities were proficient at what so that we could quickly come up with an overall efficient VFX solution for the show.” A particular cinematic scene stands out. “I think the moment where the Helicarrier takes off out of the ocean was particularly successful – It’s not a big sequence in the scheme of things, but it really has the epic sense of scale and grandeur that we were striving for, and provides the uplifting feeling called for at that point in the story. Also, it’s a series of almost, if not entirely, 100% digital shots that are always a tall order to pull off successfully.” As for the final big screen presentation of The Avengers, Sirrs states, “IMAX wasn’t an issue as the IMAX release was simply derived from the finished DCP version, after we had completed all the VFX work. Stereo was a different matter though, not from a shot design point of view but more from logistical and scheduling aspect. The show was originally scheduled as 2D release so we ended up having to squeeze in the stereo conversion work into an already tight post-production schedule. I’m sure that the conversion team was working around the clock toward final delivery – they couldn’t wait to pry the VFX finals from our hands as soon as they were ready!”
“Everyone here at Fuel VFX is incredibly proud to have worked on another Marvel film,” states Simon Maddison. “We always have a lot of fun along the way and no matter how hard it gets, when you see this stuff on the big screen it is always immensely satisfying.” Dan Rosen of Evil Eye Pictures agrees. “We enjoyed working with the entire Marvel team,” says Rosen. “They strike a great balance of bringing multiple VFX vendors together to tackle really complex films.” Summing up his experience on the epic project, Bryan Grill remarks, “Overall I thought the movie and visuals worked very well together. Always pushing for reality in a super hero world was the goal from the start. I felt all the visual effects in the movie were executed and represented in glorious fashion. It’s especially great to be able to collaborate with our friends at ILM and to be part of a Marvel Studios production, and one that has proved to be hugely successful, breaking records on its way to a $1 billion dollar worldwide haul.”
Production stills © 2011 MVLFFLLC. TM & © 2011 Marvel. All Rights Reserved.
VFX images © 2011 MVLFFLLC. TM & © 2011 Marvel. All Rights Reserved. Images courtesy Disney, Fuel VFX, Evil Eye Pictures, Trixter and Method Design.
Trevor Hogg is a freelance video editor and writer who currently resides in Canada.
