Motion capture animation in blockbuster movies is all the rage these days. From The Hobbit to the new Planet of the Apes films to The Avengers, the line between acting and effects work has been so blurred that a debate has sprung up about whether or not these mo-cap performances – which, in some cases, have been phenomenal – deserve acting awards. Matt Reeves and Jonathan Liebesman, directors of Dawn of the Planet of the Apes and Teenage Mutant Ninja Turtles, think that the answer is a resounding yes.
Speaking of TMNT, it’s the next big film to utilize a significant amount of motion capture animation. All four Turtles, Splinter, and Shredder (to a degree) were created with the help of mo-cap technology – which, according to visual effects supervisor Pablo Helman, is possibly the most advanced mo-cap work that’s ever been put to film.
Last year, we were invited to the set of Ninja Turtles and got the opportunity to talk to Pablo, who happens to be one of the most prolific FX people working today. He’s been involved with everything from Indiana Jones to Star Wars to The War of the Worlds to Chronicles of Riddick to Men in Black to Independence Day, and his next big job is the Teenage Mutant Ninja Turtles reboot.
Much of what Pablo discussed with us was incredibly technical, to the point that it was occasionally difficult to grasp what was being intimated (and we were there!). But he also talked a bit about the difficulty of capturing emotional transitions, the design of the Turtles, Splinter, and Shredder, and trying not to use green screen. He even let slip a few plot details that the producers probably would’ve preferred he keep to himself.
(Note: When Pablo joined us, we had been checking out the motion-capture helmets that were used for the different Turtles.)
We’re a group of journalists. We were admiring all the rigs.
Pablo Helman: Ah, okay. Well, we’re super excited about what we’re doing because this performance capture technology that we’re developing hasn’t been used before, maybe ever. So these two little cameras that you see there, they’re HD cameras and up to now all the performance capture that has been done has only been with one camera and with standard definition. So this is HD and two cameras. So we’re able to capture all kinds of nuances and define the performance.
How do you graft that onto the design of the turtles? What is it actually capturing that you’re able to use?
That part is called retargeting, and basically what you do is-there are two sides to it, there’s an A side and a B side. The A side is getting a digital double of the actors and that scan of the heads that you see is part of that digital double. And then you create a creature, and all those points that you see on the heads, they get retargeted to the creatures themselves. That would be one thing, and we’re developing the technology to retarget that.
Do they look human enough to really use one? We haven’t seen one.
Let’s look at them. [At this point, Pablo showed us the models of the Ninja Turtles.] They’re very heroic, the design is very dark. So you can see that some of the designs, they’re not quite human [in proportion], but they could be retargeted [for the animation]. In other words, the cheeks are where they are, the eyes are a little more separated, they are a lot taller because this is about 6′ 5″, Raphael is 6′ 7″, so they’re pretty big but you can see they’re pretty human-like.
We’re really conscious about the mythology of the franchise and what’s been done before, how we will evolve. It’s kind of a very heroic render of the Turtles and we’re thinking that’s why we went with the methodology of casting four actors and getting the performance [with motion capture technology] as best we can. We did a completely new pipeline and new ways to capture that technology. And we target that performance into these faces.
How many cameras are on set to pick up the external markers and the bodysuits?
Well, yeah, that’s the second part of it. So this project scientifically has to do with the performance capture of the face, also the performance capture of their bodies […], and there are two HD cameras capturing basically what they do with their bodies, plus two cameras that are capturing the face. So the idea is that the director directs the actors on the set and the choices that are made in editorial have to do with what he shot.
So he shoots a movie like he normally would, and that also gives the movie a very kind of grounded look, it’s not a CGI environment anymore, it’s all kinds of smoke on the set, there’s no green screen anywhere because these guys – basically the actors are going to be erased. Um, we should clean plates and some other stuff that we normally do for these effects, but I try not to use green or blue screen, it’s just what it is. And the idea, again, is that Jonathan Liebesman and Michael Bay work together and we end up with something that is grounded and that is physically possible. Plus, all the amazing things that the Turtles can do because of their bodies and their power.
Does all the armor and everything end up getting computer-generated after the fact? Can they ever wear anything?
Sometimes they’re wearing things, it’s that whole thing about how do you come up with a methodology for the actors to do what they do. So Donatello has a lot of gear in there so when he’s on the set he has some stuff that he handles, there are some props that are hero props like the swords, and the nunchuks, and some of the weapons, they are real hero stuff because they throw it around, they show up as props. So it’s a combination of digital filmmaking and real filmmaking that we’re trying to do.
It looks like they all have really different armor and clothing, a lot of different choices here. Is any of it cobbled together from various sources?
I think that was a mandate, to make sure that if we’re going to go into this organic way of making a movie that the four characters are completely different. I mean, if you were to cast a movie with actors, you normally want four people that interact with each other. If you want to say, you know, he’s the funny one, he’s the strong one, and angry, he’s the smart one, and he’s a leader. So when you make those choices, you want those actors to reflect those kinds of behaviors and you want the characters to be completely different so that’s why their faces are completely different from each other, too. And they act differently, completely differently, and being that this is an origin stor,y you go back to seeing them grow a little bit and you understand why they act the way they do.
Were these created before they cast the actors or…?
They were created before because it took, like, a couple years to come up with these and say, “This is the movie we want to make.”
What’s the closest reference – is there a specific book that’s the closest reference for these?
Well, I mean, you know, I don’t know about other people, but for me it is about real references, so what you do is a study on the character. You say, ok, this guy is probably a Tom Hanks kind of a person, a heroic guy, quiet, this one is a very smart guy. So you go back to movies and you put clips together that more or less exemplify what kind of person the characters are. The funny one, and the very strong and angry one, like I said. So you do kind of a bible for every one of the characters, so when you’ve left the set after three months of shooting and then going to post-production for about a year or so, you’re not dealing with actors anymore, you’re dealing with animators that are basically getting the performances from the actors into these. Plus, whatever happens in editorial, and needs to be changed.
Do the actors need to learn how to move differently? How does that differ from humanoid?
Yes. I think it is humanoid movement, but it is about a pose. That’s why when you see these things they’re not in a rest pose, they’re acting. Every one of them has an acting position and a pose, a body pose the actors get into with the characters. We talked about the [characters] a lot, who these people are, given references and also because they’re four brothers it’s a lot about how they relate to each other and what kind of chemistry they have with each other. The funniest things happen when the camera is off. If you just take a look at the actors they’re really funny with each other and they play around. Sometimes we just look at them and record some stuff that they don’t even know, just to see how they interact with each other and incorporate that into the movie.
Does the audition process involve them putting on the suits? How do you know these guys can handle wearing a motion capture suit?
I don’t know, I think it was something that was told to them. But it wasn’t about that, it was about finding the character first – finding the voice and the tone and also seeing how they interact with each other. And it wasn’t about wearing the suits, although, besides the helmet they’re wearing, there’s a – which is also a new technology, a motion capture technology that we’re developing with those grey suits – they’re not just grey suits with marks, they have a specific computer pattern […] all over their bodies, so the computer knows exactly who every one of the characters are.
So they have to act like that. That’s why it’s important for those people to be on the set, […] the blue screen stage, with smoke, with the actresses, with Megan and with Will, with Shredder, with Bill, with everybody else on the set and doing the dolly pieces and acting out the physicality of what they do.
Teenage Mutant Ninja Turtles hits theaters August 8, 2014.
Follow me on Twitter @benandrewmoore.
Can you talk about the decision to make Shredder an all CG character and why that needed to be – ?
Actually, Shredder is not an all computer graphic character, Splinter is.
The PR people had mentioned Shredder.
Yeah, at the end there’s a section [where] he does some natural things, so in that case he’s half what we shoot and half CG because we only built 50% of the suit, but it’s Bill Fichtner who is wearing the suit.
Is he a human then for most of the film and then he becomes something else?
Yes, he’s human until the end and I can’t tell you what [happens] at the end, so that’s why. But even at the end, the movie is completely grounded and that’s also Jonathan’s view [and] Michael Bay’s view that everything has to be grounded and real.
Now, Splinter is a complete CG character because he’s a rat. Although, we do have an actor for it that is doing the voice and is interacting with the Turtles, especially in the later set which is where they do most of the talking. But then he also gets captured like that and retargeted as a rat.
Does he also fight?
He does fight, yes.
And how big is he compared to these guys?
He’s about 4′. [At this point, Helman unveiled a miniature of Shredder.] That’s Shredder. So we built up to here, you see, and all this stuff is going to be CG because of these dots. But it is a physical suit.
How tall is Shredder?
He’s, I think, 6′ 7″. He’s pretty much like a Turtle. Michelangelo is the one that’s 6′ 2″, the smallest one.
Is there a specific sequence in the film that excites you in terms of visual effects? Something that’s never been done?
You know, [it’s] the whole visual effects endeavor, in the sense that the performances are what we’re going for. If we don’t have the Turtles, we don’t have a movie. That’s why the whole research and development effort that we put together with ILM has to do with performance capture and capturing nuances in the delivery of the behavior up to the point where we’re tracking the eyeballs completely. So if somebody is moving their eyes, we’re tracking that into a computer.
We’re basically looking for what’s called a turn, an adjustment, in which the actor changes from one behavior into another. We would like to see the actor thinking and changing behaviors. And that’s the most difficult thing in CG. In CG you can be happy or you can be sad or worried, whatever, but the transformations in going from happy to sad, that is the most difficult thing. It’s that thing you see in an actor and you get emotional about it because you see that person reacting to something. That is so difficult with CG. That’s why we wanted to capture that from real actors.
So you feel like this will be the first film that audiences will see that kind of emotional transition in characters?
I do, even from a scientific point of view. The fidelity that we’re doing it from, capturing this is something that we haven’t seen before. It’s terabytes of stuff, to the point that part of the research on the ground that we have to do, has to do with developing the pipeline that can handle that amount of information.
Is there anything in the film that the actors can’t motion capture themselves, where it’s up to animators to pull it off?
Sure, that’s the thing that is also part of the research and development, it’s how to get 80% of the performance and then also being able to just go and touch parts with animation, or add a wing, or add or change a line – those kinds of things that you would normally do on the set. So that’s part of this whole thing.
How much of a suit is fitted to the specific actor? I even see these are their faces.
All of them, because these helmets are fitted to every one of the actors. This helmet doesn’t fit Leo – it’s only for that character. You can see the [markers on the suit[ there so the computer knows which character is which. I’ll tell you, I’ve done many films and I have never, I mean, I’m always trying to do films at the edge of technology, pushing the stamp, but this is something that I’m really excited about.
Five years from now, how would you want to see this specific technology evolve?
I think you always want to have goals for performances, we’ve seen the whole visual effects technology. If you follow the films in the last ten years they are always moving towards trying to get performances out of the actors into the digital realm. And the whole thing about that, the press is always saying how that doesn’t look human, or that doesn’t look creature-ish, or that doesn’t look this or that.
The idea is to, if we start from a scientific point of view, capturing that performance, how do you translate it into a computer? It’s kind of a psychology, kind of a quest. You really want to get into the psychology of the behaviors and the choices people make because the only thing the computer does is mirror that, it doesn’t know why they’re making their choice. You see that in an actor, though, you see how somebody is thinking about something that changes; everything is connected. The computer is only mirroring that, so if we capture that from an actor then we can get at those things.
It’s a complicated thing because if you listen to how directors arrive at characters, they have conversations with the actors. They develop that character for four or five months before they shoot. Then they shoot for a very intense ten week period and they develop relationships […] within themselves. All that, at the end, gives you a performance that is believable and makes sense in the movie.
In CG we don’t have that, we have 15 animators that are basically dealing with the director. And every animator has a different way to animate each of the characters. It’s the same as if the director were to cast Robert De Niro for one scene and then when you come back to the same scene or reverse [and] cast Al Pacino for the same character, it’s different. Animators are different. It’s a real puzzle, how you put this together. That’s why I jumped at the chance of doing this movie a different way, and at the same time [a] grounded [way]. I would like at the end of the process for people to see the movie and they can’t stop looking at the characters, thinking about, knowing it’s a visual effect but not knowing how we did what we did.
Teenage Mutant Ninja Turtles hits theaters August 8, 2014.
Follow me on Twitter @benandrewmoore.
Are Shredder’s weapons and cape practical then?
Yes, he has some weapons, but some of them move. For instance, if you see this [suit], this is like the end of the movie, but at the beginning of the movie there is a kind of escalation technique. You start with not as much [of the suit] and [it] moves, and he does different things with it.
Do we see his face much?
Yeah, we see his face in the whole movie, but you don’t know who he is. You don’t know that he is who he is.
So he’s a character in a film that sort of reveals himself?
Yes, as Shredder.
Have you guys started animating already with the data you have?
We have started. This is the performance part. There’s also the action part and the action part is mainly done with [stunt coordinator] David Leitch, orchestrating the fights. All that stuff is done with stunts, and the stunts, they’re not wearing helmets because the performance comes from the actors, but the stunts are wearing the motion capture suits and we’re capturing that motion. That gets captured in scenes […], so with stunts we’re capturing all that motion and we’re replacing the Turtle parts with CG versions of them.
And then we’re doing what’s called a volume capture with the actors so that they can give us the face performance of those stunts. So they look at the performance and they act, however much exertion or punching or pain […] or screaming will be in there too. So once we do that then we marry all those things, the motion capture of the stunts, the performance of the actors and then you replace all the actors and everything we need to with digital aspects.
It’s such a big project. It’s so different in the sense that if you were to graph a visual effects movie, you start with, let’s say visual effects and then character development and visual effects, then character development – every time you have character development, you have [to have] visual effects because these are the actors. So, that’s why it’s such a challenging thing because you end up with so many shots.
Are you doing any set pieces that are not in practical locations?
All of them [are practical]. Yeah, basically that’s why we came to New York. All the pieces that we are doing like all the rooftop stuff we’re doing is in New York. It is at night, so we’re working nights. All the stuff we’re doing like in sets like this one, which aren’t real sets, but real locations, they’re real sets. And the end of the movie is also going to be shot in a very prominent part of New York, which is very challenging because it’s like four people. It’s going to be a real nightmare for everybody.
Does that involve recreating New York in some capacity?
Yes. Obviously the structure part of it. But it’s going to be seen from really high up.
Are you doing any work on the environment, too?
Well, yes. On top of everything I just said in terms of performance, then you have all the action part of it and the action has to do with destroying parts of New York and doing a very scientific survey of different parts of New York and recreating that and then rigging it for destruction so that it can be assimilated for very complicated explosions. I’m not just hyping it, I really am excited about all the development part of this. What I do is pretty geeky, I know. [Laughter]
How much time will be spent in the volume?
It kind of depends on reshoots and things like that that need to happen, but basically while we do stunt work we develop a list of things that need to be captured later. I would say probably 2-3 weeks of volume work will be done.
Can you talk about whether visual effects are involved transformations? I feel like Turtles are so connected to the transformation sequence and it seems like they’re mutating people. I guess that’s a plot point, so I’m curious if there’s a lot of designing for visual effects?
There is not much transformation per se; we’re going to see how they transform, if that’s what you mean, but from a human point of view. We’re just going to see how they grow up a little bit and how they become who they are. But not in a Hulkish kind of way. It’s more a gradual thing for them.
The old movie had baby turtles dancing in ooze – didn’t know if there’s a more evolved version.
If you take a look at the science, they’re pretty heroic and they tell a lot just by looking at this. I mean these are creatures, that if you are going with it, if you are just following the logic of the mythology of this, these are creatures that are not being accepted by the world, so they feel that way. And that’s part of the movie, too. How did they grow up to be isolated? They’re always fighting in shadows, they’re never seen outside. They’re always going through manholes because they don’t want to be seen. But they also don’t want to be seen because they feel somehow short-changed; they’re different from everybody else. That’s part of the movie.
What happens to all of these models?
You mean after [the film is done]? Some of them go to ILM. Some of them go to Jonathan’s office or Michael Bay’s office. That’s it. It’s really cool to start the work with mock-ups, that’s something you don’t have to do because we’re working with the computer and they’re digital characters so you don’t have to do that. But, my background is from trying to not do just only one technique of computer. Just because you can do it on a computer doesn’t mean you should do it on a computer. I do a lot of miniature work, too in the other movies I have done and I’m going to keep doing that because I think it’s the best way to get there.
Teenage Mutant Ninja Turtles hits theaters August 8, 2014.
Follow me on Twitter @benandrewmoore.