What Facial pass techniques do you use on R3D ?

[PatrickFaith]

I have about a half an hour of shots I need to do a bunch of facial passes on. Sometimes it's a blemish, other times it's clean plating a section and possibly overlaying something. Person is almost always moving, so it's not this easy photoshop type thing. I tried using AE but I was going insane moving the keyframes to maya or nuke, and frequently couldn't complete the facial work in AE. Maya is ok, but the integration to the r3d just isn't there. I'm mainly using nuke rotopaint nodes with integrated 2d trackers (sometimes doing projections, but I am crappy at that). I have a ton of work on this at 4k, so any hints, tutorials or classes I can take would be very helpful. Couple of classes I've liked on digital tutors are "fiery skin effects in Nukex" and "Digital-Makeup-Using-Facial-Tracking-in-After-Effects". I really like the new AE facial tracking, especially how it integrates with painting, but the lack of AE integration with nuke and maya has driven me crazy(i.e. copying ae keys to a notepad is beyond hacky). Struggling a bit, and trying to get mastery of tools where I can get all the way to full A level feature quality.

 

[David Battistella]

Mocha pro has some tools built it specifically to make this a speedier process and track and replace right in the software.

You can also send Mocha's tracking data to Nuke.

You have to get your head around the fact that Mocha is a planar tracker, but then it's all cupcakes from there.

 

[Björn Benckert]

https://www.youtube.com/watch?v=Qo6iBmYnqh8

http://www.nukepedia.com/toolsets/frequency-separator

https://www.youtube.com/watch?v=R6hFtbufIPs

https://www.youtube.com/watch?v=LE5e2GF7lOI

Frequency Separation, is something you can do with a few nodes also in nuke. Use it all the time.

 

[PatrickFaith]

Mocha pro has some tools built it specifically to make this a speedier process and track and replace right in the software.

You can also send Mocha's tracking data to Nuke.

You have to get your head around the fact that Mocha is a planar tracker, but then it's all cupcakes from there.

Thanks, I haven't used mocha for the last couple of years. Where I am going crazy is exporting the facial tracking data that I created inside AE, like the percentage of a eye open(which i would tie to a model in maya, then recomposit in nuke). Is there a way of moving the facial tracking data from AE into mocha, then output that from mocha. My history from mocha was doing the tracking in mocha then copying that back into AE, but can't figure out how to move keys generated inside AE to mocha (then out to nuke).

 

[PatrickFaith]

https://www.youtube.com/watch?v=Qo6iBmYnqh8

http://www.nukepedia.com/toolsets/frequency-separator

https://www.youtube.com/watch?v=R6hFtbufIPs

https://www.youtube.com/watch?v=LE5e2GF7lOI

Frequency Separation, is something you can do with a few nodes also in nuke. Use it all the time.

I've never heard of frequency seperation, that is very cool ... watching those videos ... thanks.

 

[fred gasc]

I've never heard of frequency seperation, that is very cool ... watching those videos ... thanks.

Indeed usefull. Thanks.

 

[Marc Wielage]

Struggling a bit, and trying to get mastery of tools where I can get all the way to full A level feature quality.

I think A-level feature quality requires experienced union workers, particularly makeup artists (who are costly but make a huge difference), great lighting, great lenses, proper filtration, and experienced colorists. A plug-in or VFX will not do this. To me, 90% of it has to happen in front of the camera first.

Having said that: I can and do use combinations of windows, keys, negative midtone detail, and NR to soften skin on occasion. I often find you can get better results with that than you can than using traditional blur or mist modes. The 3D keyer within Resolve 12 helps a great deal -- that's a very powerful tool. Another key is to slightly sharpen the eyes after defocusing the skintone, which helps give the faces a dramatic pop. Qualifying the skin and omitting eyes, lips, hair, and so on is a given; sometimes hold-out tracking mattes are necessary to pull this off.

 

[PatrickFaith]

Having said that: I can and do use combinations of windows, keys, negative midtone detail, and NR to soften skin on occasion. I often find you can get better results with that than you can than using traditional blur or mist modes. The 3D keyer within Resolve 12 helps a great deal -- that's a very powerful tool. Another key is to slightly sharpen the eyes after defocusing the skintone, which helps give the faces a dramatic pop. Qualifying the skin and omitting eyes, lips, hair, and so on is a given; sometimes hold-out tracking mattes are necessary to pull this off.

Generally my r3d shot is coming in perfectly, but I have a lot of things like tracking points on faces where the augment needs to occur (i.e. dirt/wound/water/object) and/or adding water drops to eye lashes. A good example would be Phil's shot of that model that has the dragon makeup on, and I need to add a reptilian second eye lid blink every now and then to "sell the shot" to the audience, so that added "effect" has to be seamless with the r3d. In order to do this I'm doing a lot of detailed facial tracking, then vfx merges with the r3d in the comp.

With David's suggestion and with what your saying about hold out masks, what I think I need to do is master the new nukex 9 planar tracker which kind of works like mocha but on steroids. The new nukex 9 planar tracker is pretty amazing, but isn't initially obvious how you clean plate and augment skin / facial features (i.e. it will do the mask while aligning the mask to a 3d planar tracker, then allow vfx feathered composite on that plane within the mask). The new planar tracker also doesn't slip like the old one, and you can do multi point spline masks on the plane that is tracked (rather then the old days when the planar tracker was more for rectangles on building sides, or for screen replacements).

 

[Björn Benckert]

Generally my r3d shot is coming in perfectly, but I have a lot of things like tracking points on faces where the augment needs to occur (i.e. dirt/wound/water/object) and/or adding water drops to eye lashes. A good example would be Phil's shot of that model that has the dragon makeup on, and I need to add a reptilian second eye lid blink every now and then to "sell the shot" to the audience, so that added "effect" has to be seamless with the r3d. In order to do this I'm doing a lot of detailed facial tracking, then vfx merges with the r3d in the comp.

With David's suggestion and with what your saying about hold out masks, what I think I need to do is master the new nukex 9 planar tracker which kind of works like mocha but on steroids. The new nukex 9 planar tracker is pretty amazing, but isn't initially obvious how you clean plate and augment skin / facial features (i.e. it will do the mask while aligning the mask to a 3d planar tracker, then allow vfx feathered composite on that plane within the mask). The new planar tracker also doesn't slip like the old one, and you can do multi point spline masks on the plane that is tracked (rather then the old days when the planar tracker was more for rectangles on building sides, or for screen replacements).

build a 3d head/face as a simple model. could be done from image based modeling does not need to be perfect, then 3d track it in place and project your stuff on to it.

best is ofcourse to make a perfect head like this but it does not have to be if you reproject back the original image : https://vimeo.com/28449088
when you get the 3d projection in place you can use it to blink talk or for retouch: https://vimeo.com/28346749
same projection based modeling we used for all the goats head movements and eyes that where just solid blacks. and also to remove square hole on the noses that where there for the actors to look out trough : http://www.syndicate.se/peptobismal-boy-rased-by-goats/

 

[PatrickFaith]

build a 3d head/face as a simple model. could be done from image based modeling does not need to be perfect, then 3d track it in place and project your stuff on to it.

best is ofcourse to make a perfect head like this but it does not have to be if you reproject back the original image : https://vimeo.com/28449088
when you get the 3d projection in place you can use it to blink talk or for retouch: https://vimeo.com/28346749
same projection based modeling we used for all the goats head movements and eyes that where just solid blacks. and also to remove square hole on the noses that where there for the actors to look out trough : http://www.syndicate.se/peptobismal-boy-rased-by-goats/

Those are awesome, I think the peptobismal has one of the best emotional impacts of any ad i've seen. For my film is easy for me to get a 3d obj of all faces and I found a nuke tutorial at cmivfx that lines up the model with the shot similar to how you explained it : https://cmivfx.com/store/234-nuke-face-replacements .

 

[PatrickFaith]

I btw just had a class from Victor Perez on transform ordering, especially how it effects heavy transforms on the face with high rez shots. I see a lot of shots where the face gets "blurry"/glossy when a lot of operations are going on. Victor explained how it's important to have the transformed grouped, also how filtering on the transforms need to be grouped. Another thing I was doing was spreading out my rotopainting, but that was distributing my transforms, also creating blur. Anyway I can't thank enough people dealing with daily productions taking time out to do classes.

 

[Mike Nagel]

Patrick,

just to share my experience:

latest Mocha will beat the Nuke 9 planar tracker most of the time... and u can get the data very pain free into Nuke... there is also MochaImportPlus, which is an import script that makes the workflow a bit easier (not required though)...

Mocha tracking data is superior... if u have tracking points on faces (which makes everything easier), then Nuke tracker may provide great results as well...

I would only use Mocha for tracking data though, do all VFX beauty work in Nuke... freq sep, traditional paint tool / clone tool retouching etc, which - once u have good tracking data - is pretty much like Photoshop...

on some general skin touch ups, u can animate the fixes manually over the timeline in Nuke, use key combo to jump 10 frames forward and move fix into place... for some skin stuff it does not have to be perfectly in place, and will be much faster than getting tracking data every time...

the 3D head modeling is great, if u can get it done properly without making the skin look stretched and unnatural in the final comp (when re-projected)...

and yes - obviously - use a merge node (--> diff operation) to QC that u really only affected the parts of the image that needed manipulation...

- M

 

[PatrickFaith]

Patrick,

just to share my experience:

latest Mocha will beat the Nuke 9 planar tracker most of the time... and u can get the data very pain free into Nuke... there is also MochaImportPlus, which is an import script that makes the workflow a bit easier (not required though)...

Mocha tracking data is superior... if u have tracking points on faces (which makes everything easier), then Nuke tracker may provide great results as well...

I would only use Mocha for tracking data though, do all VFX beauty work in Nuke... freq sep, traditional paint tool / clone tool retouching etc, which - once u have good tracking data - is pretty much like Photoshop...

the 3D head modeling is great, if u can get it done properly without making the skin look stretched and unnatural in the final comp (when re-projected)...

- M

Excellent info, thanks. That's pretty much what I'm finding in that the nuke 9 planar tracker is real good on the face as long as the head doesn't drastically move of the plane - then it falls apart.
Foundry said they will totally update this in february beta, also I'm hoping for some magic on connecting that with mari. Main issue I have on integrating mocha with nuke is sending the tracking of the eye data over to nuke, is just way too hacky right now (very doable but I hate editing text files and such, for the amount of data I have is just too time consuming). I'm actually just putting on a normal tracker on the eyes for now in nuke for a few shots, and pushing the "eye" comps till february where I have to make a final decision. I do have a good tracker on the head, 240 fps with the xsen suite (i.e. all actors where the suite during filming, but I don't have the fixed head camera looking into the actors faces like they do for avatar/hobbit/planet_of_apes) . I btw haven't really found a good object tracking approach inside nuke, I have to go outside nuke for object tracking, which adds workflow complexity/cost. I really need to get a clean nuke object tracking workflow going, as I have a bunch of projection mapping work I need to do fairly soon in nuke (i.e. project a r3d on a nuke 3d object and rescan that object back to a 2d plane). Thanks for any suggestions, I'll look into them all.

 

[Mike Nagel]

Patrick,

are u sure u are using the correct workflow for the eye ?

Mocha is a planar tracker, it will provide the same data scheme as the new Nuke 9 planar tracker, it will just be more accurate in most scenarios... ;-) and there is absolutely no need to edit text files...

u track in Mocha just the eye - or track the head and then export the eye shape linked to the head track (this is a Mocha workflow that u gotta wrap ur head around, which can save quite some time) - it depends whether u need the eye socket movement or the pupil itself > export either to a Nuke Tracker node or Corner pin node... there is no need to to use the old ASCii txt files at all, but u can if u want to...

and u can copy the Tracker node to the clipboard and then paste it in Nuke... it literally takes 2 secs... I love it

naturally: QA/QC ur track in Mocha before u copy to Nuke... if it is off, re-run those frames only and get better tracking data - Mocha has tools... only jump to Nuke once u have good data - save urself the time from doing it multiple times...

I use the Tracker node (CornerPin can have odd results) > set to stabilize > create VFX/beauty fix nodes roto etc > add a copy of same Tracker node set to match move... done...

if for whatever reason the tracking data is off at some points in the timeline (which if it is a very difficult track can happen), u could add a Transform node in between the 2 Tracker nodes and apply a transform just on those frames (that are off) to nail stuff into place...

also u have to be careful how u organize ur pipes in Nuke... if u segment off the main pipe and then use the stabilize/match move approach and then merge back with the main pipe, the results can be different than doing the stabilize/match move in the main pipe (I don't remember top off my head, just putting it out there in case it does not work as it should)...

the 3D head model approach is the best, but it takes time so for smaller fixes unnecessary and a few times I've seen it, the skin looks generally artificial afterwards (even on those parts that weren't retouched)... but I'm sure that is due to inferior technique being applied here...


I wanna see that movie when it's done !!!

- M

 

[PatrickFaith]

Mike, I was using the new AE with an ancient copy of mocha. Actually i think there is some mocha deals going on right now, i should just buy the new version of mocha using the black friday deals.
(that is ... i do the face tracking using the new AE functions that give those keys, and outputting the face tracking keys from AE was my problem where I had to go into text mode, looking at mocha pro I still can't see how I output the AE face/eye/blinking/etc tracking info to nuke via mocha)

 

[Mike Nagel]

yeah, u need the standalone interface - it reads .r3d files and u can lower resolution to get faster playback - depending on shot the track will still be good... I'm not sure if u can export from AE Mocha to Nuke directly.... I'm sure they have a trial version...

 

[PatrickFaith]

Hmmm ... i think the catch 22 is that mocha pro is standalone, so doesn't have the AE face tracking stuff. Mocha Plus inside AE does have access to the facial stuff, but Mocha Plus can't export to nuke. I guess if I bought both mocha plus and mocha pro, I could output inside AE the tracking info with the facial key data in an AE format. Then in mocha pro in stand alone mode I could read the AE format then output to nuke. Even with the 50% sale that would be around 800$ though (plus mastering nuke/maya/protools is pretty much completely tasking my brain).

 

[Mike Nagel]

contact them and see if the new Mocha in AE with the facial recognition can export a format for Nuke... if not get trial versions and test this... I doubt they implement something for AE that is more sophisticated than the standalone version... the facial thingy is prob more of a helper... doesn't mean it is more accurate than tracking facial features urself...

they got tons of free tuts on their site for all kind of scenarios including beauty retouch so check that out... they use the standalone most of the time... it's all just about getting the best tracking data, and Mocha is fast...

 

[PatrickFaith]

contact them and see if the new Mocha in AE with the facial recognition can export a format for Nuke... if not get trial versions and test this... I doubt they implement something for AE that is more sophisticated than the standalone version... the facial thingy is prob more of a helper... doesn't mean it is more accurate than tracking facial features urself...

they got tons of free tuts on their site for all kind of scenarios including beauty retouch so check that out... they use the standalone most of the time... it's all just about getting the best tracking data, and Mocha is fast...

By working with this, and taking the concept you mentioned about stabilizing the face first, I've decided to do the faces with "practicals" during ADR. Basically film real actors faces "fixed"/head_not_moving and/or stabilized from the ADR session, then since I have an exact model of the face moving in 3d for that face in a scene (which was obtained via motion capture) , I'll project the ADR session of the actor as practical onto the 3d virtual surface in nuke, then render the surface while compositing. It cracks me up doing face shots with my red dragon during the ADR sessions as vfx practicals, and using those practicals for 3d projection maps for vfx rendering onto the 3d map of the actor via the xsen suite angles (basically virtually hang a projector on the actors head, and project the adr session onto that virtual head during compositing).

I normally think of shooting my red camera and adding vfx, not doing a scene fully in vfx and projection map practicals of actors onto that vfx model - I need to get with the program (I wonder if TRON or Lawn Mower Man was done like that ... but more mechanically? )

 

[Mike Nagel]

why are you doing it that way ?

why not directly manipulate the motion capture head model ?

the actors will have to get make-uo etc if u shoot them when they come in for ADR plus they gotta hopefully act as good as they did in the take u chose for the edit, which is usually the best take out of a few...