CG HDR Experiment

[Gavin Greenwalt]

DISCLAIMER: THIS IS NOT HDRx™ SO DON'T JUDGE THIS AS RED TECH.

Just wanted to see what I should expect from a workflow and limitations standpoint when working with the Red HDRx™ system.

I rendered off two passes. Once which is the (-0.25) offset frame without motion blur and one centered on (0.0) offset with 0.5 frame motion blur (180*) which is pretty close I believe to what you would get from HDRx.

Of course my system is just a crude slap together solution and purely experimental/speculation/educational but it might give an idea of the situations in which HDRx will perform great and when it will fall apart.

If you have a large overexposed area such as thin branches for instance in this shot you'll almost certainly lose them completely in the normal exposure. Which means the entire canopy in a shot like this will probably need to be filled in with the HDR exposure. Magic HDR I believe will not produce a desireable result so this would be a case where MNMB would be necessary. Luckily it looks like optical flow in such a situation would do a great job in motion for the most part. Where it didn't do a great job was in the flailing arms. I actually increased the dynamic range of my hypothetical "normal" exposure so that his arms would be exposed properly for the most part in the normal exposure. This will probably be important to keep in mind: Make sure your flapping and flailing bits which are moving quickly stay in the normal exposure for best results. This might be impossible though with a very bright background in which case even a little motion blur clips.

The optical flow did a pretty good job but there is still some wonkiness where there is a lot of parallax and overlap. The more parallax and overlap in my experience the worse optical flow in general performs. Also it has a really hard time with fast moving reflections since you have two different vectors in the same pixel which is impossible for it to discriminate between so reflections in cars I imagine would cause a little trouble as well.

Anyway, this was just for my own personal education to give my brain something to look at and reflect on. It's kind of like what RED is probably doing as far as I can tell so I imagine the trials and tribulations we see when HDRx is released will be similar.

*hides from Jim's Goat*

http://www.mediafire.com/?f6o63rfga619v

 

[Zach Gray]

I did the following test quite a while ago. Funny how it looks pretty similar to HDRx™. Not quite the same, since I was averaging three exposures, but still, looks like Magic Motion...

Gavin, I wonder if the oflow will consider a group of frames or work with single frames. As long as the oflow doesn't pop and dance around, I doubt most people will care. If it's searching and adding funny distortions when it fails or retracks, that could be strange.

Folks kept talking about motion blur artifacts, and I wanted to see what they might look like. Judge for yourself.

The two most realistic ideas seemed to be combining images shot at high frame rates and a rapid read with no reset.

Maya/MentalRay allows you to specify the shutter angle, and also create a motion offset where only a portion of the time of the frame is considered in the calculation. I didn't render the images at different exposures in this experiment, but that would be a given with changing shutter speeds.

 

[Deanan]

I did the following test quite a while ago. Funny how it looks pretty similar to HDRx™™. Not quite the same, since I was averaging three exposures, but still, looks like Magic Motion...

Gavin, I wonder if the oflow will consider a group of frames or work with single frames. As long as the oflow doesn't pop and dance around, I doubt most people will care. If it's searching and adding funny distortions when it fails or retracks, that could be strange.

Hmmm, not really similar to HDRx for both yours and Gavins.

 

[Gavin Greenwalt]

Hmmm, not really similar to HDRx™™ for both yours and Gavins.

The two passes (Normal 180* exposure and short <10* exposure at start of frame) or the cheap "HDRx" Easy/MNMB processing? Or Both? Or is that asking too proprietary of information?

Do you have any plans to release a C++ API for HDRx™ blending to arbitrary images or is it part of the debayer process?

With the capabilities it offers I imagine just about every shot is going to have HDRx™ turned on now which means every FX plate is going to need to match the look and feel of HDRx™. We could write lens shaders which emulate something which looks kind of like it through trial and error or will RED be releasing pretty exacting specs under some sort of "Do not use on non RED projects" license?

This test was a good break in for me to see what gets the look and what doesn't but being able to write a Nuke node to take in two plates and spit out an exactly matching plate would be very nice. Of course this means that we'll be rendering double of all 3D plates that would need to match. Stereo style rendering (render hero eye, cache shaders etc...) could make the impact minimal. But ughhh Stereo and HDRx™? You just know everyone is planning on doing it.

 

[michael zaletel]

Why do you guys keep digging? RED shouldn't even be sharing this information with the public. Do you really expect them to explain the real reasons why your speculations aren't even close on a public forum all of their competitors are reading hourly?

I, for one, hope they don't say another word. They definitely don't need to. All they need to do at this point is figure out how to profitably manufacture and efficiently distribute a bajillion of these things and support them at the quality of service RED has become known for. As far as I can tell, that's the only challenge left... and likely the most important.

-michael zaletel

 

[Gavin Greenwalt]

Why do you guys keep digging? RED shouldn't even be sharing this information with the public. Do you really expect them to explain the real reasons why your speculations aren't even close on a public forum all of their competitors are reading hourly?

-michael zaletel

Protecting IP is the job of patents.

Matching CG and Photography is the job of compositors. Also if you don't know how it works, when it works best and what causes problems you're apt to just wander out into the jungle shoot 3 weeks of footage get home and discover a quarter of it could have looked better by adjusting your shooting or exposure slightly.

Lower animals solve things through trial and error. Advanced tool making species understand how their tools work and with that knowledge can use them to their full potential.

If you don't understand how an engine works you are going to spend a LONG time trying to get it to start. But if you understand how an engine works you know where to look, what to check and what depends on what to function. You might even be able to improve on the stock model by tweaking it to run in a manner you prefer.

When you're trying to recreate a photograph you have to understand what you're looking at to find its unique characteristics and qualities. Lenses are imperfect you need to match the distortion, CA and vignetting if there is noticeable amounts of any. Noise vs Grain. Is the noise logrithmic or linear? What's the shutter angle? Is there any bias to the motion blur? Is this anamorphic? What is the aperture like? etc etc etc.

I would say that "Magic Motion" is a VERY distinct look. If half of your frame is a matte painting or 3D asset it can't look completely different.

 

[Álex Montoya]

Gavin, it isn't released yet. Give it a rest. We'll all have time to find out how it works and its limitations.

 

[Jeff Kilgroe]

Protecting IP is the job of patents.

...Until someone incorporates your patented algorithm into their code and you have no way to challenge it other than to offer up comparative image analysis and a bunch of techno-babble that no courtroom audience is likely to understand.

Lots of colas on the market, but Coke hasn't ever divulged their recipe and they probably never will. Do you actually think that a patent on that will keep someone from duplicating it if they were to share it? Do you have any idea how easy it is to minimally alter a logical algorithm to circumvent a patent? Perhaps RED doesn't even have a patent yet? Nor will they finalize any patent applications until they are confident they have applying for protection on the best ways they can come up with for doing it.

 

[Gavin Greenwalt]

Gavin, it isn't released yet. Give it a rest. We'll all have time to find out how it works and its limitations.

Getting tools out the door through a pipeline is not a quick process. Look at how long it took to get REDCode worked into all the main apps. I mean, look at Storm, that's been something of the 'missing' link since before RED announced REDCine.

Stereo is still just trickling out. And its tool-set is still pretty lousy. Everyone is going to want HDRx. I think just about eveyrone is going to use HDRx. So everyone better be ready at every step of the way to handle it.

 

[Zach Gray]

Hmmm, not really similar to HDRx™™ for both yours and Gavins.

Cool. That was just some messing in the old days (February) when folks were talking about how HDR on red could work. Everyone was freaking out about how the motion blur would look. I did a little CG test to see if it would be as bad as people were saying. (on this old thread from feb http://reduser.net/forum/showpost.php?p=555262&postcount=211)

Obviously, it's not that big of a deal and looks good - and you could argue that it even looks better.

Believe me, I've got faith in you guys and the Foundry. You wouldn't put something out that your're not proud of. I also enjoy the technical dialog and really like it when you guys chime in. I feel like I learn a ton when you, Graeme, Stuart share what's happening, and even when Gavin pushes for more info.

I;m really grateful for the open conversation that you guys have, so I don't want you to confuse my interest for lack of confidence in your tech. Thanks for sharing what you do.

 

[Wolfgang Woehl]

Gavin, throw camera motion sensor metadata into the mix and a number of potential problems will simply disappear, I guess.

 

[Michael Olsen]

At the same time, this tool (HDRx) is still going through major development, or so it seems, by two different teams (internal RED and the Foundry).

Perhaps it would be more prudent to wait for the technology to arrive at a more finished state so as not to base foreseeable trials and tribulations off of something which is similar on the surface, but could actually be markedly different in final execution? As you said, this is not HDRx, just a model which is surely imperfect, as HDRx itself is imperfect at this point. Cart before the horse and all that...

 

[Gavin Greenwalt]

Gavin, throw camera motion sensor metadata into the mix and a number of potential problems will simply disappear, I guess.

Only if you shoot static landscapes on a tripod. If you limit your scene to static nodal camera moves optical flow will perform near perfectly. It's when you have parallax and semi-transparent layers moving at different rates you run into trouble and deforming objects. People fit all of those categories really well.

This isn't something though that optical flow could really ever overcome. If you have a piece of silk going one way and are panning the other there is no way for an algorithm to discern which is which so both have to share the same blur. Nor can you expect an oFlow algorithm to ever be pixel accurate without very controlled environments. So it's not really a question of refining--there are definite limits on optical flow and what it can and cannot do even with the theoretically perfect programmer.

Post Production motion blur isn't a new thing. There has been a lot of attention and development put into it. It's just a question of application and integration in this instance.

What I'm most interested in is speculation on how people are going to render and integrate magic motion. Is there a 'third way' besides rendering with and without motion blur or a baked in lens shader?