Faster than realtime full-debayer transcoding to DPX, QT, etc

[laguun]

What do you use for color correction that allows renderfarming?
...
The only devices I know of that can currently do that in real time are Scratch, Pablo (as a beta), Digital Vision (don't know if they've actually delivered that or not), and possibly Speedgrade, although I don't know if they've delivered on that or not either.
So what are you using that lets you process color corrected files on 64 cores?

speedgrade has nice renderfarming.
http://www.iridas.com/metarender/
But we dont have enough licenses to use the speedgrade here in Berlin (yet) - the license is used mainly in Hamburg for comercials. However it might be necessary to add one here because the speedgrade render workflow is pretty nice.

In Berlin we are still using an in-house tool for the renderfarming which is interfaces to the discreet systems. Still some NDAs (not with discreet btw) in place & the the inhouse tools will remain inhouse for the foreseeable future as its to complicated (and partly buggy) to be released for public used. The trick is grading using "offline 2k high" and then conforming. The setup however requires lots of manual work. Nothing for one/two-man operations.

Anyhow, many German customers have used the service since its introduction in Q3/2008, among them several highly reputated one. The record so far is held by "douglas" - they shot their making of on red and had ~ 18 hours of footage, which took 4 shifts.

Its amusing that especially videoclips (as the #1 german chartposition Polarkreis 18, which we have done digital lab for) now shoot a lot of footage.

...you're pretty much looking at Scratch or Apple Color for this. Apple Color can't do full debayer. Scratch can't render farm. Neither can Pablo or Digital Vision.

Fine tools, however at this moment in time, I wouldnt recommend both for feeding larger render infrastructure.

And, BTW, allow you to play it out to tape, which will take four hours (plus setup) by itself?

Yes. We offer HDCAM and/or blu-ray "dailies". The process isnt sequential, but paralell, playout starts in the mid of the night.
If necessary SR, but most people then go completly uncompressed via dpx. Also prores, but the prores-encoding then is the bottleneck as we have few osx pcs.

 

[Adam Glick]

So, Laguun... Are you saying you have a working solution for distributed processing of graded r3d footage but you can't tell us what it is because of an NDA?

 

[M Most]

In Berlin we are still using an in-house tool for the renderfarming which is interfaces to the discreet systems. Still some NDAs (not with discreet btw) in place & the the inhouse tools will remain inhouse for the foreseeable future as its to complicated (and partly buggy) to be released for public used. The trick is grading using "offline 2k high" and then conforming.

The bigger trick is getting a Discreet system to handle R3d files directly, which, at least publicly, they don't do, as they are all based on Linux (with the exception of a Windows build of Lustre, which to my knowledge doesn't yet have R3d support either). So my guess is that at least some of those NDA's are with Discreet, despite your statement to the contrary.

The biggest issue with render farms and color correction software is that most modern color correction packages rely on hardware support for implementation of the corrections, primarily NVidia hardware. A complete software implementation of the NVidia calls would be necessary for a render farm approach, and in this particular case, that would have to be combined with a software based decompression and debayer step, as well as the scaling. I don't know of anyone who's implemented that yet, but apparently you do :wink:

 

[laguun]

So, Laguun... Are you saying you have a working solution for distributed processing of graded r3d footage but you can't tell us what it is because of an NDA?

no.
r3d processing, distrubuted render are in iridas speedgrade.
the inhouse tool has some ndas still, as the developpers are undecided what to do with it.

 

[laguun]

A complete software implementation of the NVidia calls would be necessary for a render farm approach, and in this particular case, that would have to be combined with a software based decompression and debayer step, as well as the scaling. I don't know of anyone who's implemented that yet, but apparently you do :wink:

iridas is doing exactly this, but our approach here is much easier (read: primitive).

Grade on r3d in 2K high and then replace with debayered rgb from fullscale 4k.

Requires nothing more than a specialized edl-parser as monkey/crimson in FCP-land for grading. Products as automatic duck (*hint*) are somewhat more sophisticated.

 

[M Most]

iridas is doing exactly this, but our approach here is much easier (read: primitive).

Grade on r3d in 2K high and then replace with debayered rgb from fullscale 4k.

Requires nothing more than a specialized edl-parser as monkey/crimson in FCP-land for grading. Products as automatic duck (*hint*) are somewhat more sophisticated.

Now I think I understand. What you're doing on the farm is debayering and rescaling, then you're taking that resulting RGB image file and substituting it for the proxy you used in color correction. This is what I thought you might be doing, and it basically doesn't answer what I posted needed to be done, which is to create HDCam delivery elements in real time, at high quality, with creative color correction included. If you need to do this, and deliver it at the end of the day, you cannot take time to render anything, regardless of how quickly you can get that done. It must be treated as a tape to tape session would, which is essentially a real time process. I'm not saying this is good, bad, or sensible, but it is what a number of clients expect, and it is the reason a lot of them won't shoot Red for short turnaround material. We can accommodate them using Scratch because it can work in the manner and in the timeframe they expect.

 

[laguun]

If you need to do this, and deliver it at the end of the day, you cannot take time to render anything, regardless of how quickly you can get that done. It must be treated as a tape to tape session would, which is essentially a real time process. I'm not saying this is good, bad, or sensible, but it is what a number of clients expect, and it is the reason a lot of them won't shoot Red for short turnaround material. We can accommodate them using Scratch because it can work in the manner and in the timeframe they expect.

Hmmm... we deliver *full debayer* on *4K* for hours overnight, having multiple hdcam/blueray burners feeded paralell. I dont understand how you would be able to get
a) the quality
b) the speed
c) the amount
with a single scratch.

Maybe now its my turn to misunderstand - how would you deliver, lets say 4 hours, in full (not 25% raw resolution) quality overnight?

 

[M Most]

Hmmm... we deliver *full debayer* on *4K* for hours overnight, having multiple hdcam/blueray burners feeded paralell. I dont understand how you would be able to get
a) the quality
b) the speed
c) the amount
with a single scratch.

Maybe now its my turn to misunderstand - how would you deliver, lets say 4 hours, in full (not 25% raw resolution) quality overnight?

It wasn't delivered overnight. It was delivered the same day. They showed up at 10, and the tapes were shipped out by Fed Ex at about 6PM. The ONLY way to accomplish this was by using 2K High playout directly from Scratch, with no rendering required. One can argue the finer points of perfect antialiasing, and the small nuances that might be gained from doing a full render. But the differences are rather subtle to most eyes, and the fact is that the images were just fine, considerably better than a lot of HD film transfers, color corrected to the client's specifications (in my opinion, that's FAR more important than having slightly better antialiasing that nobody watching on anything less than a 60 inch screen can see), and delivered in the time allotted. The tapes went to a very major pay cable network, and the only word I got back from the client is that both they and the network were very pleased with the results. Like I said, this is neither good, bad, or sensible. It just is.

There's theory, and there's reality. One can argue that all that is acceptable is the "best." Or one can deliver something that is visually very close to that, make clients very happy, and get paid. That's reality.

 

[laguun]

I see.
For one-shift sessions while the day, your scenario makes sense.
However there are many scenarios in which the 75% resolution reduction (2K bayer is less than 1080p) in the workflow you are explaining show up extremly. Especially low-light and nolise are *much* worse if only one pixel out of four is used.

If we start at 10 in the morning, we can only deliver ~2 hours played out in maximum quality until 6pm. Also, compared to the scratch workflow much more computing and network power has to be used, and an assistant colorist is mandatory (for feeding the full quality qeues in the background).

In my opinion, the better quality justifies the additional spending - and the customers, which we show the difference, clearly see the degradation of 2048*1024 bayer on the class 1 monitoring and projection here.

However for TV, last-minute jobs and producers/directors who are happy with 80% instead of 99%, 2k high is certainly highly interesting.

We used 2K post for several TV-commercials which were intended to be gritty, pure 2k jobs or webspots.

And sometimes, our 2k ends up as webcast where all detail is destroyed - i hate seing such destroyed encodes -as this one:
http://www.flankenwechsel.de/image/tvspot/flankenwechsel.html

Btw how does scratch handles 2K on a 4k project - is still only 1k decoding or has this been fixed meanwhile?