Click here to go to the first RED TEAM post in this thread.   Thread: Komodo REDCODE RAW Explained

Reply to Thread
Page 5 of 8 FirstFirst 12345678 LastLast
Results 41 to 50 of 73
  1. #41  
    Senior Member
    Join Date
    Mar 2011
    Location
    Denver, Colorado
    Posts
    348
    Quote Originally Posted by Blair S. Paulsen View Post
    I'd expect the encoder on the DSMC2 cameras is purpose built ASIC for DWT RedCode, so not able to author DCT RedCode. Even if I'm wrong and it's a FPGA, the second issue would be max quality of a DCT bitstream at 300MB/s, which I understand to be the write speed limit of the mini-RedMags. In a theoretical world I see the allure of an all DCT based ecosystem, but as a practical matter I don't think that's going to happen.

    An efficient, client side heavy codec was exactly what we needed to get the digital cinema train out of the station. Now that writing to compact, commodity media in camera at >500MB/s is viable - proposed flavors of CF Express potentially quite a bit more - lower compression ratios and less complex codecs can be employed. Thanks to the ever dropping cost of storage, the bigger data footprint is no longer a killer. Before long we'll forget what wavelets even are... ;-)

    Cheers - #19
    Can DCT be used with all video editors or only Redcine-X at this time, is the file still an R3D? If the compression is different how do programs like FCP X handle this?
    DSMC2 Helium 1402
    DSMC2 Helium 6423
    Reply With Quote  
     

  2. #42  
    Senior Member Bastien Tribalat's Avatar
    Join Date
    Dec 2009
    Location
    Cannes area, France
    Posts
    756
    Quote Originally Posted by Jon Dishler View Post
    Can DCT be used with all video editors or only Redcine-X at this time, is the file still an R3D? If the compression is different how do programs like FCP X handle this?
    It works flawlessly in Premiere and Resolve. (I don't have FCPX so I can't tell you)
    VIDEO EDITING - COLOR GRADING - VFX
    APPLE FINAL CUT PRO, AVID MEDIA COMPOSER
    ADOBE CREATIVE CLOUD, DAVINCI RESOLVE, REDCINE X PRO...
    Reply With Quote  
     

  3. #43  
    Senior Member Robert Hofmeyr's Avatar
    Join Date
    Sep 2009
    Location
    South Africa
    Posts
    759
    Quote Originally Posted by Bastien Tribalat View Post
    It works flawlessly in Premiere and Resolve. (I don't have FCPX so I can't tell you)
    Working in FCPX with latest Red apple workflow software.
    Reply With Quote  
     

  4. #44  
    Senior Member
    Join Date
    Jan 2008
    Location
    Ontario Canada
    Posts
    548
    Quote Originally Posted by Jon Dishler View Post
    Can DCT be used with all video editors
    DCT compression is the basis of ProRes, ProRes RAW, Blackmagic RAW, and going back further, Mini DV / DVCPRO and DVCAM.

    JB
    John Brawley
    Cinematographer
    Ontario Canada
    www.johnbrawley.com
    Reply With Quote  
     

  5. #45  
    Senior Member Bastien Tribalat's Avatar
    Join Date
    Dec 2009
    Location
    Cannes area, France
    Posts
    756
    Quote Originally Posted by Robert Hofmeyr View Post
    Working in FCPX with latest Red apple workflow software.
    Good to know ��
    VIDEO EDITING - COLOR GRADING - VFX
    APPLE FINAL CUT PRO, AVID MEDIA COMPOSER
    ADOBE CREATIVE CLOUD, DAVINCI RESOLVE, REDCINE X PRO...
    Reply With Quote  
     

  6. #46  
    Junior Member
    Join Date
    Jan 2020
    Posts
    13
    Thank you Phil and happy birthday. An inspiration and amazing resource for us all.
    Reply With Quote  
     

  7. #47  
    Senior Member
    Join Date
    Feb 2015
    Location
    Brooklyn, NY
    Posts
    124
    Quote Originally Posted by Phil Holland View Post
    Cosine encoding, chunkier though easier to encode and much easier to decode. Cons, limited compression ratio range and certainly not doing 22:1 to get cinema grade images. But you will have some reasonable range to expand recording times. And the benefit, and I mentioned it early on when I was looking at Komodo 6K footage, I was able to turn my GPU acceleration off and get realtime 6K REDCODE HQ playback. That will allow more modest hardware to be used, speed up encoding and decoding, and on the camera side opens up the world to smaller and less power hungry professional tools.
    Thank you Phil for this interesting discussion of how Redcode has changed with Komodo. I was wondering why the compression ratios were limited and the data rates were comparatively high. This answers that.

    DCT Redcode seems to reduce encode/decode compute requirements at the cost of higher file sizes (and fewer compression ratio options) as compared to DWT, which may be a key factor in making the Komodo possible (very small/low cost), and maybe will result in cheaper and/or more powerful DSMC3 brains.

    There are two issues that I currently see with changing to DCT, however. First, is the media cost. Helium 6K 24fps FF at 8:1 is 91 MB/s, whereas the Komodo at the same settings, but using MQ is 237 MB/s (Jarred said MQ is visually similar to 8:1 on DSMC2). That's 2.6x the data. If I were to shoot 1200 minutes of footage (as I plan in my next few projects) on Helium at 6K 8:1 I would need about 6.4TB of storage. If I did the same with Komodo, I would need almost 17TB. That's a significant cost in media and transfer times that will add up over time. Following Moore's Law, it will take more than 2 years for 17TB of storage to be the same cost as 6.4TB today.

    The second issue is that I don't see the lower compute benefit of decoding with Recode DCT while editing and coloring. Maybe it makes a big difference playing back within a Red brain, but there doesn't seem to be much benefit on a computer in my testing. I tested 6K 24fps FF at 8:1 from my Epic-W and your Komodo 6K FF clip (the one with the chip chart) on a Ryzen 3900X 12-core computer with 32GB RAM and an RTX 2070 8GB GPU. I used Resolve 16.2.4 Studio (latest version with Komodo support) set to GPU accelerate debayer and decompression. In the edit panel with a 4K timeline, my CPU only uses 5% on both clips. The GPU load is 40% for the Komodo clip and 50% for the Helium clip. Throwing on a couple of color correction nodes and some temporal NR in the color panel only adds about 5% more to the GPU load for both. Maybe there's something I'm missing, but I don't see a significant reduction in GPU load with Komodo footage. I would have expected it to be much more lightweight like other DCT codecs.
    Epic-W #376 (stolen)
    Reply With Quote  
     

  8. #48  
    Senior Member
    Join Date
    Dec 2014
    Location
    Sarajevo
    Posts
    964
    Quote Originally Posted by Todd G. Peterson View Post
    Thank you Phil for this interesting discussion of how Redcode has changed with Komodo. I was wondering why the compression ratios were limited and the data rates were comparatively high. This answers that.

    DCT Redcode seems to reduce encode/decode compute requirements at the cost of higher file sizes (and fewer compression ratio options) as compared to DWT, which may be a key factor in making the Komodo possible (very small/low cost), and maybe will result in cheaper and/or more powerful DSMC3 brains.

    There are two issues that I currently see with changing to DCT, however. First, is the media cost. Helium 6K 24fps FF at 8:1 is 91 MB/s, whereas the Komodo at the same settings, but using MQ is 237 MB/s (Jarred said MQ is visually similar to 8:1 on DSMC2). That's 2.6x the data. If I were to shoot 1200 minutes of footage (as I plan in my next few projects) on Helium at 6K 8:1 I would need about 6.4TB of storage. If I did the same with Komodo, I would need almost 17TB. That's a significant cost in media and transfer times that will add up over time. Following Moore's Law, it will take more than 2 years for 17TB of storage to be the same cost as 6.4TB today.

    The second issue is that I don't see the lower compute benefit of decoding with Recode DCT while editing and coloring. Maybe it makes a big difference playing back within a Red brain, but there doesn't seem to be much benefit on a computer in my testing. I tested 6K 24fps FF at 8:1 from my Epic-W and your Komodo 6K FF clip (the one with the chip chart) on a Ryzen 3900X 12-core computer with 32GB RAM and an RTX 2070 8GB GPU. I used Resolve 16.2.4 Studio (latest version with Komodo support) set to GPU accelerate debayer and decompression. In the edit panel with a 4K timeline, my CPU only uses 5% on both clips. The GPU load is 40% for the Komodo clip and 50% for the Helium clip. Throwing on a couple of color correction nodes and some temporal NR in the color panel only adds about 5% more to the GPU load for both. Maybe there's something I'm missing, but I don't see a significant reduction in GPU load with Komodo footage. I would have expected it to be much more lightweight like other DCT codecs.
    What's the cost difference between Komodo and Helium?
    Reply With Quote  
     

  9. #49  
    Senior Member Robert Hofmeyr's Avatar
    Join Date
    Sep 2009
    Location
    South Africa
    Posts
    759
    Quote Originally Posted by Todd G. Peterson View Post
    Thank you Phil for this interesting discussion of how Redcode has changed with Komodo. I was wondering why the compression ratios were limited and the data rates were comparatively high. This answers that.

    DCT Redcode seems to reduce encode/decode compute requirements at the cost of higher file sizes (and fewer compression ratio options) as compared to DWT, which may be a key factor in making the Komodo possible (very small/low cost), and maybe will result in cheaper and/or more powerful DSMC3 brains.

    There are two issues that I currently see with changing to DCT, however. First, is the media cost. Helium 6K 24fps FF at 8:1 is 91 MB/s, whereas the Komodo at the same settings, but using MQ is 237 MB/s (Jarred said MQ is visually similar to 8:1 on DSMC2). That's 2.6x the data. If I were to shoot 1200 minutes of footage (as I plan in my next few projects) on Helium at 6K 8:1 I would need about 6.4TB of storage. If I did the same with Komodo, I would need almost 17TB. That's a significant cost in media and transfer times that will add up over time. Following Moore's Law, it will take more than 2 years for 17TB of storage to be the same cost as 6.4TB today.

    The second issue is that I don't see the lower compute benefit of decoding with Recode DCT while editing and coloring. Maybe it makes a big difference playing back within a Red brain, but there doesn't seem to be much benefit on a computer in my testing. I tested 6K 24fps FF at 8:1 from my Epic-W and your Komodo 6K FF clip (the one with the chip chart) on a Ryzen 3900X 12-core computer with 32GB RAM and an RTX 2070 8GB GPU. I used Resolve 16.2.4 Studio (latest version with Komodo support) set to GPU accelerate debayer and decompression. In the edit panel with a 4K timeline, my CPU only uses 5% on both clips. The GPU load is 40% for the Komodo clip and 50% for the Helium clip. Throwing on a couple of color correction nodes and some temporal NR in the color panel only adds about 5% more to the GPU load for both. Maybe there's something I'm missing, but I don't see a significant reduction in GPU load with Komodo footage. I would have expected it to be much more lightweight like other DCT codecs.
    All good points Todd. Storage costs are getting cheaper fast, but for smaller budget work like mine the additional storage costs are definitely something that needs to be budgeted for. Jarred did mention that Komodo compression will become more optimised over time, so the data rate will decrease while the quality will remain the same - hopefully we will end up with less than double the data rate for similar quality.

    Regarding the computation requirements, I think maybe Red is still optimising the software. I tested Phil's short clip (which looks incredible BTW) in Redcine on my macbook pro and playback is quite a bit slower than my 6K 8:1 Dragon footage (realtime playback of the Dragon footage was possible at quarter res, whereas the Komodo footage could only do realtime at 1/8th res). I was playing both clips back off my internal SSD, so I doubt that read speeds were the bottleneck.

    Pretty sure everything is still going to change, so my plan is to wait a bit and see where things end up, both in terms of storage and speed.
    Reply With Quote  
     

  10. #50  
    Moderator Phil Holland's Avatar
    Join Date
    Apr 2007
    Location
    Los Angeles
    Posts
    11,783
    Quote Originally Posted by Todd G. Peterson View Post
    interesting discussion
    Just tackling a few points mentioned while I'm up late. First and foremost, it's day 1 for this new style of transform to be implemented. Real hardware optimization and other things are coming. Resolve shouldn't be considered the test bed as they need to tap into developed SDK that's been fully tested and deployed. Same for any 3rd party software really. There will be a time that's more useful to test, but right now it's early days with REDCODE RAW w/ DCT. More or less you won't see the main benefit right this second, but tomorrow when things get squared up, you'll be doing crazy things.

    I don't use AMD processors for a host of reasons, but the first thing you should test is disable all GPU acceleration in REDCINE-X Pro and try playing back both 6K 5:1 and 6K HQ at full debayer just off the CPU. That should tell you a bit about the potential of what's coming.

    In reference to your filming Helium at 6K 8:1, bare in mind you are filming at 8:1 which is already a higher compression ratio than what's achievable at the higher data rates your camera offers. You *can* film 6K at 3:1, which would be more in the HQ range. The net effect of what you hoping to achieve is whatever REDCODE RAW LQ data rates end up at (LQ is currently not implemented in Komodo, but it's coming). Knowing what I know about DCT, if I had to guess you'd be looking somewhere between the approximate data rates of 6:1-9:1 depending on RED's target and deployment. I strongly don't think things will evolve to a lower data footprint to 12:1 today, but that's just a hunch. But yeah, for a more apples to apples sort of thing, you'll be doing REDCODE RAW LQ to get to the footprint you're after.

    The power behind DSMC2 at the developed hardware, ASIC, and all that is you do have this crazy ability to do things like higher REDCODE RAW compression levels, but at the same time I've been an advocate of really not reaching beyond 12:1. I'm not just making that recommendation because I picked a number out of the sky. I'm making that suggestion after rigorous QC as well as projected and high resolution finish image quality tests.


    Modern codecs are getting absolutely fascinating as well as the wants, needs, and desires of the people who purchase these cameras. Somewhere in between Uncompressed RAW, Compressed RAW, and raw that isn't RAW, various methods to use non-linear to linear methods, combined with people who only look at the data footprint and don't think or care about much else will be the answer. The questions will be and always will be is RAW important to you? Is higher bit depth (16, 15, 14, 12, 10) important to you? Do you have issues with using chroma subsampled acquisition over RAW? If not, then do you want 4:4:4, 4:2:2, and *gasp* 4:2:0. Do you want to capture in 10-bit H.265 LOG over say 16-bit REDCODE RAW? This chaotic minutiae extends to the features of the sensor designs, camera features, how things are marketed, etc.

    The concern I've had for a bit here is the consumer and entry level market want "RAW", but don't even know what RAW is anymore nor the underlying reasons for all of this. Then the extension from there is they want LOG, which is thankfully a given at this point in however people choose to get there. But a core audience of mass market image makers has drifted into the zone of more of the video mindset, which happened before, we're here again. We just want to roll more footage and buy less tape. In the filmmaking world we didn't exactly have the ability to do extended recording on a 1000' roll of film, so it was a pretty alien concept outside of saving cash on film by shooting to modified 3-perf and even 2-perf movements on what was previously 4-perf S35.

    As we reach further towards higher image quality, though compression schemes and things can evolve, there is a point where we're just going to be dealing with chunkier data rates. Nobody thinks about capturing beyond 16-bit for instance much, but there will be a time when that's possible, though I'm certain there will be amazing chatter about how we don't need that if we are only making images for 10 or 12 bit panels. I've fortunately "ascended" a bit beyond those style of conversations over the years. The market is too dynamic to be concerned about what the needs of higher end filmmakers are versus entry level shooters, but we just need to be sort of aware of both far sides of the spectrum. Now there's cross talk and overlap confusing things which will have a negative impact on some parts of the industry and a positive impact mainly on the lowest priced tools available. Which puts into question what higher end tools will look like and be priced like in the future.

    Technology is advancing at a rate higher than market adoption at the moment. These next few years are going to be weird in many ways. The faster paced mobile market is sort of the pin point example of how crazy things are. Previously we'd get a processor per generation, then for a bit it was 2 releases, now there's 6. Which is also putting what was 18 month product timelines more squarely into the 3-6 month range. Same for mobile image sensors, with 4 or the 6 latest technologies (that I'm even aware of, just looked up a couple of new ones which can be dev'd, so make that 8) not being available in any device yet.

    The only fortunate thing for professional filmmakers is we have a very established global 2K/1080p HD market with 4K/UHD 2160p being the highest growth trend and the emergence of UHD 8K (still not much chatter about DCI 8K, but it will be there at the end of the day) becoming a standard. So in terms of that worry about what you are delivering to, there's your answer on what your "needs" are from your tools. Long distance future, yes there are plans for 16K displays, but consumers will be looking at that transitional gap harder than anything as 90-120" televisions aren't common place in the home. Before we get there we have some new panel tech and likely round three of HDR being deployed. The journey to 4K and 8K were always somewhat obvious still in a community that doesn't fully agree with that, but the less obvious things for me are the world beyond 8K as a delivery medium. Mainly because the general ecosystems aren't even being explored yet outside of 360 video (yes, YouTube supports 16K and has for a very long while).

    I diverged a bit off topic, but that's what's going on industry-wise. Professionals still need professional tools. We're in a moderate amount of chaos with some of the lower end stuff being positioned against the higher end gear, so expect those conversations to only get worse with producers who have to deal with that already. But there are far fewer higher end gigs than lower end gigs, which is really what happened in the last 10-15 years. Now there's tools to cater to the entire market as the demand become rather demanding.

    Pros for filmmakers, really capable tools at every price point. The IQ/performance/feature gap is in some ways shrinking between higher end tools as well as oddly expanding in some ways.
    Phil Holland - Cinematographer - Los Angeles
    ________________________________
    phfx.com IMDB
    PHFX | tools

    2X RED Monstro 8K VV Bodies and a lot of things to use with them.

    Data Sheets and Notes:
    Red Weapon/DSMC2
    Red Dragon
    Reply With Quote  
     

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts