Real-Time playback 120fps RED VIDEO on 120Hz monitor? (120fps web video now possible)

[Mark Rejhon]

Hello,

What is the most loss-free way of converting a 120fps slow-motion RED video, to a 120fps real-time playback?
i.e. What would the best RED workflow be, to avoid loss of video quality?

Web browsers are already able to display videos at 120fps when running on 120Hz monitors:
http://www.blurbusters.com/faq/real-time-120fps-video
In general, any computer capable of playing 4K video, is usually capable of 1080p@120Hz playback in real time, even via web browser.

Thanks,
Mark Rejhon

 

[MichaelP]

I believe it's mentioned in the body of the webpage. The issue will be post of audio and how that gets back into the file.

Tip for web developers: To avoid re-encoding video using ffmpeg, it is also possible to set “.defaultPlaybackRate = 4.0″ in JavaScript for HTML5 <video> element, in order to play back a slow-motion 120fps video in real-time. This makes “Slo-Mo Play” and “Real-Time Play” buttons possible!

 

[Mark Rejhon]

I believe it's mentioned in the body of the webpage. The issue will be post of audio and how that gets back into the file.
Tip for web developers: To avoid re-encoding video using ffmpeg, it is also possible to set “.defaultPlaybackRate = 4.0″ in JavaScript for HTML5 <video> element, in order to play back a slow-motion 120fps video in real-time. This makes “Slo-Mo Play” and “Real-Time Play” buttons possible!

I'm just curious about the RED workflow that would allow export of 120fps footage to a 120fps H.264 file with minimal loss of quality. I imagine you wouldn't want to use re-encode using ffmpeg! The same MP4 files play at a full 120fps in Windows Media Player on Windows 8 systems running on 120Hz computer monitors.

This would allow downloads of 120fps HFR movie trailers (for those who own 120Hz monitors).

 

[Mark Rejhon]

I'm just curious about the RED workflow that would allow export of 120fps footage to a 120fps H.264 file with minimal loss of quality. I imagine you wouldn't want to use re-encode using ffmpeg! The same MP4 files play at a full 120fps in Windows Media Player on Windows 8 systems running on 120Hz computer monitors.

This would allow downloads of 120fps HFR movie trailers (for those who own 120Hz monitors).

No one knows? What kind of workflow (with minimal degradation) works with translating 120fps RED videos, to real-time 120fps H.264 files compatible with Windows Media Player / HTML5 web browsers (with no speedup scripts) running on 120Hz computer monitors??? I'm surprised. Windows Media Player already supports 120fps native video files under Windows 8, and the same file plays full frame rate in HTML5 <VIDEO> elements too as well...

With the talk about HFR movies, and the web as a legitimate 120fps distribution/demo medium, it's going to be important to support workflows that does this, even at this early stage -- for experimentation.

Any comments from RED employees? I'm the owner of Blur Busters, and I got an inquiry from a RED user through blurbusters email.

 

[Craig Parkes]

No one knows of a workflow that works with converting 120fps RED videos, to real-time 120fps H.264 files compatible with Windows Media Player / HTML5 web browsers (with no speedup scripts) running on 120Hz computer monitors??? I'm surprised. Windows Media Player already supports 120fps native video files under Windows 8, and the same file plays full frame rate in HTML5 <VIDEO> elements too as well...

With the talk about HFR movies, and the web as a legitimate 120fps distribution/demo medium, it's going to be important to support workflows that does this, even at this early stage -- for experimentation.

Any comments from RED employees? I'm the owner of Blur Busters, and I got an inquiry from a RED user through blurbusters email.

Outside of Douglas Trumbull I think you'll find that there isn't a tonne of people even experimenting with 120fps playback. The financial costs compared to the benefits for most forms of content don't really stack up, both on the distribution side and the acquisition side. Because of this I doubt you'll find many NLE's racing to support 120fps playback, and if the common NLE's don't support it then it'll simply be seen as a bridge to far for most content creators.

Where I think you'll find the most interest in high frame rate playback will be in video game development and VR. In saying that the workflow shouldn't be that difficult

In terms of converting the video to those frame rates for experimentation, currently I would imagine a practical workflow would be to shoot 120fps on a 30 time base, bring all your clips into an NLE, drop them on a timeline and speed them up by 400%, sync audio to the sped up clips do your edit, export audio as a stand alone sound file. Then remove the speed up on all the clips, send the slow mo sequence to REDCINE-X export as an 1920x1080 image sequence, then mux the image sequence to h.264 high profile 5.1 at 120fps using a command line encoder.

Might as well be a 1920x1080 export, because h.264 at 5.1 only supports that resolution at 120fps frame rate. That would be the most lossless way to go given current tech (h.265 will offer better options).

Most complicated thing would be maintaining sound sync for narrative work, it'd be a little hit and miss I think getting the audio files to sync up in the NLE with the sped up clips, and then retain sync when muxing the final audio back into the h.264

Possibly a few gotchas there in terms of time base, but don't see any reason why the above methodology wouldn't work in theory.

ADDENDUM - One thing you WOULD need to check is what video cards actually support full hd 120fps playback.. While it's in the standard, I'd imagine a lot of h.264 hardware decoders probably aren't actually manufactured to manage that at the moment, so hardware limitations are likely to come into play. If you have the CPU power to decode full HD h.264 at 120fps this isn't as much of a concern, but again, not sure which playback software will actually be optimized to handle that efficiently at full HD.

It'd be a hell of a strain on most systems.

 

[Mark Rejhon]

ADDENDUM - One thing you WOULD need to check is what video cards actually support full hd 120fps playback.. While it's in the standard, I'd imagine a lot of h.264 hardware decoders probably aren't actually manufactured to manage that at the moment, so hardware limitations are likely to come into play. If you have the CPU power to decode full HD h.264 at 120fps

Generally, all nVidia and AMD cards that can play 4K 30fps are capable of playing 1080p 120fps.

Granted, that means spending at least $100+ on a graphics card, about $250 (ASUS VG248QE) on a 120Hz computer monitor, and $500 for a GoPro Hero 3 (120fps @ 1280x720), you can get into 120fps HFR for less than $1000 today. Why should RED workflows have less capabilities than consumer equipment such as this, since a $500 camera can do 120fps@720p already and 120fps@1080p is just around the horizon? The capabilities are coming faster than some people think. It is true it is niche, but 120fps is actually cheaper than expected as many HDTV's have an undocumented 120fps mode, and the computer graphics cards capable of 4K30 automatically can do 1080p@120 -- something now apparently easily had in a $125 nVidia graphics card (Geforce 650 in this particular case).

There is now a HTML5 way to automatically detect the refresh rate of a web visitor (via JavaScript requestAnimationFrame() counting technique, as that callback is now synchronized to VSYNC on most browsers) which works in Chrome 18+, Opera 15+ and the FireFox24+ beta. It is used by http://www.testufo.com (view in Chrome to see your monitor's refresh rate) ... and any graphics card outputting 120Hz could be automatically presented with the choice of downloading a 120fps HFR video, since most visitors running 120Hz would be almost certainly running a graphics card capable of playing back HFR video (90%+ certainity). This greatly simplifies content distrubution, as you now can even record statistics of which web visitors are running 120Hz computer monitors, and present extra options to them, etc.

It's actually apparently simpler than many industry insiders think, since in the next year, marketing 120fps HFR videos to the big-money-paying high end video game audience, is one way to create 120fps HFR buzz; and expand 120Hz monitor users to a wider audience too.

 

[Craig Parkes]

Generally, all nVidia and AMD cards that can play 4K 30fps are capable of playing 1080p 120fps.

Granted, that means spending at least $100+ on a graphics card, and at least $250 on a 120Hz computer monitor, and $500 for a GoPro Hero 3 (120fps @ 1280x720), you can get into 120fps HFR for less than $1000 today. The capabilities are coming faster than some people think. It is true it is niche, but 120fps is actually cheaper than expected as many HDTV's have an undocumented 120fps mode, and the computer graphics cards capable of 4K30 automatically can do 1080p@120 -- something now apparently easily had in a $125 nVidia graphics card (Geforce 650 in this particular case).

Being capable of doing it and ACTUALLY doing it are two different things from experience. As h.264 decoding a 120fps video for 120fps playback is really a marginal case at the moment for manufacturers, performance over the long term is something I would want to benchmark before making recommendations to third parties.

But that's just me being cautious. As I said, it's in the standard for 5.1 playback, so theoretically you are of course right.

From a practical stand point of day to day video production though, the costs are much more in processing 120fps video in an NLE/finishing suite compared to 1080@30fps. You are immediately looking at 4 times as much data and four times as much rendering,when production budgets outside Hollywood blockbusters are generally going down, not up. So while the gear is out there to do all this, the impetus to do it isn't so great that it'll be supported by mainstream workflows, but from an experimental point of view if you get your edit out to an image sequence then you should be able to then re-encode it out to whatever frame rate your playback device can handle.

You just won't be able to monitor it at that frame rate through production. To be fair, the Hobbit couldn't monitor 48fps frame rate 3D during editing in Avid either, (when finishing shots they could on the Mystika) so you'd be in good company at least.

 

[Mark Rejhon]

Makes sense.
For now, at least anybody can use external tools to easily convert a slow-mo 120fps to real-time HFR 120fps.
So a workflow exists, even though transcoding could be needed. Just hope that more software can do this directly; to bypass the transcoding need or do it in a more lossless manner.

It may be years, but Blur Busters is helping trailblaze the 120Hz web -- I actually corresponded with Mozilla (Bug #856427) to fix their 120Hz support in FireFox (fix now in 24+ beta) including some debugging tests I sent them. Could be a ripe audience in a few years, since I'm actually being quite loud in the browser world and w3c standardization, and now all 5 major browsers are mostly 120Hz-ready (with minor exceptions).

That said, some sites such as Blur Busters is making it easier for consumers to enter the 120Hz world (for gaming). This is a stealth wave of consumer upgrades into apparently HFR-capable computers. This potential marketing audience shouldn't be ignored -- especially since this is theoretically a good audience to market HFR movies to (raising the profile of HFR). Web ad detects 120Hz monitor, ad plays back 120fps HFR video; very targeted high-effectiveness HFR marketing.

 

[Craig Parkes]

If I were you, I'd focus all my attention on Oculus/VR support. That's where 120fps will be 'mandatory' for the user experience if it continues to catch on the rate it has been.

 

[MichaelP]

Outside of Douglas Trumbull I think you'll find that there isn't a tonne of people even experimenting with 120fps playback. The financial costs compared to the benefits for most forms of content don't really stack up, both on the distribution side and the acquisition side. Because of this I doubt you'll find many NLE's racing to support 120fps playback, and if the common NLE's don't support it then it'll simply be seen as a bridge to far for most content creators.

Doug Trumble's project is stereo 3D with each eye shooting 60fps and a patented process that aggregates the eyes into a 120fps experience. For creative editorial, 60fps (59.94) is already accomplished in a 720p project type. Some NLE's allow for 1080p/59.94 editing as well. The conform and stereo 3D VFX work as well as playback is being done in Eyeon Fusion and every shot is pretty much an effect. I consulted a bit on the post process for this project. We went back in time a bit to sync the 120fps from Eyeon using a 30fps LTC scratch audio track that a Pro Tools slaves to with the 5.1 soundtrack during the post process.

Michael

 

[Björn Benckert]

I'm just curious about the RED workflow that would allow export of 120fps footage to a 120fps H.264 file with minimal loss of quality. I imagine you wouldn't want to use re-encode using ffmpeg! The same MP4 files play at a full 120fps in Windows Media Player on Windows 8 systems running on 120Hz computer monitors.

This would allow downloads of 120fps HFR movie trailers (for those who own 120Hz monitors).

You understand that you would need to cut down on quality to 1/5 compared to 24p if you wanted the same download speed... Not so sure if so many people actually enjoy 120p so much so they would want the sacrifice in quality, I for sure like the texture of 24P much better than High framerates, I even think 30p is ugly, but thats maybe just me.

 

[Craig Parkes]

Doug Trumble's project is stereo 3D with each eye shooting 60fps and a patented process that aggregates the eyes into a 120fps experience. For creative editorial, 60fps (59.94) is already accomplished in a 720p project type. Some NLE's allow for 1080p/59.94 editing as well. The conform and stereo 3D VFX work as well as playback is being done in Eyeon Fusion and every shot is pretty much an effect. I consulted a bit on the post process for this project. We went back in time a bit to sync the 120fps from Eyeon using a 30fps LTC scratch audio track that a Pro Tools slaves to with the 5.1 soundtrack during the post process.

Michael

Thanks Michael, I should have pointed out a 720p proxy workflow rather than a 1080p one at thirty. I would imagine proper 1080p/59.94 editing is still pretty uncommon given the lack of a full supporting broadcast standard, so a 720p proxy workflow certainly makes the most sense.

Have you looked much into Oculus Rift VR? I imagine with the amount of experimenting going on in the gaming world with the dev kits and their push to get the screen refresh rate up to 120 as well as increasing the definition that it'll be the first environment where such work really finds a audience with consumers.

 

[Mark Rejhon]

If I were you, I'd focus all my attention on Oculus/VR support. That's where 120fps will be 'mandatory' for the user experience if it continues to catch on the rate it has been.

Already there in that territory, too.
- I had a small contract with Oculus VR, via Palmer Luckey.
- I regularly read Michael's blog (see Down The VR Rabbit Hole: Fixing Judder
- See my correspondence with Michael Abrash (search "rejhon") and Michael's compliment to me.
- Blur Busters started because of a tweet reply from John Carmack.

That said, the Blur Busters focus is extremely broad, focussing on almost everything 120Hz and upwards. Originally, major coverage was currently towards LightBoost strobe backlights, since that creates CRT motion clarity on LCD's, but has expanded since to almost everything 120Hz, including trailblazing the 120Hz web.

 

[Mark Rejhon]

You understand that you would need to cut down on quality to 1/5 compared to 24p if you wanted the same download speed...

That said, 120Hz monitor users are using computers that run graphics cards over 5-10x faster than the average mainstream graphics. Now that there's a web-browser-based method of detecting whether a web visitor is using a 120Hz monitor, the web is a viable potential 120fps distribution medium in the coming years. Internet speeds are also typically faster among these users, too... This is only a trickle now, but could change.

I for sure like the texture of 24P much better than High framerates, I even think 30p is ugly, but thats maybe just me.

I often like my movies in 24p too as well (I preferred Hobbitt in 24Hz than 48Hz)

...However, there are also use cases where I just love HFR. For example, a movie about racing cars I prefer to watch in HFR, while a story like Hobbitt I really do prefer 24p. Or a live Olympics broadcast. IMHO, if you're doing HFR, 48Hz is crap to my eyes -- do 120Hz instead. "Go big or go home". Mathematically, for sample-and-hold displays, you get a 80% reduction in motion blur on sample-hold 24fps->120fps, while you only get a 50% reduction in motion blur 24fps->48fps. Strobing the whole display (flicker) also reduces motion blur (e.g. CRT, plasma, LightBoost, old 48fps projectors) -- It's good for gaming and VR though (as Michael Abrash and John Carmack talks about strobed displays as a solution to motion blur). Strobing is good for blur reduction but create flicker eyestrain and introduce wagonwheel motion artifacts. That said, going to a HFR is another method of reducing motion blur without using flicker. So for this reason, flicker-free 120fps HFR is very good.

BTW, some good animations (works best in Chrome Browser on sample-and-hold displays such as your common LCD monitor):

Animation of motion blur at different framerates: www.testufo.com/#test=framerates
Animation of motion blur reduction via flicker/strobing: www.testufo.com/#test=blackframes
Animation of motion blur caused by eye-tracking: www.testufo.com/#test=eyetracking

According to motion blur mathematics, 120fps HFR isn't actually the final frontier yet if you wanted to completely control motion blur (fully source-controlled blur, rather than display-injected blur, or forced eye-tracking-based blur), while avoiding strobing -- since there's still opportunity for eye tracking-based motion blur (.For 120fps HFR, there is still about 8 pixels of motion blur for 1000 pixels/sec motion (1/120sec = 8.33ms). This is because on sample-and-hold displays, there is always 1 pixel of motion blur for every 1ms of static frame duration during 1000 pixels per second panning motion. (scientific papers cover the sample-and-hold effect as well). Eye-tracking-based motion blur ( www.testufo.com/#test=eyetracking ) is dictated by the length of a visible refresh, shortening the visibility of the frames is either by flicker or by raising the framerate. From the math, 24fps->48fps only cuts display-enforced eye-tracking-based motion blur by half (i.e. panning shots have half the blur). If you want to completely avoid flicker while also completely avoiding wagonwheel artifacts, then Michael Abrash's desire for 1000fps@1000Hz is quite prescient. That is necessary in order to create a flicker-free CRT-quality (perfectly sharp panning motions without needing to use flicker at any frequency). That probably won't happen within our lifetimes (unless done via interpolation), but it certainly illustrates that even 120fps isn't the "final frontier" when aiming for a "holodeck" experience. Discrete-framerate displays even at >500fps still have human-noticeable artifacts, though there are definitely gradual diminishing returns as you go higher.

The bottom line is that 48fps HFR IMHO, is just a small improvement and isn't yet "worth it" to my eyes.
Give me 24fps (for the relaxing film experience) or 120fps+ (for the hyperreal experience).

 

[MichaelP]

I think content is a big factor in HFR as referred to by Mark R. I would be much more interested in seeing "Saving Private Ryan" done as HFR 3D than the Hobbit. I think the realness, and the "in the picture" benefits of HFR would have better served the Private Ryan type story.

Michael

 

[Mark Rejhon]

I think content is a big factor in HFR as referred to by Mark R. I would be much more interested in seeing "Saving Private Ryan" done as HFR 3D than the Hobbit. I think the realness, and the "in the picture" benefits of HFR would have better served the Private Ryan type story.
Michael

Agreed -- the story is a big part of HFR. I really preferred Hobbit in 24fps.

I like HFR, but I'm not satisfied with 48fps HFR (only a 50% improvement in motion blur) -- it should be strobed 60fps HFR or non-strobed 120fps HFR -- and applied to the correct story. Personally, 120fps non-strobed is probably easier on the eyes, since it eliminates flicker (although 120Hz brief-strobed yields really beautiful pans that are as perfectly clear as stationary images; CRT-like motion clarity zero motion blur at a more acceptable 120Hz flicker). The pans on a LightBoost strobe-backlight 120Hz monitor is 12x clearer than on a common 60Hz LCD monitor (e.g. where there was 12 pixels of motion blur, there is only 1 pixel of motion blur for same speed motion).

(Strobing == method of reducing motion blur == like http://www.testufo.com/#test=blackframes viewed in Chrome browser)

 

[Mark Rejhon]

HyperMatrix (another leading 120Hz gaming expert) of 120hz.net, in cooperation with me, just successfully created an AVI recording of a 1080p@120fps recording of a Battlefield 3 videogame, at an approximately 80 Mbit/second bitrate.

He has an SSD RAID and a Titan SLI setup (>$2000 of GPU) which allowed him to do 120 screen captures per second uncompressed (750Mbytes/second recording) without slowing down the GPU's below 120fps@120Hz. We've successfully created an AVI of it, and I've downloaded it already to my computer -- it plays flawlessly on computers running an nVidia Geforce 680, 690, 700-series and Titan's, as expected. We are currently in the process of testing, and testing other containers (MP4 instead of AVI), with some minor difficulty with the tools (due to the newness of doing HFR 120fps), but I have here, on my disk, sitting a draft 1 gigabyte, 1min40sec video file, HFR at 120 frames per second, H.264 in AVI container, that plays successfully at 1080p@120fps on multiple enthusiast gaming PC's.

We'll probably create a 20 second reduced-rate (~30 Mbit/sec) for public release, that runs on slower gaming PC's (those with $100 graphics cards).
That would be only about 80-100 megabytes or a bit less.

Eventually I'd like to ask some RED user to create a short demoreel (15-20sec) 1080p@120fps demonstration video from a RED camera for web publishing. Blur Busters would be happy to give this publicity, as we have the 120Hz monitor target audience. You users have a top quality 120Hz camera, I have the 120Hz audience. I've detected more than 5,000 visitors running 120Hz monitors in the last 3 weeks. Since the Blur Busters focus is primarly on high-end gaming, and heavily promotes 120Hz usage, I have a disproportionately high number of 120Hz visitors (among the world's highest penetration of 120Hz visitors, probably). The TestUFO website (running www.testufo.com in Chrome/FF24+/Opera15) is the world's first website that is able to detect the refresh rates of visitors -- via a new Javascript requestAnimationFrame() counting technique -- thanks to it synchronized to refresh because of a new W3C draft published just a few months ago -- so it's now possible to count 120Hz visitors, and thus detect a potential 120Hz audience.

 

[Mark Rejhon]

Ok, it's out now!

The world's first publicized 120fps video gameplay recording. Full screen 1920x1080p/120fps, for 120Hz computer monitor users. This is an earlybird of the promise of future consumer HFR 120fps, which HFR directors / RED CINEMA may wish to pay attention to, since this was created only with existing enthuisiast gaming equipment and free software tools. The computer that recorded real-time video simultaneously during real-time full framerate 120fps@120Hz gameplay, is cheaper than the cheapest RED camera. The workflows of the future must become ready for HFR 120fps. Demand your video software vendors to support HFR 120fps, even years in advance.

Battlefield3 video game, 120 frames per second at 120Hz on a Geforce Titan.
H.264 at 1080p @ 120fps, running directly in web browser HTML5 <VIDEO> element.
The bitrates have intentionally been slimmed down for consumer use. Remarkably, 25 Mbps looks near-indistinguishable from the original actual gameplay: It doesn't look like a recording of the gameplay.

Gameplay Recording HFR 1080p @ 120fps

25 Mbps version -- Play 1080p@120fps Inside Web Browser -- Download 1080p@120fps File: BF.MP4 (100 MB)
8 Mbps version -- Play 1080p@120fps Inside Web Browser -- Download 1080p@120fps File: BF1.MP4 (35 MB)

Pre-reqisite: Preferably a 600-series Geforce/7000-series Radeon or similar, running on a 120Hz computer monitor (as little as $250).

The lower bitrate version may actually play on older GPU's, and actually plays on 60Hz screens too (dropping every other frame). I've heard of a report that a 3-year old GPU was successfully able to play this file, so feel free to give it a try if you can obtain a 120Hz monitor. Note that the 50" SEIKI 4K HDTV, while it does only 30Hz at 4K, it acually natively supports 120Hz when running at 1920x1080p. So you can use your 50" SEIKI HDTV as the 120Hz computer monitor to play these files at full frame rate (by installing 120Hz with ToastyX Custom Resolution Utility)!

-- SUCCESS: Media Player Classic Home Cinema; perfect playback
-- SUCCESS: Internet Explorer 10; perfect playback
-- SUCCESS: FireFox 24+ Beta; usually smooth playback
-- SUCCESS: Windows Media Player; usually smooth playback
-- SUCCESS: (partial) Google Chrome; sometimes pauses, sometimes perfect. Wait for buffering.
-- UNTESTED: Safari
-- FAIL: Opera
-- FAIL: VLC Media Player


Got some good feedback already:

I just watched it, and it was amazing, I've never seen such fluid motion in video form on a browser!

Looks amazing, excellent fluidity and smoothness; good job md! :thumb:

Sweet Jeebus, this looks incredible.