Filming for VR?

[BrendanLeahy]

Is everything proprietary and closed-off? I can't seem to find any articles or instructionals on how to best film and prep for the oncoming wave of VR. I'm really keen on making films in this field, but it seems like I keep hitting brick walls.

 

[Phil Holland]

There aren't many one answer answers on this front. At least not at the moment. Even stuff I've done for one particular device and even project has many "right answers".

 

[Michael Lindsay]

Don't worry nobody had 'cracked it' ...

S3d 360 VR makes normal s3d seem very easy..

Nearly every approach needs heavish-heavy post...

 

[BrendanLeahy]

Don't worry nobody had 'cracked it' ...

S3d 360 VR makes normal s3d seem very easy..

Nearly every approach needs heavish-heavy post...

Oh so it's not so much what you do with rilming, the VR is mostly proprietary post production work of stitching things together?

 

[Gavin Greenwalt]

The problem isn't stitching things together. The problem is stereo synthesis. If you want to just create a 360 sphere that's flat "VR" is easy to delivery today. Just solve for the lens distortion, project onto a sphere and extract your UVs. Then you render a camera inside the sphere based on your head position. Now add stereo to the mix and things get hard. The way to capture stereo video is to shoot it twice with separation in between. There isn't actually 3D depth to the image so how do you "stitch" together stereo pairs into a 3D sphere... you can't. Imagine you shoot facing north with two cameras. And you shoot East with two cameras. Now let's say the VR person is looking to the North east. You can't use the right eye and make it the left eye of the North camera and the left eye of the east camera and use it as your north eye, those two cameras need to stitch together in the middle and they're pointed at 90 degree angles from one another. Eyes are generally looking in parallel. It's similar to what happens in a 3D movie if you rotate your head. Even with ciricular polarization the stereo effect breaks down because your eyes are no longer aligned with the two stereo images. The effect is broken.

So what you have to do is take your spherical stereo pairs and estimate depth from them using the parallax between the pixels. From there you can then generate a 3D cloud of depth and re-render this 3D mesh estimating the geometry of the scene as if it were actually 3D. Now when you roll your head a new set of stereo pairs are generated in real-time.

But it's not computationally easy, and it's not without depth artifacts. On the computation side you have to remember that you're rendering geometry in real-time. Ideally the depth should be at around the same resolution as the images. So you need about a 4k depth map. So you're displacing 4k x 2k = 8,000,000 vertices per second. That's a lot of geometric distortion to generate. You're also then having to render this at 60fps.

And even this solution (which is the best we have) is glitchy. You'll see stretch marks etc and occlusion won't be filled in. That's generally what happens with post converted stereo. Except now instead of post converting for stereo from two eyes you have to create from essentially the possibility of 4 eye stereo convertion (in case someone tilts their head). So now you have multiple displaced spheres where you're patching possible occlusions depending on the their head perspective and you're more likely rendering not just one 8m vert mesh but probably 3-4 layers of meshes. It's a lot of rendering and it's a lot of hand work to post convert.

And you have to shoot in 360 degrees. So no grip equipment, everything has to be practical. You have to paint out the tripod. Nobody can be within sight of the camera. This is why animated films make the most sense for VR. Just like animated films have had the easiest time transitioning to stereo.

 

[BrendanLeahy]

The problem isn't stitching things together. The problem is stereo synthesis. If you want to just create a 360 sphere that's flat "VR" is easy to delivery today. Just solve for the lens distortion, project onto a sphere and extract your UVs. Then you render a camera inside the sphere based on your head position. Now add stereo to the mix and things get hard. The way to capture stereo video is to shoot it twice with separation in between. There isn't actually 3D depth to the image so how do you "stitch" together stereo pairs into a 3D sphere... you can't. Imagine you shoot facing north with two cameras. And you shoot East with two cameras. Now let's say the VR person is looking to the North east. You can't use the right eye and make it the left eye of the North camera and the left eye of the east camera and use it as your north eye, those two cameras need to stitch together in the middle and they're pointed at 90 degree angles from one another. Eyes are generally looking in parallel. It's similar to what happens in a 3D movie if you rotate your head. Even with ciricular polarization the stereo effect breaks down because your eyes are no longer aligned with the two stereo images. The effect is broken.

So what you have to do is take your spherical stereo pairs and estimate depth from them using the parallax between the pixels. From there you can then generate a 3D cloud of depth and re-render this 3D mesh estimating the geometry of the scene as if it were actually 3D. Now when you roll your head a new set of stereo pairs are generated in real-time.

But it's not computationally easy, and it's not without depth artifacts. On the computation side you have to remember that you're rendering geometry in real-time. Ideally the depth should be at around the same resolution as the images. So you need about a 4k depth map. So you're displacing 4k x 2k = 8,000,000 vertices per second. That's a lot of geometric distortion to generate. You're also then having to render this at 60fps.

And even this solution (which is the best we have) is glitchy. You'll see stretch marks etc and occlusion won't be filled in. That's generally what happens with post converted stereo. Except now instead of post converting for stereo from two eyes you have to create from essentially the possibility of 4 eye stereo convertion (in case someone tilts their head). So now you have multiple displaced spheres where you're patching possible occlusions depending on the their head perspective and you're more likely rendering not just one 8m vert mesh but probably 3-4 layers of meshes. It's a lot of rendering and it's a lot of hand work to post convert.

And you have to shoot in 360 degrees. So no grip equipment, everything has to be practical. You have to paint out the tripod. Nobody can be within sight of the camera. This is why animated films make the most sense for VR. Just like animated films have had the easiest time transitioning to stereo.

Gavin - thanks so much for this in-depth info! I was wondering if you had more specifically on the "flat VR" process, and how to shoot for that? Stereo sounds quite tricky, so I feel it might be best to try starting out small and going from there for my work!

 

[Michael Lindsay]

youy can have fun with a Red camera and this kind of thing

pano pro

http://www.pano-pro.com/index.php?route=product/product&path=61&product_id=50

0-360

https://www.0-360.com

 

[Gavin Greenwalt]

Gavin - thanks so much for this in-depth info! I was wondering if you had more specifically on the "flat VR" process, and how to shoot for that? Stereo sounds quite tricky, so I feel it might be best to try starting out small and going from there for my work!

The basics are to cover every angle and then Stitch the panoramas together then export a UV map for each of your cameras along with the blend alphas. Generally lots of warping and projection is involved.

 

[BrendanLeahy]

I wanted to bump this again. I'd very much like to find out if anyone has affordable rigs for RED cameras now, or if there are any all-in-one price-friendly VR cameras (please no links to gopro rigs).

 

[Meryem Ersoz]

affordable rigs for RED camera VR? surely you jest. Any VR rig for RED is going to be very pricey and probably very beta. The go pro rigs are the affordable rigs, period.

Have a look at the Nokia Ozo. It's probably the closest thing to affordable pro VR, though it's not quite available either, I don't think. Even that is 60K.

There was a lot of chatter at NAB about VR but only a few have cracked the post-production problems. Until the few become the many, none of it will be either cheap or easy....

 

[Sam Sheppard]

I've done 1 of these professionally and 2 for my own education.

A good way to get a cheeky free 'education' is to book a demo with one of the post-production vendors who is focusing on tools in this area. It's impossible to demo the tools if the client doesn't know how it all works so, in order to make the tools look as good as they can, the demo operator needs to explain the VR concepts that they're demonstrating.

Both The Foundary/Nuke and SGO/Mistika released VR press releases featuring VR tools for NAB 2016. I didn't go so I didn't see them. If you contact your local office both will probably be more than happy to arrange something for you. You can probably learn everything that you need to know in an hour :ihih:

 

[PatrickFaith]

...

Both The Foundary/Nuke and SGO/Mistika released VR press releases featuring VR tools for NAB 2016. I didn't go so I didn't see them. If you contact your local office both will probably be more than happy to arrange something for you. You can probably learn everything that you need to know in an hour :ihih:

I'm running the new Nuke CaraVR under nuke 9 right now. I btw really like nuke 10.
From todays shoot (using the Nodal approach Phil explains but with a red):

 

[Emir Aksay]

Facebook has developed a 3d camera sytem and software that they will release open-source. I've done some experiments with a diy rig similar to the Odssey rig. We got really nice and fast results but with the facebook announcement and too much work on hand I took a break on this thing..

I can post test images if anyones interested..

 

[Elsie N]

Facebook has developed a 3d camera sytem and software that they will release open-source. I've done some experiments with a diy rig similar to the Odssey rig. We got really nice and fast results but with the facebook announcement and too much work on hand I took a break on this thing..

I can post test images if anyones interested..

The Facebook camera system looks like a good tool for this kind of work. I hope to do some of this myself... but not to the point where it is my main work. And even then it will require a special kind of project to merit this way of shooting.

This has a place in how we tell a certain kind of story.

 

[BrendanLeahy]

Facebook has developed a 3d camera sytem and software that they will release open-source. I've done some experiments with a diy rig similar to the Odssey rig. We got really nice and fast results but with the facebook announcement and too much work on hand I took a break on this thing..

I can post test images if anyones interested..

Yes please! It'd be great to hear any of the challenges you ran into as well.

 

[Emir Aksay]

Yes please! It'd be great to hear any of the challenges you ran into as well.

This test was to make a video to see if it worked so its just a few frames which I looped to see the 3d effect..

It was a pain uploading the file to youtube
https://youtu.be/GZ1COA8vCHI

Well as you can see its not perfect but, my main goal was to get 3d without using 2 cameras for each eye..

 

[Elsie N]

After reading this, I'm not so horny for helping further destroying civilization as we once knew it. :-(

 

[John Meese]

I just did a shoot with the Nokia Ozo with Abel Cine in NYC. They brought a tech - Mike - he was awesome.
Very very easy to shoot.

You just set your white balance and "exposure" - which is basically the noise floor a la shooting on a phantom, and you go.
Then you can quickly see a stitch of it with an oculus rift.

Was super super easy to work with.

Image quality is pretty good - I mean, it's going on the rift, so I couldn't 100% see how good it was over a go-pro rig, but it gives you more sterioscopic viewpoints than the go pro - more overlap of the lenses.

Also no go pro vr issues - it's all synched up and ready to shoot. Plug and play.

 

[jussi rovanpera]

I just did a shoot with the Nokia Ozo with Abel Cine in NYC. They brought a tech - Mike - he was awesome.
Very very easy to shoot.

You just set your white balance and "exposure" - which is basically the noise floor a la shooting on a phantom, and you go.
Then you can quickly see a stitch of it with an oculus rift.

Was super super easy to work with.

Image quality is pretty good - I mean, it's going on the rift, so I couldn't 100% see how good it was over a go-pro rig, but it gives you more sterioscopic viewpoints than the go pro - more overlap of the lenses.

Also no go pro vr issues - it's all synched up and ready to shoot. Plug and play.

Can the OZO do 3D easily, or just flat VR?

 

[David Peterson]

I wanted to bump this again. I'd very much like to find out if anyone has affordable rigs for RED cameras now, or if there are any all-in-one price-friendly VR cameras (please no links to gopro rigs).

"Affordable"?? Ha! You need at a ***minimum**** 6 cameras to properly cover full 360 degrees (more cameras is better however!).

And even buying 6x R1 MX (the cheapest of the REDs, but a ***very*** bad idea for this! As their body is too long) would become a very price proposition!

And as for all those "all-in-one VR solutions", they fall into one of two categories (or fit into both at once!):
a) no better image quality than a GoPro rig
b) waaaaay more expensive, not price friendly at all, unlike a GoPro rig (which even then is hardly dirt cheap... you could buy a Sony FS5 for instance easily with what the hardware/software costs are for a basic 360VR GoPro rig costs)

A GoPro rig is still a smart choice for getting into 360VR, it is the minimal level needed for basic professional work (and in many market regions... it is also the **best** on offer as well! I think it is the best possible being done in my country New Zealand at the moment for instance, I don't see or hear of anybody usings DSLRs let alone REDs in a 360 configuration in NZ?).

The next leap up is just to use **MORE** GoPros! As 6 GoPros is nice, but sixteen would be even better! Is what Google Jump uses:
https://www.google.com/get/cardboard/jump/

The next significant leap up would be to use multiple Blackmagic Design Micro Cinema Cameras (with a focal reducer and Rokinon 10mm on each BMMCC), so you can get 60fps ProRes HQ quality out of this (or even shoot in raw if you're mad enough, or the client budget allows for it).

The next big leap up after this would be to use 6x (or more!) Kinefinity Terra 5K cameras once they come out. As high resolution 5K, high frame rate options too, plus global shutter sensor, plus a very small form factor, equals a very attractive starting point for a 360VR rig!

And then you'd have a RED Weapon or Arri Alexa Mini set up right here at the top, easily available and affordable to all those newly minted billionaires. For the rest of us.... not so much, as even the rental cost alone would be staggering! Let alone buying them.

Currently I'm personally using a 6x GoPro rig, my plan is to basically follow this path I just described above. First, I'm going to throw even more cameras at it! (at least 12, so that I can do 3D 360VR at a minimum) Then if budgets and clients allow, and this 360VR direction is succesful for me like I hope it will be, I'll get 6x BMMCC (especially if they happen to go on a half price sale! Like the BMPCC did. Hope so, pretty please!). Then if I get very lucky and this exceeds all my reasonable expectations, in a year or two down the track I'll get 6x Kinefinity Terra 5K (however this is very unlikely, practically speaking. But one can dream!).