Iphone as camera tracking device?

[Christoffer Glans]

Is it possible to use the iPhones accelerometer and gyro to collect data for camera tracking in post? How accurate is it?
If we put an Iphone on an Epic and had some way of recording camera tracking data on it, then pull that data into let's say After Effects, wouldn't that work for stabilizing and advanced camera tracking for VFX?

 

[Kaspar Hornikel]

It defindetly is possible, Shot Assistant improve their App in this kinda direction... https://www.facebook.com/ShotAssistant?fref=ts
https://www.facebook.com/ruben.sluijter/videos/10153499701927152/?fref=nf
But i dont know how hard it is to effectively use the Data in VFX.

 

[Maik Müller]

Is it possible to use the iPhones accelerometer and gyro to collect data for camera tracking in post? How accurate is it?
If we put an Iphone on an Epic and had some way of recording camera tracking data on it, then pull that data into let's say After Effects, wouldn't that work for stabilizing and advanced camera tracking for VFX?

Interesting idea. I can see two problems 1.) data-resolution / quality 2.) getting the data synchronized to the footage.
The first one depends on the quality and amount of the sensors and how accessible they are through the SDKs/APIs. I guess there is something on the Android-Market thats pretty good and beats the iPhone. The second problem could be solved through a simple audio-connection via headphone/mic-in.

BTW, i could also imagine using the cameras of a FirePhone to get more data.

That being said, someone is working on a promising solution. SteadXP is developing a specialized device and software:
http://www.reduser.net/forum/showthread.php?121384-post-stabilisation-using-gyro-data-SteadXP
http://steadxp.com/

P.S.: I still hope we will one day see a stabilizer that uses REDs own gyro-data.

 

[Christoffer Glans]

Interesting idea. I can see two problems 1.) data-resolution / quality 2.) getting the data synchronized to the footage.
The first one depends on the quality and amount of the sensors and how accessible they are through the SDKs/APIs. I guess there is something on the Android-Market thats pretty good and beats the iPhone. The second problem could be solved through a simple audio-connection via headphone/mic-in.

BTW, i could also imagine using the cameras of a FirePhone to get more data.

That being said, someone is working on a promising solution. SteadXP is developing a specialized device and software:
http://www.reduser.net/forum/showthread.php?121384-post-stabilisation-using-gyro-data-SteadXP
http://steadxp.com/

P.S.: I still hope we will one day see a stabilizer that uses REDs own gyro-data.

Yeah, the SteadXP inspired me to this idea. I think that the main thing, as you said is the issue with data-resolution. If it doesn't measure coordinates at least 24-25 times a second it won't match up with realtime footage. I downloaded an app to record the accelerometers data and export it as a .CSV file, but I don't know how to convert it to an actual After Effects 3D camera. That's the only step I need in order to try if it works or not.

 

[Dr. Sassi]

Just brainstorming here! I have no idea how to code this.

Wouldn't it be possible to write an app that turns the iPhone motion data into sound, and have that fed into the audio channel of the RED?
There will be a shift about a few frames, but that is easily to fix, if the delay can be a stable value. With two channels of sound and 20-20Khz, there is plenty of room for information, e.g., at least 4096 steps per "axis".

As long as the on set sound is taken by an external sound recorder, the two channels are free and the motion data is not all over the place.

(In 3D we often use RGB data to store XYZ, so why not use sound? Left channel XYZ, right channel HPB))

Perhaps one day RED's gyro will have a data channel as well, then it could be synced.

====

(An idea that I had send in last year to RED was, to use e.g., the rotation data from the gyro to connect it with the focus setting. Turning the camera, changes "A to B" focus settings.)

 

[Maik Müller]

Yeah, the SteadXP inspired me to this idea. I think that the main thing, as you said is the issue with data-resolution. If it doesn't measure coordinates at least 24-25 times a second it won't match up with realtime footage.

Yep, one set of coordinates per frame is the minimum. More is better and shutter-speed is another variable here. The biggest deal remains the software-part, because it's not just about shifting around the image, thats only the starting point. Motion-Blur is a problem for example.

I downloaded an app to record the accelerometers data and export it as a .CSV file, but I don't know how to convert it to an actual After Effects 3D camera. That's the only step I need in order to try if it works or not.

Interesting starting point, how much data-sets per second do you get?

Reminds me... When google announced Project Tango, i immediately thought about using it for camera-tracking AND focus-pulling.
https://www.google.com/atap/project-tango/

 

[Maik Müller]

Just brainstorming here! I have no idea how to code this.

Wouldn't it be possible to write an app that turns the iPhone motion data into sound, and have that fed into the audio channel of the RED?
There will be a shift about a few frames, but that is easily to fix, if the delay can be a stable value. With two channels of sound and 20-20Khz, there is plenty of room for information, e.g., at least 4096 steps per "axis".

As long as the on set sound is taken by an external sound recorder, the two channels are free and the motion data is not all over the place.

(In 3D we often use RGB data to store XYZ, so why not use sound? Left channel XYZ, right channel HPB))

Perhaps one day RED's gyro will have a data channel as well, then it could be synced.

====

(An idea that I had send in last year to RED was, to use e.g., the rotation data from the gyro to connect it with the focus setting. Turning the camera, changes "A to B" focus settings.)

Cool ideas.

 

[Björn Benckert]

The THC reciver thing does store the gyro info I think.

One thing with those gyros, both in iphone and in the once inside the red´s is that they don´t handle constant movements, only +/- accelerations so it´s quite tricky to figure out what´s going on if you make a perfectly constant pan and so on.

A while back I bought one of those Xsense moven motion capture suits, basically it´s a suit that has accelerometers on all limbs of the body and link the movments to a somewhat reasonably scalled 3D bone structure. The thing works almost perfect... but the shit thing is those last missing precentages that comes from the estimatimation that is needed to get a solve. That little error is what creates the jitter and make the CG slide on the floor etc. We did a lot of testings not only with the suit but also Xsense SDK and did some tests to make a gyro help thingy for image based 3D tracking, thinking we could par a gyro with syntheyes and have the gyro help syntheyes when image tracking data was low. But long story short, I sold that thing and was quite happy not to have it.

Here is a very rough test with me in the suite giving data to a 3D rodent, as you can see the CG is sliding a bit and with dolly moves and such things will be even more difficult to analyze with gyros.

But the simple way would be to gaffer tape the iphone to the camera, then hit record on camera and on this app:
https://itunes.apple.com/us/app/gyrorecorder/id390416376?mt=8

and when both are recording you hit hard (not to hard but hard enough to make a gyro mark) on top of the camera. Then shoot your take and when done, hit hard on the camera again and stop recording.

Then use some formating software to rip out the XYZ rotations and then jot a setup file or import as rawdata into your VFX software. Then you just align the beginning bump from when you hit the camera with picture and tracking data and then stretch out the keyframes so that the sting also match up with the "end hit/bump" of picture and tracking data.

Then to get it right you need to insert the XYZ rotation info onto an axis that has the proper distance to the nodal point of your lens in correlation to the nodal point of your VR camera. So if the iphone gyro was sitting for example taped to the SSD side of the epic then there would need to be an offset of about 5cm to the left and depending on the lens, about 10 cm behind the lens nodal or more.

When the data is inserted to the right distance from the lens you would also need to smooth the curves in a way as again, constant movements is not visable on gyros, they give a flat line. So you can only smooth out jitter and it is very difficult to get to a prefectly steady camera from there.