Camera Tracking with Fisheye Lens?

Does anyone know if blender can do camera tracking or motion tracking convincingly with a fisheye lens? I am looking at the canon vixia mini, and I noticed, that even in 35 mm mode, there is still some slight distortion of the edges due to the fisheye lens. to achieve 35 mm mode, the image is cropped and the resolution is lowered from full hd to partial hd. so, I don’t think this is a standard 35 mm, and I am wondering if the tracking tools will be able to deal with the distortion.
(edit) I just noticed this clip made using a go-pro, which has a fairly strong fisheye effect. looks pretty solid.

Modron-
I have been threatening to do a GoPro tracking tutorial for a while now. With some dumb luck, I’ve stumbled across some settings that, if you fidget enough, will work- and work quite well. I just need to find the time to put it together. I don’t know if you would be able to transfer my success to other cameras, but…

Here’s a clip I did in the summer, with the GoPro mounted to my Quadcopter and a decent gimbal for stabilization:

I’m in the midst of a project, but hope to have the tutorial out in the next week or so.

looks pretty good. I think I will go for it and see what happens.

Ironically, I ran across the same clip you did when I was trying to figure out the settings for the GoPro tracking. I contacted him, and the artist actually didn’t use blender for the tracking portion. Just took data from AE tracking and ported it over. Then composited into blender.

Thanks for giving me another kick in the butt to get this done!

I get the feeling that guy is into minecraft.

In principle, a fisheye picture is just a regular picture that has stuff beyond the borders. If you got enough resolution, you could just crop/zoom a small rectangle on the center and track and undistort using just that, then set the camera on Blender to fisheye and manually match the actual field of view (might need to work on the distortions in post though) to render.

Hello guys,
samiboy: i just posted a comment on your video today, and ended my search in that forum …

I am so much interested in that topic, that i am surprised we find so little information on internet.
So yes gopro + blender + tracking + fish eye + rolling shutter + camera settings … and so on
I always ended up with very bad solutions (solving > 3) and then reconstruction looks weird.
I am far to be an expert in that topic, and i would be happy to help here, compare results, do some tests …

Thanks !

Hello there,
Can you give us some tips before the tutorial is ready like :

  • what gopro model do you have ?
  • which resolution FOV did you use ?
  • what camera parameters did you use in blender ?
  • what were the main issues you faced ?
  • How many points did you track ?

    Many thanks !! :smiley:

If you are serious about tracking in Blender you really need to get Sebastian’s “Track,Match, Blend”.
It goes in depth on what to do with distorted footage and how to track with it.

http://www.blender3d.org/e-shop/product_info_n.php?products_id=147

cool, this means i am “serious” as i do have the DVD :wink:
The thing is that i cannot have “straight” lines in resconstruction mode adjusting Focal length, K1, K2, K3 due to big fish eye, (and i guess rolling shutter effect) of the gopro.
Did you try with a gopro ?

This is why i am asking for help. Either because there is something i did not understand in the DVD
either because there are some other tricks to apply with a gopro

I am not sure it is explained how to find camera parameters for any kind of camera in the DVD … is it ?

Thanks for any hint you can give :smiley:

I calibrated my GoPro Hero 3 medium angle 1080p video using CamCalib (opencv). Basically you shoot a video of a printed paper of checkerboard and let that utility analyze the video.

Settings used here:
focal length 3.96, width 6.248, aspect 1.000, K1: -0.300, K2: 0.098, K3: 0.000

Result:

thanks radialmind :smiley:
I will do some test with this, and post my results here middle of next week !

Here is my first success:


bad lighting and too short, but i had to do several (++) tries
@radialmind: thanks, your camera parameters seems to work pretty well.
The weird thing is that at some poing i had a solve error around 12, i checked carrefully my tracks to see if it was sliding, but they were ok, so i did not understand.
I decided to add more tracks (14 in total) and then solve error went to 0.6 :smiley:
I will give another try soon with a longer shot and better lighting.

Does someone know if there is a web where all camera parameters are kept as a reference ?

Here is a better test:

not perfect as solve error is close to .9 but looks pretty good to me :smiley:

Looking good. I also notice some rolling shutter, that also affects the trackability. Not sure if you can improve much on this one, unless you use a stabilizing gimbal. I have the same issue and I see that the image seems to vibrate under the 3D scene.

In the compositing setup I notice that the video is undistorted first. I prefer to utilize the original footage and distort the 3D scene instead. Do you know if this is possible? I gave this a try by just removing undistort and introducing distort, but it basically fish-eyes the resulting render, it doesn’t preserve the actual position. This means it seems to slide around the footage and it also clips off just before the rectangular edge of the movie screen is reached (probably because the distortion just distorts the 2D final render image, not the actual scenery itself).

The Deshaker plugin for VirtualDub can fix rolling shutter artifacts (at least to some extent). Though i imagine things might get a bit messy with high-FOV fisheye footage…

@radialmind: The only option i can think of is to distort again the final composited result, and then zoom inside to remove black areas.
But then we are loosing a big part of the original footage …

Rolling shutter is really annoying on such cam.
Deshaker from virtual dub is only stabilizing the vibrations, but does not remove rolling shutter (from what i understood)
I used http://public.hronopik.de/vid.stab/ to stabilize my video (after the 3d tracking) because it was pretty shaky

so my next try will be with a better original footage hopefully less shaky
by the way, i also tried to stabilize my video, and do the tracking after … but i guess this is somehow distorting the video even more. and gave very bad results
Do you think this is poosible ?

@radialmind: I will give a try in the coming weeks with CamCalib. Did you try to calibrate with the “wide angle” of the gopro ?

it still seems pretty complex to do fisheye renderings with blender :smiley:
http://blendertarium.ottplanetarium.org/forum/viewtopic.php?f=4&t=31 do a render with blend file named FisheyeRigOctober2012VersionWithDefaultInterface.blend

It got an option to deal with rolling shutter (though, if i’m not mistaken, you might need to rotate the video so that the shutter is rolling top to bottom for it to work). Here is an explanation of the setting: http://www.guthspot.se/video/deshaker.htm#rolling%20shutter%20setting

It only does a good job with mild distortions though; any motion that can’t be modeled as linear within a frame will probably still come out messed up. From my understanding Deshaker just simply interpolates the motion between the top and the bottom when trying to correct rolling shutter distortion, so motions that change in less than a single frame will not be dealt with properly.