Ok, this is exactly what got me into Animation in the first place, to be able to record an actor over a green screen, key him out, create s 3D scene and then combine the two. I have done some green screen composition in the past (on FCP), but only combining 2 2D footage together. I have also learned how to key in Blender using the node editor and I have tested compositing with a 2D footage or even something from a 3D View - and I am very impressed.
What I am looking for is for the actor to really be seen as if he is in the scene and not for the scene to be seen like another 2D video behind him. How do I make the character/actor fully imbedded into the scene?
Make a blend project to key your footage, use the compositor/keyer node and save out to an image sequence with an alpha channel.
Make a new blend project with a 3D scene, import the previously keyed footage as an image sequence apple to a Plane object (use the addon “Images as Planes”), make sure that you turn on it’s transparency via alpha.
Place new plane in the scene.
Remember to add a “Track to Object-camera” constraint to your plane object, so that it orients towards your camera all the time.
Wow, that’s nice! i really appreciate it. After testing it, I still have few problems with this approach though.
When I send the scene to the Node Editor/VSE (and when I finally render and save the scene), only the first frame from those imported images is seen, even when I scroll to the end of the strip (250 frames). How can I make them play as video with the rest of the scene?
I can’t view the content of the plane in the 3D View window until I switch to texture mode (which turn everything else in the scene to texture mode). Will I be restricted to compose that way?
Images import as individual frames in the Outliner. Is it possible to import them as a group to reduce clutter?
But again, thanks a lot for your assistance. I am closer than I have ever been.
well, i would suggest to consider some more aspects:
if your camera is moving you need to matchmove/3d track you camera (you need some tracking marker for that in your footage, dont forget you need also perspective change in the footage to have a great 3d track)
match the lighting of you actor to the lighting of your 3d scene or the other way (what ever you doing first)
a plan or concept before you start, make some scribbles/storyboard what your want to show. this makes it easier to shoot the right camera angles
i dont think you need to place the keyed actor on a plane in the 3d space, it will be harder then to make some keying and integration adjustments. better to render more layers for the 3d scene if you need to hide the actor behind some CG
image you need to change something at the key of the actor and you placed it in the 3d scene as a plane, then you have to rerender the whole project. which can take hours on compley scenes. but if you have the actor just as a 2D layer in the composit then you just need to adjust the keying there and the CG stay untouched. rerendering in just some minutes done.
Thanks pingking23 for your suggestions. I am working on all them right now. Using the footage as a 2D in a compositor (I am not sure if you meant a node editor or a VSE), as far as I know the 3D scene then becomes just the background. It will be as if I just use two 2D images/footage. Unless I am missing something to make that setup succeed, I have tried that and it’s not working.
I am trying to get my character to be seen interacting with objects in the scene, even by just walking on the hallway of a scene, I am not sure if I am explaining myself clear. 3pointEdit’s suggestion is close to what I am looking for even if like you said, it has its own complications. I am still trying to get images on that plane to play when I finally render the scene. Otherwise, if I could find a video format and codec that can carry an alpha channel, I could settle with that 3pointEdit suggestion. Quicktime crashes my Blender ALL THE TIME!
If there are different methods and tricks, including at the shooting stage, I am all ears (well, eyes!).
Sorry for the essay. I have spent so much time trying to figure this thing out, it’s crazy.
Once you Import Images As A Plane (using the addon) you can specify an image sequence, that is a keyed series of frames that represents your movie of the subject. You may have to alter the duration and frame offset of the texture, so that the movie plays correctly.
The great part of adding a textured plane in the 4D scene is that you get the shadows for free (very hard in compositor) and you can do focus racks. I often use 2 3D window views. One in render mode, the other in wireframe to drag objects and select them etc. This way you can watch the effect in render space and still interact with the objects as you would normally.
But it is true that render times can be frustrating.
Thank you all for the time you took to help me out, I really appreciate it. I took the time to explore all your suggestions, and it seems like things are starting to work. Images Sequence now play all frames (thanks to 3pointEdit’s last post that made me realize I should look into the texture tab). Also MOV files don’t crash Blender anymore after I installed a new 2.75 version. I guess there was some kind of bug.
While we are still at this, how do you guys handle this frame-based workflow with so many files involved? Let’s say, I am doing a music video and final editing is done on FCP. First, I key the footage that need keying on the node editor, and then use it as suggested with the ‘images as planes’ node in the scene. Then I render/output as still images. Then I go to FCP and import them to use with other footage yada, yada, yada…I know this is about Blender, but other than by nesting sequences together, how do you manage all these files. I am also going to ask this question on a Mac Forum as it might be more appropriate there, but how can I import these files into a sequence and make them play at the right FPS? When I change Editing Timebase, it don’t seem to have an effect on this, each image still takes 10 seconds.
Other NLEs like FCP create proxies and put them in folders as well, they just do it for you so you don’t realize how much goes on ‘under the hood’.
Blender can collect images sequences (for proxies) into one project based folder and manage that for you but it won’t help with images sequences that you may use for keying etc. You should create your own folders. Perhaps there’s a good idea for an addon in that? As an aside you can actually save PNG files with alpha then simply rename them as .jpg and Blender will still use the png files with alpha in the VSE in place of the regular proxy jpg files.
What would be the advantage of renaming PNG files and how can you rename all files in the folder at once?
I am going to check into that proxy folders idea and see how that works too, the more I know the better. I learned the basic design of VSE but I am not quite comfortable with it yet. I now plan to take full advantage of the ability to create 3D scenes and to use the node editor. I would love at some point to be able to do everything under one roof.
Anyway, someone suggested this to me over at Apple forum, seems like pretty straight forward. Just for someone else who might want to know:
after output your image sequences from Blender, Open QuickTime 7 and go to the menu File > Open Image Sequence, and navigate to and select the first frame of the image sequence.
Click Open and you will be prompted to select the frame rate you want the images to run at, for example, 29.97 fps. Click OK, and QuickTime will open displaying the first frame. You can play it and check it (depending on the horsepower of the computer.
Now, in QuickTime, go to the menu File > Export. When the export pane open, set your Export Options to Movie to QuickTime Movie. Then click the options button and set the video as you prefer, for example ProRes 422 29.97 fps 1920X1080.
Click OK, navigate to where you want the file stores and set the name to the name you want for the file and click.
Yes that is correct but I don’t think it will let you create a quicktime with alpha, for that you must use the QT Animation codec at 32bit (24bit + alpha). I’m not sure but I don’t believe you can do that anymore.
The renaming thing is a hack to get alpha in proxies, which they typically cannot have. You can get command line apps that will do batch renaming of files.
Cool. The alpha thing is no longer an issue, now that I have been able to find out how to make the image sequence play all frames (which means I can just use the node editor to key my footage). I also successfully tested an option to key in FCP and then export a QT file with Animation or ProRes 4444 codec and and use that video in the ‘images as planes’ node.
The process I described above of using QT7 to convert an image sequence into MOV file is after I have composed a keyed footage with the 3D scene. At that point, I will no longer need to export an alpha channel. Sorry if I did not make myself clear before.
The proxies idea sounds interesting, I will definitely check it out.