Layer composting with masks after applying effects

Compositing, not Composting

I’m a bit stumped and hope someone can help me out. I have a smoke sim that is interacting with the scene. Currently it’s baked at 300 which doesn’t take too long, but still has pretty good quality. Only problem is that there is still some blockiness in this sim at 300. 500 makes it smooth, but introduces some crazy distortion. I had an idea to keep the 300 bake and simply blur the smoke a little to hide the blockiness. This works, but as you can see from screencap, I get some halos of sorts around the objects in the scene.


It seems Blender is compositing the scene and layer masks like this:


What I’m wanting is this:


Where Blender will take the smoke, blur it, THEN apply the mask. Is there a way to steal that mask its using in Render layers to do this, or should I be looking at z-depth, or…???

Any advice would be great.

Thanks!

1 Like

You should make the alpha over inputs the other way.

If I do that, the smoke is completely behind the background. I need it enveloping the objects in the foreground, basically like the still in the grab, but without the halo from the blur.

Are you blurring the alpha channel of the smoke too? Why not route the alpha from the smoke layer to the factor input of the Alpha Over node. If it needs a blur do that separately.

It should just work with a separate render layer for just smoke, and a layer for background.


The problem (and, the source of the “halo”) is that you are using a “blurry mask.”

Now, one thing that you can do, sometimes, is to produce that “blurry mask” and then its inverse. (Like a photographic-negative of a photograph taken of that mask.) Use the blurry-mask for one part, and the inverted blurry-mask for the other. Then, combine the two parts.

In this way, the two pieces that are being combined do not overlap: where “Mask” is more-opaque, “Not-Mask” is exactly that much less-opaque. This removes the halo-effect that’s being produced by arithmetic addition when the two channels are combined.

So, in your second diagram (“what I’m wanting is this” …) I would insert another box between “Background” and “Alpha Over” and label that box “NOT-BlurryMask.” And, to make it clear what I was doing, I’d draw a dotted-line between that box and the one labeled “Mask,” and probably slide “Alpha Over” to the right.

I tried using the Alpha into the factor and it didn’t give me what I needed. Close, but not quite. I don’t understand how to do an inverse mask, but I appreciate the help. I did find though that running the alpha into the viewer gave me a nice z-depth matte that I think will work. I’ll have to do a few render passes to make this happen, but I did a test on a still and comped it all in AE and it works great. Just takes render time.

thx ! It was helpful!