Proxy-based baking in Blender

[eye208]

I’d like to discuss a proxy-based bake method that I briefly mentioned in the Cycles baking thread (where it was slightly off-topic):

@eye208"

Proxy-based baking with Blender, in a nutshell:

• Create high-poly model H.
• Create low-poly model L.
• Unwrap L.
• Create proxy P from a copy of L.
• Bake from H to P (object space, selected to active = on).
• Assign baked map to L as material M.
• Bake M to L (tangent space, selected to active = off).

The whole point of proxy-based baking is to get a normal map encoded to the tangent space of L but sampled through P. The above workflow will give you that. In fact it is somewhat superior to xNormal’s cage-based method because it does not require the topologies of P and L to be identical. Which means you can guide the bake vector either by deforming P or by inserting additional geometry (edge loops, insets, outsets).

This triggered the following response:

@JSM"

This is wrong on so many levels :eek:
The whole reason you have cages is to prevent artifacts(ray misses in the bake). (which blender does a ton of) What you end up doing will not prevent ray misses. The currently is no “reliable” way to control ray misses/distance in blender, this is where cages come in.

Also Xnormal can handle split edges…I’ve done it many times…in fact this reduces gradients in the normal map, the only caveat is that the hard edges for edge split modifier can only be were the uvs have been split, this actually is the same rule that applies to all 3d software/baking tool.

(Source 4 years working on a game project…making many, many, normal maps)

Let’s take a quick look at an example that shows how proxies help improve the results of a bake:

If you bake normals from high-poly to low-poly geometry without using a proxy, you’ll run into several typical problems, depending on whether you use split edges or not. These problems can be observed in the two bake results on the right. The object “lowpoly_nocage” has its sharp edges around the caps split, the object “lowpoly_nosplit” is entirely smooth. On the former, you can see the ray misses that JSM mentioned. They occur near the split edges at the top and bottom. On the latter, there are no such ray misses, but since the bake rays are always cast along the (interpolated) surface normal, the area between the top and bottom caps is distorted, and there is a visible seam where the normal map wraps around.

If you look at the corresponding normal maps, you can see that the surface normal interpolation on the smooth object results in a bake that is full of color gradients even in areas where the distance between the high-poly and the low-poly model is constant (e.g. on the top and bottom cap). Color gradients don’t compress well and may cause aliasing artifacts in some lighting situations. So they should be avoided.

The object “lowpoly” exhibits none of the above problems because a proxy mesh was used for baking:

As you can see, the topology of the proxy is slightly different from the target mesh. The proxy is entirely smooth, but there are additional edge loops to terminate the surface normal interpolation near the edges of the caps. This eliminates both the artifacts from missing rays and the distortion. The proxy can also be used to control the bake distance if necessary.

All baking was done in Blender. No external tools were used. If this method doesn’t work for you, please post an example where it fails. I will look at it.

[Ohto]

Thank you for this opening! I have been working with normal maps for some time, mostly not producing good success. Will play around with cage like you instructed and see what happens.

[fahr (Chad Gleason)]

I’m a bit confused. So the method you mentioned is good or bad? Also, how do you “create a cage”? Is this a setting? Are you baking in BI or a test build of cycles? This thread seems to be continuing an argument from another thread, but I’m not sure which method you are describing is associated with which example…

im actually very interested in making normal baking work well, so if you could clarify your post a bit that would be awesome! Thanks for starting this! I’ve had similar baking problems in the past and never solved them. This could help tremendously!

[Piotr_Adamowicz]

How can the cage have a different topology than the target? If this was an object space map, yeah, that might work, but for tangent space this simply doesn’t seem like it could work since the tangents on the lowpoly and the cage will be different. I seem to be missing something.

[myclay]

I don´t understand the Material assigning onto the Lowpoly.

after assigning a material and the Normalmap in object mode,
I click onto baking and all I get is a flat normalmap without any information in it.

It would be nice to see a pictured or even better a timelapse video.

[fahr (Chad Gleason)]

Screw a timelapse! Let’s go full-on tutorial!

[Ohto]

Thumbs up on tutorial!

[NinthJake]

Yeah I don’t really understand these instructions either. Modifying the cage after unwrapping the lowpoly model distorts the UV map so that you have to make a separate unwrap for the cage, then you can’t apply the map to the lowpoly object. And why do you need to bake the material to the lowpoly object if you already have an (presumably correct) map on the lowpoly object already? Because if it doesn’t fit then it makes no sense to bake it down again.

Sorry but I just don’t get this at all. A cage shouldn’t need a UV map in the first place, in fact a real cage don’t even have to resemble the lowpoly object at all and don’t need a material. Ideally you should just be able to make a mesh that encloses the high-poly object completely and then you wouldn’t have to worry about it anymore.

[metalliandy]

Hey all,

I thought I would chime in with my 2p

First of all xNormal can handle both split/hard edges & smoothing groups perfectly well. I use them every day in production with immaculate results and have done for so for years.

Cages are there to stop ray misses, casting intersections/overdraw and to control the length & sometimes angle of the rays cast.
You really don’t want to encourage a situation where you are using different geometry for you cage than your low poly mesh simply because it’s not feasible in a production environment to build 2 meshes. People should always use the low poly mesh rather than building a special custom cage. I really can’t stress this enough. By advocating the building of a custom cage you are reinforcing bad practice which is extremely hard to break, especially when it is regarding normal map baking. Artists should be learning why their geometry is causing issues in the first place rather than spending extra time (and money) fixing those issues with hacks and unreasonable workflows. With a correct workflow custom cages are simply not necessary or desirable.

Waviness is not inherently bad or erroneous & it is view dependent so in many cases you wont notice it depending on the angle of the mesh in the viewport, but of course it is desirable to reduce this in many cases.
In the example in the first image can be fixed by using more sides and having it more closely match the HP mesh.
The reasons that you are getting the waviness in the last image is because you are using 8 sides which isn’t enough to represent the curvature of the surface in the HP mesh which causes distortion. In addition the normals are being averaged out along the surface of your low poly mesh which again is causing projection errors in the mesh.

Adding 2 support loops to the top and bottom of the low poly mesh controls the vertex normals more accurately and fixes these issues for very little cost (geometry is cheap, time is not).

From left to right: HP, 16 sides with control loops, 16 sides without control loops, 8 sides with control loops, 8 sides without control loops

From left to right: HP, 16 sides with control loops, 16 sides without control loops, 8 sides with control loops, 8 sides without control loops. Vertex normals shown.

Without the loops the normals (red arrows) are averaged across the surface which gives an uneven bake. Adding loops controls them better and give much more pleasant results.

The gradients on normal maps are not inherently bad or erroneous either. Yes they can sometimes cause issues but artefacts caused by gradation that many people see is most often due to either a mismatch in the tangent basis between the renderer (baker) and engine and/or by LODs reducing the texture resolution to the point where the bake no longer matches the geometry. Of course LODs on meshes too can cause the same errors because the bake no longer matches the geometry. If the baker and renderer both match 100% then it is entirely possible to view a perfect normal map on a mesh with no hard edges or smoothing groups if desired. However, for the vast majority of cases is desirable to add smoothing groups/hard edges around each UV shell/island which will reduce the gradation to a more acceptable level for both compression and for help with tangent basis mismatches and texturing etc. It’s free so there is no real reason not to use it in production.

I have uploaded the test meshes to sketchfab for people to take a look at if they want.
Everything was baked in xNormal using the LP as a cage. All the meshes have hard edges around the UV borders (aside from the HP of course).

[JSM]

^what metalliandy said :evilgrin:

baked with Xnormal^

Also I did tutorial about baking in xnormal awhile back… was my first attempt at making tutorials…so I bounced around all over the place:

[eye208]

@metalliandy

First of all xNormal can handle both split/hard edges & smoothing groups perfectly well. I use them every day in production with immaculate results and have done for so for years.

There was a thread in this forum last August where people claimed they needed smoothing groups support in Blender’s FBX exporter because xNormal and other tools couldn’t handle split edges. Maybe they were all wrong?

You really don’t want to encourage a situation where you are using different geometry for you cage than your low poly mesh simply because it’s not feasible in a production environment to build 2 meshes. People should always use the low poly mesh rather than building a special custom cage. I really can’t stress this enough. By advocating the building of a custom cage you are reinforcing bad practice which is extremely hard to break, especially when it is regarding normal map baking. Artists should be learning why their geometry is causing issues in the first place rather than spending extra time (and money) fixing those issues with hacks and unreasonable workflows. With a correct workflow custom cages are simply not necessary or desirable.

I agree absolutely. However, there are people who want fine-grained control over bake distance for whatever reason, and that requires support for user-defined bake cages. The purpose of this thread is to show what’s possible. Whether it makes sense is another question.

In the example in the first image can be fixed by using more sides and having it more closely match the HP mesh.
The reasons that you are getting the waviness in the last image is because you are using 8 sides which isn’t enough to represent the curvature of the surface in the HP mesh which causes distortion. In addition the normals are being averaged out along the surface of your low poly mesh which again is causing projection errors in the mesh.

I specifically wanted to show how to eliminate both the extreme waviness and the color gradients without adding geometry to the lowpoly mesh. The low polycount was deliberately chosen to prove the reliability of the method.

Adding 2 support loops to the top and bottom of the low poly mesh controls the vertex normals more accurately and fixes these issues for very little cost (geometry is cheap, time is not).

That’s what I did to the cage. Of course you could just use the cage with the support loops as the final lowpoly and call it a day, but then again, the intention was to show that the method works even if the cage has a different topology. You can tell from the responses that this wasn’t exactly obvious to everyone.

[JSM]

@michalis"

@JSM
Basically, what about converting (transferring is more precise) a height map 32 bit exr to a normal map?

Unfortunately I don’t think that this would work very well;
Most likely because the height map will not have enough info to create all the tangent normalmap info…because the tangents for each direction (xyz) are stored in a color channel…i.e.

“Usually the red channel of a tangent-space normal map stores the X axis (pointing the normals predominantly leftwards or rightwards), the green channel stores the Y axis (pointing the normals predominantly upwards or downwards), and the blue channel stores the Z axis (pointing the normals outwards away from the surface).”-polycount wiki

Whereas a height map only contains up and down (z?) depth info.
At least that’s how I see it…
In my opinion you should always (try) to get your normal map from baking highpoly to lowpoly tangent normal map directly…
Even overlaying normal map detail textures generated from a 2d program can generate bad results.

edit: after seeing what you doing in that thread you linked, this seems that this would be an exception to the rule…probably would fail on really low poly models though?

[eye208]

@JSM

[ATTACH=CONFIG]288526[/ATTACH]

There is waviness in that bake. See how the top emphasizes the highlights while the bottom has stronger shadows.

[ATTACH=CONFIG]288527[/ATTACH]
[ATTACH=CONFIG]288528[/ATTACH]

You are doing it wrong. Not just in the Blender bake but also in the xNormal one. Note how the greebles on each side of the box point towards the center and the normal map is full of gradients. That’s what you get if you use a cage without support loops. And no, a bevel is not a support loop, even if you use multiple steps. The purpose of such a loop is to terminate vertex normal interpolation. Rounded edges don’t do that.

[JSM]

@eye208

There was a thread in this forum last August where people claimed they needed smoothing groups support in Blender’s FBX exporter because xNormal and other tools couldn’t handle split edges. Maybe they were all wrong?

Blargh!! I remember that thread made me grit my teeth lol
I certainly think they were wrong

[mAlkAv_An]

@eye208

There was a thread in this forum last August where people claimed they needed smoothing groups support in Blender’s FBX exporter because xNormal and other tools couldn’t handle split edges. Maybe they were all wrong?

This is not true. The point was to get smoothing group support to swap models between 3D modeling apps without breaking the mesh. For xNormal or a game engine it doesn’t matter whether the edges are physically split on blenders export or not.

[JSM]

@eye208

…but also in the xNormal one. Note how the greebles on each side of the box point towards the center and the normal map is full of gradients.

Going to have to disagree here…the reason why they are pointing inwards…is because of the fact That I’m trying to get the normal map to do too much depth in this case(no baker can fix this). As far as the gradients…you will have those no mater what in baking…you just want to decrease the occurrence of them using hard edges(which I did use…but as you can see still some gradients).

[eye208]

@JSM

Going to have to disagree here…the reason why they are pointing inwards…is because of the fact That I’m trying to get the normal map to do too much depth in this case(no baker can fix this). As far as the gradients…you will have those no mater what in baking…you just want to decrease the occurrence of them using hard edges(which I did use…but as you can see still some gradients).

Here’s what you would get in Blender, using the method I described earlier:

The normal map has no gradients at all:

The lowpoly mesh is as lowpoly as it could possibly get:

[NinthJake]

I would still like a more explained tutorial of your method since it does seem to work well but no-one understands it

[Piotr_Adamowicz]

He forgot to mention he switched his cube to flat shading somewhere down the line. That’s why there are no gradients. It does work, though I think the first bake should be in object space instead. I also think this is way too much trouble - it just saves a few loops here and there, but can take hours to set up for more complex models.

Edit: Oh, it was mentioned it was object space. Right then. Carry on

[eye208]

@NinthJake

Modifying the cage after unwrapping the lowpoly model distorts the UV map so that you have to make a separate unwrap for the cage, then you can’t apply the map to the lowpoly object.

There is actually quite a long list of things you can do to a mesh without distorting its UV layout. Basically all subdivision tools (knife, loop cut, inset, outset…) are safe to use.

Moving vertices in XYZ space, splitting edges, and toggling between flat and smooth shading don’t distort the UV layout either but are very likely to affect tangent space. This is why the first bake is done in object space where tangents don’t matter.

And why do you need to bake the material to the lowpoly object if you already have an (presumably correct) map on the lowpoly object already? Because if it doesn’t fit then it makes no sense to bake it down again.

The second bake is merely a conversion from object space to the tangent space of the target mesh.

A cage shouldn’t need a UV map in the first place, in fact a real cage don’t even have to resemble the lowpoly object at all and don’t need a material. Ideally you should just be able to make a mesh that encloses the high-poly object completely and then you wouldn’t have to worry about it anymore.

You need the shared UV layout to convert cage surface coordinates to target surface coordinates. There may be other ways to do this (shrink wrap, nearest surface projection etc.) but they would only complicate things and make them less predictable.