Rendering Soft outlines in Unreal Engine
Ever since I first wrote about creating mesh outlines in Unreal Engine I have wondered if it was possible to render them as soft outlines instead of harsh binary lines. A good example of soft outlines can be found in Valve’s games like Left 4 Dead or CS:GO.
I have several tutorials on different approaches to rendering outlines. There is one for multi-color using a post processing and another using a translucent mesh material to locally render the outline. The latter can improve performance as a much smaller area on screen runs the material shader. As a bit of a joke I even wrote one blog on rendering outlines for shadows. In this article however I’ll focus on the experiment to create blurred soft-edge outlines.
The basis for all these effects rely on Custom Depth, which is a special depth render buffer where we selectively render meshes into. We use this scene depth data to then figure out mesh edges.
Blurring the Outline
There wasn’t an easy way to blur the pixels at the time nor a decent way to down-sample the render target containing the outlined objects in order to make this blur operation cheaper. Overall it came to be too costly to come with a decent solution and so I stuck with the binary outlines you may have been in several of my earlier posts. Recently I decided to revisit this issue by doing a quick experiment using a modified version of SpiralBlur, a material node that’s available in Unreal Engine, that is using some custom (HLSL) shader code to blur the pixels using a spiral blur method.
Looks pretty decent, right? The effect is more expensive than the binary outlines since we must perform several blur steps in the SpiralBlur-node to get to look decently smooth. Without going the custom engine route there is no way to downsample the post-process step where we sample and blur the information from the Custom Depth buffer. Later in this article I will talk about the performance of the effect itself.
The Material Graph
Unfortunately, I couldn’t find any official UE4 documentation on the Spiral Blur node, there is a wiki page on how to use the material node. The built-in implementation takes the scene textures and over several iterations creates….a spiral blur. The default settings are at about 128 iterations, which is pretty heavy! I’ve used this node as a reference to create my own, which samples the Custom Depth buffer instead of the scene color.
The node graph for it is reasonably simple and most of the logic happens inside the custom node which I added as a code sample below.
Outline Custom Node (HLSL)
float3 CurColor=0;
float2 NewUV = UV;
int i=0;
float StepSize = Distance / (int) DistanceSteps;
float CurDistance=0;
float2 CurOffset=0;
float SubOffset = 0;
float TwoPi = 6.283185;
float accumdist=0;
if (DistanceSteps < 1)
{
return Texture2DSample(CustomDepthTexture,CustomDepthTextureSampler,UV);
}
else
{
while (i < (int) DistanceSteps)
{
CurDistance += StepSize;
for (int j = 0; j < (int) RadialSteps; j++)
{
SubOffset +=1;
CurOffset.x = cos(TwoPi*(SubOffset / RadialSteps));
CurOffset.y = sin(TwoPi*(SubOffset / RadialSteps));
NewUV.x = UV.x + CurOffset.x * CurDistance;
NewUV.y = UV.y + CurOffset.y * CurDistance;
float distpow = pow(CurDistance, KernelPower);
CurColor += ceil(Texture2DSample(CustomDepthTexture,CustomDepthTextureSampler,NewUV))*distpow;
accumdist += distpow;
}
SubOffset +=RadialOffset;
i++;
}
CurColor = CurColor;
CurColor /=accumdist;
return CurColor;
}Performance
Performance was recorded on a 850M mobile GPU at 1280×720 with ~1.5ms measured and on my GTX 980 Ti it runs at 1920×1080 with 0.8ms for the post process material. It’s heavily depending on the amount of iterations in the spiral blur. You will want to keep the DistanceSteps and RadialSteps as low as possible while maintaining a smooth edge. In the demo I settled on 4 DistanceSteps and 8 RadialSteps which is 32 iterations.
Conclusion
To conclude, the answer is yes! It’s entirely possible to make soft outlines in Unreal Engine. I imagine there may be more efficient ways of blurring the custom depth buffer to get similar results, I simply leveraged the available shader code to quickly get to a proof of concept. It’s important to note that being far away from the object with an outline can make it look slightly less smooth, a thinner outline helps and is a matter of tweaking.
Check this video for how to fix in recent engine versions https://youtu.be/453mdZhaC0w
Did anyone fix the issue we are getting past 2019 ?
I can’t figure out how to make it work on recent versions…
Anyone find a fix the “undeclared identifier ‘CustomDepthTexture’ ” issue? Working in 4.24
Hi Dear Tom Looman
Nice work but i have a material compile error,
Error : undeclared identifier ‘CustomDepthTexture’
How can i fix it?
Hi Tom,thank you for this effect. unfortunately I get compiler error too. May I ask you to answer this issue? Thanks!
[SM5] /Engine/Generated/Material.ush(1588,26-43): error X3004: undeclared identifier ‘CustomDepthTexture’
I am getting this error. Any idea why? I cant get the material function to compile cuz of this
I used 4.26 to create a custom node but this kind of problem also occurred
Hey Tom, if you have a spare minute, please take a look at this https://www.reddit.com/r/unrealengine/comments/c1svjn/help_limit_or_remove_inward_brightness_on_mesh/
Hi Tom,
this effect looks really good but unfortunately I get compiler errors in 4.21.
I recreated the material function and the post process material a few times from scratch but the error remains :(
The problem is in the custom node and the compiler error already shows up in the material function (“error X3004: undeclared identifier ‘CustomDepthTexture’)
So is there a solution for that problem?
Thanks and best regards,
Daniel
float3 CurColor=0;
float2 NewUV = UV;
int i=0;
float StepSize = Distance / (int) DistanceSteps;
float CurDistance=0;
float2 CurOffset=0;
float SubOffset = 0;
float TwoPi = 6.283185;
float accumdist=0;
if (DistanceSteps < 1)
{
return Texture2DSample(SceneTexturesStruct.CustomDepthTexture, SceneTexturesStruct.CustomDepthTextureSampler,NewUV);
}
else
{
while (i < (int) DistanceSteps)
{
CurDistance += StepSize;
for (int j = 0; j < (int) RadialSteps; j++)
{
SubOffset +=1;
CurOffset.x = cos(TwoPi*(SubOffset / RadialSteps));
CurOffset.y = sin(TwoPi*(SubOffset / RadialSteps));
NewUV.x = UV.x + CurOffset.x * CurDistance;
NewUV.y = UV.y + CurOffset.y * CurDistance;
float distpow = pow(CurDistance, KernelPower);
CurColor += ceil(Texture2DSample(SceneTexturesStruct.CustomDepthTexture, SceneTexturesStruct.CustomDepthTextureSampler,NewUV))*distpow;
accumdist += distpow;
}
SubOffset +=RadialOffset;
i++;
}
CurColor = CurColor;
CurColor /=accumdist;
return CurColor;
}
This doesn’t output errors, but gives as a result 1/4 of the screen blue overlay of the character depth. Maybe I’m missing something?
Cesar, did you manage to solve this? I’m seeing the same issue and getting a 1/4 of the screen too.
return ViewportUVToSceneTextureUV(UV,14);
Use this UV
i have the same issue, any news on this one?
im getting [SM5] Base pass shaders cannot read from the SceneTexturesStruct, anyone know how to fix it? or how to get customdepthtextures in any other way?
That’s great, thank you for this. Finally I’ve got it running. But I couldn’t figure out how you do the scalar parameter in the material function. I don’t see an input if I use the regular scalar p.
Speed: I just tested it on Samsung galaxy S2: With steps=4 It’s running but not fluent, with steps=8 it’s waaaay to slow.
Hey Olaf, do you mean you couldn’t create the scalar input for the material function? Creating a scalar parameter input in a material function requires a function input node, if you just type ‘input’ into the search bar on the left of the shader window, a function input node will appear, and you can then set whether its scalar input in it’s properties. Hope that helps
Hi Tom,
Great article, I have this working and it’s great! I’m trying to set this up where by it occludes behind walls. I can achieve this using your other outline setup, by just adding a oneminus after the Determine Occlusion set of nodes, and then multiplying that just before the lerp input, that worked great for that.
I see this is already occluding against the mesh that has Custom depth switched on. but I can’t get how to have walls etc do that same? It’s driving me a bit nuts.
An info on that would be awesome Tom
Aaron
Hi, thanks for your guide, but I found a problem: I can’t change the outline color using “custom stencil value” (I am able to do this with normal stencil material in your previous tutorial)
That’ll require some more work, it’s quite different from my other FX that support the multiple colors. This really was just a proof of concept on the softness of the outline.
Cheers!
Hi Tom! May I ask you to answer my previous question? Thanks! :D
Hi! When “saving” it shows a pop up with this message:
The current material has compilation errors, so it will not render correctly in feature level SM5.
Are you sure you wish to continue?
After continuing, the call to LODZERO shows an error:
Error [SM5] Function LODZERO_SpiralBlur-Texture: Missing function input ‘UV’
I would appreciate any help!
Thanks
Looks like you may not have specified any inputs in the node itself (see SpiralBlur in the engine on how that works)
Hope that helps!
Thanks! I’ll look into that tomorrow at work!
Hi, I am trying to use this example but I have a couple of questions.
Does the output of the PP_BlurredCustomDepth_OccludedOnly blueprint is connected to the Emissive Color property of the material node or the Base Color property?
Also, I am getting a compiler error when trying to compile the material, it shows “ERROR” just below the custom function (LODZERO_SpiralBlur – Texture), and in the logs it goes:
“Error [SM5] SceneTexture expressions cannot use post process inputs or scene color in non post process domain materials”
Seems that its complaining for the UV property is not connected, although you do not have it in the image either, so, there must be something I am doing wrong.
Any help is appreciated!
Regards,
Lermy
Hi Lermy,
You did not set the “Material Domain” to be a post-process and it’s still considered a surface material. Click on the main node in your material to access the properties and change it to be a Post Process. This will expose ONLY the emissive channel.
Hope that helps,
Tom
What’s the “1e+04” in the substract node at the bottom of PP_BlurredCustomDepth_OccludedOnly ?
That’s 10000 units, I don’t even quite remember why I went for that – in practise defines the available range of custom depth (if you lower this value you’ll see it breaks beyond a certain range, since everything will be clamped to 1’s. If I’d clean this up I would probably map the custom depth range between 0 and 1 properly (This really was just a quick proof of concept on the method)
Fixed the duplicate line in the code, thanks for pointing that out!
@Noodlespagoodle I’m using 4.17, I removed the duplicate line and had no other errors. Did you copy/paste the Scalars from the engine SpiralBlur, maybe you made a tiny mistake there? <3
This looks like it could be incredibly beneficial to my project!
I unfortunately get errors when implementing it though – “redefinition of ‘CurColor”‘ is the initial error. If I remove that duplicate line from the code that goes in the custom node I get a bunch of other errors about missing declarations and not found Intrinsic functions. I tried this out in both 4.16 and 4.17 to no avail.
Any way you could either send me a copy of the function or help me get it working?
Much appreciated!
-DSP
I fixed the embedded code sample, I accidentally had a duplicate line in there.
awesome