Setting up PSO Precaching & Bundled PSOs for Unreal Engine

In recent years Unreal Engine titles have gotten a bad reputation for stutters and hitches. This has gotten worse as more titles move to DirectX 12 on PC. While some of these frame pacing issues can be blamed on level streaming, under utilization of available CPU cores (which has massively improved in 5.4) or simply poorly optimized game code. The most common issue we see discussed at launch is due to a lack of handling PSOs or Pipeline State Objects. These PSOs (used by the GPU) need to be compiled on the CPU if not already cached and will cause stutters as they may cost anywhere from a few milliseconds to several hundreds of milliseconds to compile before we may continue execution.

In short, a “PSO” tells the GPU exactly what state is must set itself to before executing certain operations such as drawcalls. This PSO needs to be compiled and is GPU dependent and therefore can’t be done ahead of time on certain platforms such as PC. For platforms like Xbox and PlayStation this can be done during Cooking of the project as the hardware is known ahead of time. This explains why certain game releases only suffer from stutters on PC and not consoles.

PSO Caching in Unreal Engine

As of 5.1, the engine ships with two systems trying to solve the same problem. PSO Precaching (5.1+) and Bundled PSOs (UE4 and above). In this article I’ll explain both systems and how they currently work together.

The intend as stated by Epic Games is for the new PSO Precaching solution to replace the manual pso gathering (or “bundled PSOs”) pipeline entirely. I did not find that coverage is sufficient as of right now (UE 5.3) for a hitchless experience even in a relatively simple test project.

This article will cover an implementation using Action Roguelike on GitHub to give you the best starting position for your own project. I’ll mostly skip what is already covered by the docs including things like the background information on PSOs and how other APIs and platforms handle this. So I’ll be focusing on Windows DirectX 12. You *really* should read the available documentation along with this article as it provides additional details on these systems.

This screenshot (Unreal Insights) shows a game running without any handling of PSOs. The result is enormous frame spikes when objects are first seen on screen as the PSO compilation steps stalls the game until the PSO is ready to be sent to the GPU. Here that PSO took 54.1ms to compile, meanwhile the game cannot continue rendering. Releasing in this state would surely get it covered on Digital Foundry with #StutterStruggle…

PSO Precaching vs. Bundled PSO Cache

The naming of the two systems can be a bit confusing. The “PSO Precaching” is used for the new automatic runtime “just-in-time” compilation of the PSOs. This system was introduced in 5.1 and is production ready with 5.3 and up (there were some issues and crashes in earlier versions of UE5).

The original system that shipped for years with UE4 requires manual collection of PSOs by the developer and are bundled with the game executable. These bundled PSOs are then compiled when the game first launches, for example in the main menu. You can call these Bundled PSOs or Recorded PSOs. In C++ you may often see it referenced as ShaderPipelineCache.

I’ll cover the configuration settings and my discoveries for both systems below.

How does PSO Precaching work?

PSO Precaching attempts to compile the PSOs ahead of time during the PostLoad() of the object that supports it. This works well for loading screens where the objects won’t be rendered yet. For in-game spawning and streaming this may be too late and compilation may not be finished when the object should be rendered on screen. There is a new feature for exactly this issue which can skip the draw call until the PSO is ready.

// Skips the draw command which is at a different stage from the Proxy Creation skip below. This may cause artifacts as part of the object could be rendered if split among different commands.
r.SkipDrawOnPSOPrecaching=0 (keep this off, it's no longer recommended by Epic)

// Primary CVAR to enable for precaching (its on component level for "best results")
r.PSOPrecache.ProxyCreationWhenPSOReady=1 (on by default)

There are two modes available for late PSOs. The first is to skip rendering the mesh entirely, the second renders the mesh with the DefaultMaterial instead. It’s up to the developer to decide which mode has the least visual popping.

Make sure you read the documentation as this does a good job of covering a lot of concepts new with PSO Precaching.

Optional: Get a baseline Insights trace

It’s good practice to have a baseline when doing performance testing or other forms of optimization.

Without any changes applied, run the packaged game with -trace=default -clearPSODriverCache and the Unreal Insights session browser open (InstallFolder/Engine/Binaries/Win64/UnrealInsights.exe).

Clearing the PSOs with the specified command is essential because otherwise, the game may load compiled PSOs from a previous session stored by the GPU driver.

If your stutters are severe enough then you may not need Insights for rough comparison testing…I still recommend it either way.

Enabling PSO Precache

PSO Precaching is very easy to use, you simply add the following to your DefaultEngine.ini

[/Script/Engine.RendererSettings]
r.PSOPrecaching=1

No further preparation is required. When the packaged game loads a level for the first time, you’ll notice an increase in load time where it will compile the known PSOs. A second time loading the same level should not see this same increase in load times.

Stat PSOCache

You can get some in-viewport stats if you have validation enabled. You have two CVARs for this:

r.PSOPrecache.Validation=2
r.PSOPrecache.Validation.TrackMinimalPSOs=1

Display the following stats using the stat psocache console command.

Precaching in Unreal Insights

You can best see the compilation steps in the game’s load screen using Unreal Insights. It adds a large number of tasks on worker threads during map loading and may increase the total time it takes when its first loaded by the player. These compiled PSOs do get stored by the GPU drivers meaning the next time you load this level, you won’t suffer the same penalty.

To get proper stats here you do need to enable PSO Validation mentioned earlier. Don’t forget about -clearPSODriverCache to have a clean cache every run for maximum stutters.

“PSOPrecache: Untracked” in stats and Insights are most likely global shaders and not missed material shaders. You should be able to catch these using Bundled PSOs.

Combining with Bundled PSOs

While the new system is a great improvement for games running on DX12, it will not catch everything just yet. If you have this enabled and still notice stutters and have confirmed this is due to PSOs (using Unreal Insights – simply look for the PSO bookmarks) then you can still manually gather the PSOs to fix these particular stutters.

Combining both systems is what I am currently doing in the Action Roguelike sample project for the best coverage. Without bundled PSOs I could not get a hitch free experience as of 5.3 since even basic components like DecalComponent are not supported at this time.

How to setup Bundled PSOs?

For bundled PSOs the official documentation does a pretty decent job to get you started. I won’t be repeating many of the things they already cover there and instead just elaborate on the CVARs I discovered, suggestions for capturing PSOs manually and my findings when trying out this system. I am using the same initial set up as the official docs, and modified from there. I’m keeping it brief as I don’t want too much overlap.

Setting up the CVARs

Add the following to DefaultEngine.ini

[DevOptions.Shaders]
NeedsShaderStableKeys=true

[/Script/Engine.RendererSettings]
r.ShaderPipelineCache.Enabled=1
// essentially a light mode to only capture non precachable PSOs (3 CMDs below CAN be skipped for now if you want to run purely on bundled PSOs!)
r.ShaderPipelineCache.ExcludePrecachePSO=1
// Required for ExcludePrecachePSO to know which PSOs can be skipped during recording (-logPSO)
r.PSOPrecache.Validation=2
// Above two only relevant if we want precaching enabled
r.PSOPrecaching=1
r.PSOPrecache.ProxyCreationWhenPSOReady=1

And DefaultGame.ini (may already be set)

[/Script/UnrealEd.ProjectPackagingSettings]
bShareMaterialShaderCode=True
bSharedMaterialNativeLibraries=True

Cook content to generate ShaderStableInfo

After enabling the system, you’ll need to cook the game at least once to generate the .shk files. They will be added in ActionRoguelike\Saved\Cooked\Windows\ActionRoguelike\Metadata\PipelineCaches

Copy the following two _SM6.shk files (if supporting SM6, we do in this example) to a folder somewhere on your PC, in the example: ActionRoguelike/CollectionPSOs/

ShaderStableInfo-ActionRoguelike-PCD3D_SM6.shk
ShaderStableInfo-Global-PCD3D_SM6.shk

(copying both _SM5 and _SM6 .shk files did crash for me when converting the recorded PSOs later in this process. Luckily we’re just interested in setting up SM6 for this example)

Record (Some) PSOs

Now we will record some PSOs to file which we can later inject back into our next build. For the example, don’t worry about covering every possible material or shader. This process is cumulative and multiple recordings can be merged by the next step in this process.

Launch the packaged game with -logPSO as a launch parameter (simplest way is to make a shortcut and add this as in the Target field. I run all my executables with -clearPSODriverCache so that I can consistently see stutters and not accidentally use the GPU’s driver cache which may contain compiled PSOs from an earlier run.

Quit the game and find the .rec.upipelinecache file(s) in Build/Windows/PipelineCaches/ each run with -logPSO will generate another file that can be copied into our ActionRoguelike/CollectedPSOs folder. You don’t need to delete old recordings unless you want to start from scratch as they get merged together in the next step using the ShaderPipelineCacheTools commandlet.

Convert Recorded PSOs

The final step in recorded PSOs is to convert the individual recordings into a single .spc file that will be used by the cooker whenever the game is packaged again.

You can use the following command template to convert the recorded PSOs: (View the latest .bat on GitHub)

E:\Epic\UE_5.3\Engine\Binaries\Win64\UnrealEditor-Cmd.exe -run=ShaderPipelineCacheTools expand E:\GitHub\ActionRoguelike\CollectedPSOs\*.rec.upipelinecache E:\GitHub\ActionRoguelike\CollectedPSOs\*.shk E:\GitHub\ActionRoguelike\CollectedPSOs\PSO_ActionRoguelike_PCD3D_SM6.spc

This runs the commandline version of the editor, executes the ShaderPipelineCacheTools commandlet with expand command and requires the .shk (shader stable files) copied from a previous step along with all the recordings. Running this commandlet generates PSO_ActionRoguelike_PCD3D_SM6.spc (see the docs on naming this file)

Copy the generated PSO_ActionRoguelike_PCD3D_SM6.spc file to /Build/Windows/PipelineCaches/ so it can be used by the cooker the next time the game is packaged.

If you followed along with Precaching enabled, we run this system essentially in a “light” mode where it only captures PSOs not handled by precaching using the following CVARs:

r.ShaderPipelineCache.ExcludePrecachePSO=1
// Validation required to know which PSOs can be skipped during -logPSO
r.PSOPrecache.Validation=2

Testing the PSOs

Package the game again, the generated .spc will be included in the build.

To confirm caching has worked run the packaged game with Insights using -trace=default -clearPSODriverCache or “stat unitgraph” to visualize the stutters in-viewport. Within Insights you can check the Bookmarks or Log and see if there is any new PSOs encountered.

If you load the same level and perform the same gameplay actions as the baseline before making any changes, there should no longer be any PSO related stutters.

Keep in mind that the bundled PSOs need to compile once the game first boots. This starts pretty early in the process, but if you load directly into a level from launch it may not be ready by the time the load screen is complete. Best is to boot into the main menu and confirm the log that compilation started and finished. You can verify this is happening in the log:

LogRHI: FShaderPipelineCache::BeginNextPrecompileCacheTask() - ActionRoguelike begining compile.
LogRHI: Display: FShaderPipelineCache starting pipeline cache 'ActionRoguelike' and enqueued 321 tasks for precompile. (cache contains 321, 321 eligible, 0 had missing shaders. 0 already compiled). BatchSize 50 and BatchTime 16.000000.
...
LogRHI: Warning: FShaderPipelineCache ActionRoguelike completed 321 tasks in 0.06s (0.91s wall time since intial open).

Game Example

The full example is available as the Action Roguelike Project on GitHub. I’ll list the files included for reference below.

DefaultEngine.ini

[DevOptions.Shaders]
NeedsShaderStableKeys=true

[/Script/Engine.RendererSettings]
r.PSOPrecaching=1
; keep this active for validation with 'stat psocache', Insights AND required for ExcludePrecachePSO cvar
r.PSOPrecache.Validation=2
; additional detail in logging for "stat psocache"
r.PSOPrecache.Validation.TrackMinimalPSOs=1
; settings below for bundled PSO steps to combine with PSO Precache
r.ShaderPipelineCache.ExcludePrecachePSO=1
r.ShaderPipelineCache.Enabled=1
; start up background compilation mode so we can run "hitchless" in a main menu (optional)
r.ShaderPipelineCache.StartupMode=2

DefaultGame.ini

[/Script/UnrealEd.ProjectPackagingSettings]
bShareMaterialShaderCode=True
bSharedMaterialNativeLibraries=True

The /CollectedPSOs/ folder in the project root contains the Cmd_ConvertPSOs.bat file to convert the collected .rec.upipelinecache files (can be multiple) and needs the .shk files copied from Saved\Cooked\Windows\ActionRoguelike\Metadata\PipelineCaches (requires at least one cook after enabling PSO CVARs)

The folder will eventually contain many .rec.upipelinecache files as they can be aggregated together by the commandlet. This makes capturing much easier as you need don’t run the full game every capture.

The generated PSO_ActionRoguelike_PCD3D_SM6.spc file must be copied to Build/Windows/PipelineCaches every time it’s generated by the commandlet.

You can of course modity your commands to properly automate this to avoid the mistakes of forgetting to place the updated files in the correct folders. I stuck with the exact workflow as suggested by Epic’s documentation for this example.

Automation Suggestions

Handling Bundled PSOs is a lot of work compared to the new PSO Precaching. Therefor we can only hope that it will eventually be replaced entirely saving everyone a ton of work. Until then I’d like to suggest some ideas for streamlining this process as implementing this falls out of the scope of this article.

• Have QA or playtesters with the game with -logPSO, this would ideally automatically upload the generated file to a server to avoid manual work. Make sure they run on different scalability settings too as these will create different PSOs.
• Create a simple spline actor in every level that can do a flythrough to visit all locations. This might not cover everything so keep cinematics and spawnables in mind. Perhaps these cinematics can be triggered as part of the automation after the fly through has completed.
• Have a custom map for PSO gathering. This can contain all your spawnables from gameplay such as items, weapons.

Don’t forget to run the game on the low/medium/high/epic Scalability settings for full coverage.

Additional Notes

The build configuration does not affect the generated PSOs. You can use Debug/Development/Shipping build configurations for the cooked game builds to gather the PSOs in your development pipeline.

// Use “Fast” for loading screens, “Background” for UI and interactive moments r.ShaderPipelineCache.SetBatchMode pause/fast/background/precompile

You can expose the number of remaining precompiles from Bundled PSOs to display some number or percentage in your main menu:

FShaderPipelineCache::NumPrecompilesRemaining()

There are many more CVARs available in the different PSO related code files:

RenderCore/ShaderPipelineCache.cpp
Engine/PSOPrecache.cpp

Niagara has its own logic and control CVARs for PSOs such as fx.Niagara.Emitter.ComputePSOPrecacheMode but I have not worked with any of those settings at this time.

Closing

This article aims to fill some of the knowledge gaps left by the docs and release notes. As Precaching was announced it took me longer than I care to admit before I had it fully working. Partially as its claims are bigger than what it delivers, as the simple project can’t seem to reach full coverage with Precaching and *needs* the old system for a solid 100% experience. I’m confident this will be addressed and improved in future versions, until then combining both old and new seems like the way to go.

All feedback on this post is most welcome! I’m sure I’ve still missed something or might confuse people with certain steps. I want this article to save the time I had to spend figuring this all out. (Thankfully 5.3 added more info to help get us started!)

Follow me on Twitter and subscribe below for new content!

References

• PSO Precaching
• Bundled PSO Docs
• Example Project (GitHub)
• UDN Post “PSO Precache Hitches”
• UDN Post “Enabling PSO Precaching (Automated PSO caching)”