Journey into Utility AI with Unreal Engine
Unreal Engine comes with several AI features built-in (Behavior Trees, Blackboards, Navigation Mesh and Environment Query System) but hasn’t seen many improvements in this area since the launch of 4.0 several years ago. With mixed results in our projects using Behavior Trees we decided to look at alternatives. Oz pointed me to Utility AI as a system for setting up AI behaviors, I have since been fascinated by the simplicity of the concept to replace Behavior Trees in our projects.
Jakob Rasmussen has published ‘Are Behavior Trees a thing of the past?‘ on Gamasutra which is a good read and suggests Utility AI as the next evolutionary step. In this article I explore the concept of Utility AI and how it can be integrated into Unreal Engine 4. This post is intended as a series where I continue to explore and show my results within Unreal Engine 4.
What is Utility AI?
The concept of Utility AI has been around around for years, and the great Dave Mark has been talking about it on platforms such as Game Developers Conference several times. He calls it ‘Infinite Axis Utility System’ and I highly recommend watching some of his talks about the subject.
- Improving AI Decision Modeling Through Utility Theory
- Building a Better Centaur: AI at Massive Scale
The core of the system is elegantly simple. You give an AI agent a set of possible Tasks to execute (eg. Seek Enemy, Attack Enemy, Flee, Jump, etc.) and on an interval you score each tasks between 0.0 and 1.0 where higher is better and 0.0 means the Task didn’t pass and shouldn’t be considered. You pick the highest scored and run that task. It’s that simple. Each score evaluation can be basic (eg. only check if we have something to attack, yes is 1.0, no is 0.0) or complex with many different variables to evaluate. Distance to target, hitpoints of the enemy, amount of enemies in the area, your own hitpoints, which weapon the target has, etc. You can keep going deeper and deeper to endlessly fine-tune your score evaluation.
Tasks can be given priority weights so that an agent is more likely to flee from an incoming grenade rather than hunt for food. While both may have a base weight or near 1.0 as this agent may be starving of hunger, fleeing the grenade is still the more immediate threat.
Why not Unreal Engine’s Behavior Tree?
As mentioned in the intro we had some issues with using Behavior Trees once the number of available tasks per Agent increased. Every new task could influence the other parts of the tree without this clearly showing up until you ran the tree through some actual playing. It requires a certain way of thinking to create effective trees and allow for easy extension which can be hard to get into. We also found it can be difficult to get back into a tree you haven’t touched in a while.
Furthermore, custom Decorators, Services, Tasks are all individual assets. This can blow up your asset count and jumping between assets cost mental effort and time. I find it reduces my productivity when jumping between asset files a lot. Utility AI still generated some assets such as Tasks, but the overall amount is much lower. Most is of the logic is happening inside the AI Controller class after all. The assets below are from a folder related to a still rather simple stealth game AI. It’s a decent number of assets and will continue to grow.
There is also a performance consideration to be made, but I currently don’t have comparative numbers to claim either to be faster. From what I have heard, this favors Utility.
And finally, stability of the AI systems in Unreal Engine is another factor. We’ve encountered many crashes in EQS, Navigation Meshes & Volumes. Certain features of the engine keep improving at a rapid pace such as the Animation Tools, but AI doesn’t nearly have the same velocity. It’s easy to see if you keep up with the release notes of each engine version. Encountering AI bugs or crashes is therefor a bigger issue as you can’t expect a fix in the next few releases.
The AI Tools do come with some powerful behavior debugging tools such as Gameplay Debugger and Visual Logger. I recommend checking these out and apply them to your projects. These tools aren’t limited to AI or Behavior Trees either and can be used for any kind of gameplay event.
Implementation Basics in Unreal Engine 4
So how should we go about implementing this system in Unreal Engine? In Dave Mark’s talk for Guild Wars 2 they use many C++ written evaluators that can be stacked together to create complex evaluations for each task. Having tried something similar, it proved to be too cumbersome for my specific needs and required a similar amount of jumping around like with the Behavior Tree assets. Blueprint graphs are incredibly powerful, so I implemented this in bespoke graphs and functions. After-all, it’s unlikely Guild Wars has such a powerful tool as Blueprint.
Unreal ships with a Gameplay Framework which implores you to put any AI logic inside AIController derived classes. The custom AIController has a set of functions that calculate the score for each of the available tasks given. A Task in this context is an UObject that will make the AI Pawn do things like moving to a location or firing at a target much like GameplayTasks.
All tasks are evaluated every few seconds in the AIController and the highest scoring task is selected and executed. Only if the running task can be cancelled and aren’t already running that task. A Task can start and complete in a single frame or run for however long it needs (eg. a MoveTo task will keep running until the destination is reached)
Scoring can be quite simple. In the example below we give the task a score of 0.5 if the TargetActor is set. Otherwise it’s zero meaning we don’t want the AI to consider this Task. The talks of Dave Mark go more in detail on implementing more complex scoring functions.
Other Uses for Utility Scoring
The core concept of utility scoring can be used for other features too, for example to determine the ‘best’ object for the character to look at with its head (see video below). We’ve also successfully used this as an alternative to Unreal’s Environment Query System where queries score a collection of actors or locations based on unique scoring functions to select best move-to location, spawn area, etc.
What’s Next?
The next major step is to try out more AI variations to mature the implementation. I’m excited to see how far Utility AI will go and if it’s a long term replacement for Behavior Trees in our projects. Dave Mark’s explorations so far have proven the system works, along with several successful commercial titles. So it’s a matter of finding a smooth integration into Unreal Engine without over-engineering a big framework around it. With that, I hope to share my future explorations in this area with more concrete gameplay examples and code.
If you’ve enjoyed this read, be sure to check out part two!
Awesome. I myself had for some time considered porting the C# Utility AI code (found on GitHub) to C++ and Unreal, but for me Unreal is just a hobby and I couldn’t find enough free time to get to the task, so I waited for somebody else to attempt Utility AI for Unreal. I’ve seen an attempt to modify Behavior Trees with some scoring functions, but it felt a somewhat convoluted workaround.
I guess, Utility AI is good for NPCs that have to act freely and, to some degree, randomly, as a human would. Behavior trees and state machines, on the other hand, fit better when you want strict control over an NPC (scripted actions etc.). But you can combine both – use Utility AI to decide what to do and the kick in a Behavior Tree or State Machine script to perform the action through some series of predefined activities.
I’ve also been wondering about more complex scenarios, such as entering a room with limited interaction options. For example, when an NPC is outside, it might evaluate hunger to decide if it should visit a cafe. But when an NPC enters its “work place”, it should stop evaluating hunger – it might be unacceptable for an NPC to suddenly leave the desk and run a script “search the nearest cafe”. And I found that, in theory, Dave Mark has it covered, too – he explains the solution in one of the videos you mentioned. Essentially, the environment can enforce its own set of scoring functions, removing the ones that are not appropriate for a specific location. For example, an NPC steps through his office door and a trigger event is launched that pops off the “outside” set of scoring functions and pushes on the “at work” set of scoring functions.
Hey Tom, really a great read and very informative! Really looking forward to try and create my own utility AI. I do have two question however. You said you are evaluating the best AI decision in a fixed interval. How do you manage this when an actual task is running? Can your AI quit the currently running task, if another task has reached a higher score while the task is being perfomed?
Also, where do you store your tasks? In some kind of ATaskCollectionComponent?
Each task is derived from a UObject with a bool for whether the task is allowed to be cancelled early.
I currently store my tasks straight into the AIController derived class, a component would work well too as it could allow for some more flexibility but I like having it quite straightforward and don’t think I need any AI outside of the AIController scope.
Hi Tom, thanks for the article.
You said something about the growing complexity of Behavior trees and I agree with it,
however I wonder if for more and more complexes AIs, the scoring ststem becomes very hard to tune.
Because I think that the complexity of this system is focused there.
Did you have any problem with that?
Do you tune the score system simply test playing?
So far I have tuned the system through playtesting. I don’t have a big enough scale yet to fully judge how well it holds up in more complex situations – but with a Ship AI that is doing a pretty decent job and having about 10-14 tasks already it’s holding up decently as the tasks don’t rely too heavily on being tuned all together. As desires increase for specific tasks they will eventually hit and the tuning doesn’t need to come down to 0.01 precision luckily.
I will be doing much more with this system and will continue to write about it as its use develops in the project.
Hello,
Can you show us at least one of the three score functions in the left?
https://i.imgur.com/KyV9fRg.jpg
Thank you Tom! are there any chances you will create an in depth article/series about this topic?
I’d like to do more parts on this topic, I have made some improvements over the last few months to this, but overall the focus in the game project is not so much about AI – so it might take a bit before I get another chance to make the next entry.
Thanks!
wait… shouldn’t that select float have reversed values?
Nope, “if target equals null, pick A which is 0.0 score” it’s a MovetoTarget action which would only be valid is target is not null.
Is the select task function a custom function of yours?
Yes, it’s part of the Utility AI design so it’s not part of Unreal itself.
Are you still working on this series? it would be great to read more about it or why not… have a course on Udemy :D
I still use this logic in my current project, as I get back into improving the AI further I will probably do a part 2 here as well. :)
eagerly awaiting a tut on this
Thank you! I can’t wait!
I’ve never been a fan of Behaviour Trees and tend to fall back to basic Finite State Machines or Fuzzy Logic still – the ‘Utility AI’ system looks to me to be just another implementation of a Fuzzy Logic AI – which is amusing, since this is more or less what Quake 3 was doing 20 years ago;
https://www.researchgate.net/publication/240430519_The_Quake_III_Arena_Bot
I created my own Behaviurtree node which is basicly a CASE switch.It is very very useful for different states of an AI and with this methode switching between TASKs isnt a problem anymore.
This is a great article and I can’t wait for the upcoming entries in this series! Thanks for the free content Tom.
Interesting, I’m certainly excited to try replace my behaviour trees with this system! Just wondering if there’s efficient, non-tick reliant mechanisms to replace some of the basic tasks in the Behaviour Trees, such as the simple “move to location”?
That is awesome news!!!! The AI system in the Unreal Engine is just like you said, convoluted, jump back and forth between windows is insane. They should have keep everything inside the AI Controller BP.
Hopefully they will implement this new system, it is really needed.
This really sounds like a good topic to make another udemy course about :P
I understand it doesn’t fix the “many assets” problem, but wouldn’t a good set of utility nodes for behavior trees work wonders? Similar to this plugin: https://wiki.unrealengine.com/Behavior_Tree_Utility_Plugin
almost seems like this plugin does what this utility set up is doing. why haven’t I seen this around more?
Good deal! Sounds like solid material for another Udemy course ;) (since there are no solid and complete Ai training for UE4, especially tied with anims, this is something many UE4 devs would appreciate greatly)
This sounds exciting. Running logic only a couple times a second and calculating values between 0.0 and 1.0 is something I can get behind. I’ve always had a hair-raising reaction to the mess of assets that behavior trees create and have hoped the system would be replaced eventually. Looks like I need to read up on Dave Mark’s stuff, thanks for the introduction.