#game artificial intelligence

[askagamedev]

When fighting AI controlled enemies, every now in then it happens that said enemies just stop acting. They don't react to anything, they don't act, they just stand there and take hits from me. What's the technological background of this kind of bug?

Most of the time, game AI enemies follow some kind of preset decision-making process. It's usually either a series of conditions that get evaluated until some decision is made, or it's a [finite state machine] that transitions from state to state based on timing and conditions being met. Regardless, decisions are made whenever the proper conditions are met.

Sometimes (usually through uncaught errors on the part of designers like me), the AI gets into a bad state where there's no valid actions to take. In the case of a finite state machine, this could be an orphan state with no exit condition, or a series of states that infinitely loop over and over. In the case of a condition series, it could be an edge case of some weird condition being met that has the AI brain bail out before it reaches any valid action to take. Decision-making that requires certain timing, for example, can also result in this kind of state if the timing conditions for the next step doesn't get met. Regardless, AI that don't do anything are usually stuck in a bad situation that they have no valid means of handling.

Most of the time, we build in some form of debugging visualization to help us figure out what step of the AI decision-making process each AI is currently up to. This way we can catch the various errors that crop up for unhandled edge cases. Unfortunately, the more obscure edge cases can sneak through our testing and make it out to the live game.

[Join us on Discord] and/or [Support us on Patreon]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Frequent Questions: The FAQ

[askagamedev]

Do enemies in games from the last 5-10 years ever act independently? I've watched a few talks about how the enemies in, say, the Arkham games have a master AI that coordinates between each minion. Even as far back as Left 4 Dead there's been much ballyhooed about 'director' AI. So: in modern games, is every enemy overlooked by a larger controlling structure? Where would we find independent agents?

You find independent agents when it makes sense to do so - any kind of primarily 1v1 game against AI, for example, will always be a single independent agent making the high level decisions. This means fighting games like Street Fighter and Dragonball FighterZ, card games like Hearthstone and Legends of Runeterra, and turn based strategy games like Chess and Civilization generally all have independent AIs. Any kind of solo boss monster encounter is likely an independent agent (or a scripted encounter), like in Shadow of the Colossus or the Legend of Zelda games. Any game where the victor is determined by a last-one-standing mechanic like Among Us, Fortnite Battle Royale, or Fall Guys is also likely to be independent agents.

The directed group AI makes a lot more sense when you consider that the goal of the AI mastermind is not to win but to help create the experience that you want the player to have. Individual agents with no coordination can cause bad player experiences, such as getting stun-locked by consecutive attacks from multiple enemies in combat. It’s a legitimate thing that can happen, but most it’s also extremely frustrating if it is a commonplace event. Having an AI mastermind helps smooth out these rough patches by coordinating enemy attacks so that the player is challenged but not necessarily overwhelmed. 

The AI mastermind also helps the player experience by coordinating the enemies to move into places for ways to lose or die in cool ways. The Arkham combat purposely keeps the number of bad guys in combat around Batman to a manageable level for several reasons - you want enough enemies for Batman to switch between while fighting, grouped up well enough that the player can choose which combat move to use and see a good result, but not so many as to overwhelm Batman if they hit him. For coordinated combat animations that involve more than one actor animating together (e.g. Batman using Nightwing’s staff to help him land on a bad guy), individual AI agents must move into place for the animations to track correctly when played while the other bad guys don’t interfere by blocking the camera’s view or interrupting the cool looking move by attacking. 

These goals are generally held by independent agents in the example games I mentioned above as well, but they don’t need to coordinate multiple sub-AIs to behave since there aren’t any sub-AIs to be coordinated. In the case of free-for-all games, it actually is ok in those circumstances to cause those kind of coordinated bad feels on a player because it’s actually expected gameplay (and entirely reasonable for the player to be on the giving end where they join one independent AI attacking a second independent AI). It’s the nature of the beast in general - the purpose of the AI is rarely to win, but to provide a fun and engaging experience for the players who interact with them.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Frequent Questions: The FAQ

[askagamedev]

When designing non boss enemy AI (i.e. AI for group encounters) does each enemy usually have its own set of commands or are all of them tied together under the hood to prevent things like the player getting hit with to many things to deal with at once? Also do you design AI scripts to be modular to make tweaking game balance slotting in new enemies (for things like DLC expansions ) easier? And if so how would you go about designing a "just slot the enemy in" AI system?

Game designers typically have to think about how the player experiences the encounter as a whole, rather than each individual part. The individual enemies might be interesting and fun on their own, but if you force a player to spend 15 solid minutes killing them with no break, it mentally fatigues players and causes them to check out. The number of enemies can be just as important as what each enemy does. These different factors cause different effects on the way the players feel while playing the encounter. 

Furthermore, players want a variety of experiences so we need to be able to create a variety of encounters. Sometimes we want players to feel powerful, so we give them enemies that they can stomp. It’s especially rewarding for players to stomp enemies that used to give them a hard time - it makes them feel like they’ve really grown in strength. Sometimes we want players to feel challenged, squeaking out a win over a dangerous foe. That sense of knowing that one was pushed to their limits and still managed to achieve victory is also exhilarating. And sometimes we even want players to relax a bit with reasonably easy but not too easy foes. Having a good variety of encounters helps keep things interesting.

In order to do this, we start from the experience we want the player to have and work our way backwards to build a formula and system in order to reach that goal. In order to do this, we usually have some sort of fight AI coordinator, an invisible puppet master that tells the AI when to move and when to attack. This allows us to maintain the appearance of a threat while not actually posing much of one. As one example, we can do this with tech like the “attack token” constraint - only AI holding an “attack token” is allowed to attack the player at any given time. After the AI finishes its attack, it passes the token to another who can then attack. The rest of the AIs try to look threatening by moving close to the player, wandering around, or making scary noises, but can’t actually attack without the token. In more challenging battles, we might want more than one active attack token. This keeps the relative threat to the player at any given time controlled, rather than constantly scaling with the number of enemies.

We also rank the threat of enemies relative to players and put them into specific buckets so that they are about the same in terms of challenge relative to each other. A goblin, a centipede, a giant rat, and a giant crab might look different but, if their stats under the hood are similar, they’ll all pose the same threat and take the same amount of time to defeat. If we have a way to quantify threat to the player in various ways, we can then assemble combat encounters quickly - four goblins and an elite ogre is roughly the same as three crabs and two sharks is roughly the same as two elite wombats. Then, when we place combat encounters in the environment, we can figure out about how hard we want it to be and toss a few enemies in to meet the desired threat level.

Much of system design is creating a set of rules for things to follow and then creating content to utilize those rules in a way that is engaging to the player. That’s what we do when we create the rules for combat, enemy stats, various AI behaviors, and so on. If you want to understand system design, take the game you’re currently playing and break it down in terms of a formula - a unified means of explaining a system. If it is a good formula, it should accurately predict how something new to the system will work. Then move on to a new level or area and see if your formula is correct. It isn’t just damage and numbers, but how long things take to die, how many enemies there are, how often they attack, what kind of attacks/debuffs/etc. they use, and so on. It’s all built on a set of rules we’ve composed for ourselves.

[Join us on Discord] and/or [Support us on Patreon]

The FANTa Project is being rebooted. [What is the FANTa project?]

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Frequent Questions: The FAQ

[askagamedev]

All that stuff about making Super AI was Super Interesting, but how is the normal not-playing-games-against-itself-to-try-and-win AI designed & developed?

Ok, so… Game AI is a really deep subject. Like… deep enough that there are professional engineers and designers who are employed full time to make this sort of thing happen. I can’t do the entire subject justice in a small post I threw together in one evening. So instead, you’ll all get a multi-part series on the topic because it’s something I find fun. 

Let’s take a look at the eagle-eye view of the topic.

Developing Game AI (Part 1)

Normal game AI is definitely different from experimental/theoretical AI. Experimental AI like Deep Blue, AlphaGO, the UC Berkeley Overmind, or OpenAI are developed to see how far computer learning can go. Game AI is primarily geared towards providing a fun experience for the player. This can have some amount of overlap with experimental AI, but the fundamental goals are different, which necessitates different design and implementation. From the very beginning, we developers establish ground rules for what the AI is supposed to do, as well as its limitations. We have to answer questions like:

Does the AI represent an equal opponent for the player? (E.g. Hearthstone, Chess, Street Fighter)

Will the player fight many AIs at once? Is the AI effectively disposable? (E.g. Dynasty Warriors, Doom)

Will the AI need to cooperate with the player? (e.g. Mass Effect, StarCraft)

Will the AI need to cooperate/interact with other AI? (e.g. Fallout, Skyrim)

Will the AI need to navigate non-uniform terrain? (e.g. Uncharted)

One of the bigger factors to the answers to these questions is how long (we think) it will take us to actually implement these things. We are, after all, always on a schedule. Complicated and extensible AI is great, but not if we can’t fit it into the schedule or budget. Once we figure out the limits of what the AI is supposed to do, we begin with rudimentary design and implementation.

AI development is generally broken into two separate parts - getting the AI to make a choice, then actually executing that choice. These two parts can be (and often are) developed independently of each other. 

Making the choice would be figuring out which card to play, which spell to cast, or which ally to heal. Typically, this involves some sort of prioritization system, where the AI evaluates potential options and chooses one based on some set of established rules. Is attacking more worthwhile than using a spell? How about using an item? Should the AI try to maximize its use of mana per turn for tempo, or should it try to maximize card advantage? Which enemy should the AI target for this attack? The most dangerous one? The one with lowest HP? And how do you quantitatively define “most dangerous” anyway?

Executing the choice is the less sexy, but incredibly important part that most fans don’t think about. In fact, most of the design and implementation time for AI in games is often spent making the execution work. After the AI chooses which spell to cast, how does it actually do it? The AI probably has to move into range, which requires pathfinding. The AI has to cast the spell, which means that it can’t be taking damage, unconscious, silenced, etc. The AI might have to consider environmental hazards when pathfinding. The AI might have to consider its location in the world - say that the player is trying to kite an enemy out of its home base, but the game must stream the environment from the disc. If the player goes too far from the enemy base, it will get unloaded from system memory, and likely take enemy with it. How do we handle such cases?

In addition to that, in many games there’s also an even lower level of AI that often needs to work as well. At its most basic level, the character animation system is governed by AI as well. If the player shoots an enemy, it probably needs to play some sort of take-damage reaction, right? How does the enemy know to do that? Something has to prioritize which animations are playing when, and which take priority over what. Dying usually trumps climbing a ladder or casting a spell. There’s generally an AI that handles what animations a character needs to be playing, and when to transition from one animation to another. Remember, the machine doesn’t have any inherent concept of matching motions or positions or anything; any sort of animation transition must come from some sort of set of rules that we create. That also includes keeping track of states like death, unconscious, aggression, snared, which direction to face, etc. and that AI must be able to override or pause the higher order functions too. After all, the AI shouldn’t be trying to navigate to a point on the map if it’s dead.

Building these sorts of systems require a lot of thought and work. The vast majority of developing game AI is spent solving these kind of core problems on a system-wide level. Things like emulating a personality or adding polish touches often come super late (if at all) in the development cycle, simply because getting the core elements to work requires so much time, effort, and iteration. AI can be as simple or deep as we want, but the very basic stuff requires a lot of work to set up.

Next time, I’ll talk about how we get AI to evaluate choices.

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Frequent questions: The FAQ

[askagamedev]

How is Game AI developed? Does AI start acting optimally and then they make it fun? With Dota 2 showing off an AI that can beat the best players 1v1, I wondered why I should be impressed? I thought designing AI that always plays perfectly was relatively easy and the hard part was designing mistakes to make it useful (or fun). How wrong am I?

There’s a pretty big difference between the AI you’re used to and the AI that they demonstrated at The International 7. Here’s the thing - if the rules of the game are simple enough such that there is an optimal strategy, then yes - you can start with that optimal strategy and always win or tie. If such a thing exists, the game is considered “solved”. These games are generally deterministic - you don’t even need to play in order to tell who will win. Tic-tac-toe is the textbook example of such a game - if played properly, the second player can never win. 

However, most competitive games aren’t solvable - there isn’t always one perfect way to win, because the game has play and counterplay. For example, consider the game Rock-Paper-Scissors. Is it possible to come up with a winning strategy for this game? In general, the answer is “not really” - it entirely depends on what the opponent does. If you have a good read on your opponent you can defeat her, but there is no perfect solution. An AI can be trained to play rock-paper-scissors but it can’t always play perfectly because there is no such thing as true perfect play in Rock Paper Scissors.

Another means of gumming up the work is simply to have too many confounding factors for the AI to account for everything, especially in real time. For example, on Wall Street, several firms have set up banks of computers that trade stocks at super high speeds. Super high-end machines running complicated algorithms place buy orders here, sell orders there, often microseconds apart, trying to take advantage of their inhuman reaction times to snap up favorable transactions. However, even these machines aren’t perfect. If they were, they would obsolete every other form of stock trading on the market. However, they are not infallible - they are susceptible to errors in the algorithm, incomplete or incorrect information, taking too long to crunch the numbers and make a decision, and more. The cost of these computers and the internet connections that connect them run in the millions of dollars all to shave milliseconds off of the calculation response time, and even then they are not money-printing machines. They often get it right, but they will also often get it wrong, losing millions of dollars for their owners.

The AI at TI7 was playing a very simplified version of Dota - 1v1, mid lane only, and both players were restricted to Shadow Fiend. With such a setup, there were few enough factors that the AI could process them all and come up with a winning strategy. And even then, it wasn’t a perfect strategy that was made by a human, but it was a set of strategies the AI itself built from playing millions of games against itself and finding points for improvement. Even though it managed to defeat Dendi, there were many human players who managed to outsmart the AI by doing things it had never encountered before, such as dropping items on the floor. Now consider expanding that challenge to a team of 5 heroes selected from a potential pool of 105, each with different builds and item loadouts. The complexity grows by many orders of magnitude. Then consider the actual details of the game itself - items, build, team role, scouting, creeping, and so on - and the complexity grows by many more orders of magnitude. Coming up with the correct strategy here is very difficult - the AI would need to play millions of games against good players in order to construct a viable set of strategies and counter-stratgies that would make it even remotely competitive.

Is there an optimal strategy for Dota 2? Possibly, with enough computing power and data, machines could figure it out. More likely, there will be a metagame that evolves from the available choices - a variety of viable (counter) strategies with reasonably close win percentages. But as it stands, there is no perfect script that we can just tell an AI to follow that will guarantee victory, because there’s so many different strategies and counter-strategies at play that there’s no way for humans to account for all of them.

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Frequent questions: The FAQ

[askagamedev]

When and why have bots fallen out of style in what feels like the majority of modern shooters, particularly multiplayer shooters? Such a great feature, left out far too often. Thanks, great job with the blog!

Building bots requires a lot of resources, due to the nebulous nature of how they are supposed to behave (especially in situations where goals or good strategy are less well-defined), and because problems with bots can be difficult to reproduce and test. Building a bot isn’t trivial - it requires a lot of engineering and level design time, because the AI needs information in order to function.

So… why do they act this way? Probably because bots are supposed to be:

Adjustable (Can be set to different levels)

Scalable (can work either alone or in larger numbers)

Effective (can win the game)

Not overshadow the players (e.g. you don’t want the bot to get Play of the Game)

These ideas basically require some design and engineering time just to figure out how these will actually work. However, these requirements must be built on top of the game’s fundamentals, such as:

Navigating the map

Recognizing important locations and/or powerups

Achieving objectives

Non-deterministic (will not always do the same routine)

Using weapons and powers that aren’t just “shoot the opponent”

Recognizing the game mode’s rules (capture the flag vs capture point vs death match vs …)

So in order to do this, each map will need:

A navigation map so that the bots know how to traverse the map (including special traversals, like jumpable gaps marked as such).

Points of interest marked with some way to tell which is more valuable at the moment.

Objective locations marked with some sort of algorithm to determine whether to go for the objective (e.g. consider the amount of danger present in relation to the objective), as well as stuff to actually tell the bot to stand in place and activate the objective.

And, the bot AI itself will need:

Some sort of prioritization system to decide what objectives or points of interest to chase, especially based on game mode. Typically, they’d take factors into account like distance, number of teammates nearby, number of teammates heading in that direction, number of enemies nearby, number of enemies heading in that direction, and relative value of the objective into account when determining this.

Behavior for recognizing how weapons are used, and (optionally) maps marked with areas for specific environmental use. So, for example, you’d have areas or objects on the map specifically marked as “pushing weapons can be used to shove enemies off of ledges here”, “explosive object here”, and so on for enemies to utilize specific effects to their best ability.

Some algorithm to decide what objective to pursue. One can make use of randomization, or one can decide to split things up by determining how many teammates or players are going to a specific objective or point of interest, and then choose a target based on that. Then you just need to figure out how often it needs to happen - does it happen on death? On objective captured/achieved? On some sort of other event, like a player leaving or arriving at a capture point?

You need to remember that this is all done in addition to the rest of the gameplay - interacting with objectives, shooting each other, different weapon behavior, environment art, level design, matchmaking, game modes, etc. This is all built on top of those. Then you need to start thinking about how to test this behavior, and constructing the environments in which you test them. Individual behaviors need testing, and aggregate behaviors need testing in order to make sure that the individual pieces can work together. This also necessitates useful debugging tools, such as showing the behaviors that the bot is currently running such as what it wants to do, where it wants to go, and how it plans to get there.

The biggest issue, as you may have noticed, is just how much these different systems are really interdependent on each other and how dependent they are on the map itself. There’s no way for a bot to understand what happens on a map without actively marking elements of it as such so that the bot understands how to move around. Even a small map change can cause bots to break if the navigation map isn’t properly updated. A lot of things that we understand intuitively (due to our brains being very good at parsing information) must be explicitly broken down for the AI in order for the bots to use that information. Creating functional bots requires an enormous commitment of resources, and it is a fragile system due to the sheer number of dependencies bots have. This is why many games don’t have them - they just require too much commitment in terms of resources to build.

Got a burning question you want answered?

Short questions: Ask a Game Dev on Twitter

Long questions: Ask a Game Dev on Tumblr

Frequent questions: The FAQ

[askagamedev]

Is it just a myth, or does sometimes AI features, that are more marketable and produce necessary "wows", get higher priorities (money, development time, computation time) on expense of overall quality of the AI? Or is it simply more difficult to make the AI constantly not to look bad?

The problem is that nobody really agrees on what “good” AI is. Is it AI that is difficult to defeat? Is it AI that emulates actual human psychology when making decisions? Is it AI that is fun for players to engage with and defeat? Nobody really agrees as to which of these criteria is actually what qualifies AI as “good”, least of all players. Players are really really good at saying they want one thing, but their actual behavior over time shows that they overwhelmingly prefer another thing altogether.

AI in general isn’t particularly marketable. If we devs can’t agree on what good AI is, players certainly don’t understand what it is. Their spending behavior shows that they’d generally rather see a game with “65+ hours of gameplay”, “epic storyline”, “fast-paced action”, “responsive controls”, “60 fps at 1080p”, etc. than “Intelligent AI”. It’s really hard to market something that nobody can agree on. Since players don’t really know what they want in terms of AI, we devs don’t spend quite that much time on it. Our main operating goal when it comes to AI is (usually) to create something that is fun and engaging to defeat. However, this means that the enemy AI must be predictable and solvable to the player, more like a puzzle than an actual problem-solving entity. Players generally don’t react well to AI that outsmarts them. Most of the time they get frustrated and think that it’s cheating.

[askagamedev]

In one of your past posts, specifically the one on Feb 17th 2014, you mentioned that an AI that acted sensibly (in your example an AI that would flank the player) was poorly received. For aspiring designers, should the takeaway be that designing sensible computer opponents (opponents that "play" the same game as the player in terms of mechanics and rulesets) is a bad idea? Thank you.

Is designing “sensible” AI a bad idea? Not really. It depends on the sort of role the AI is meant to fill. We can typically divide AI design into two major categories - asymmetrical and symmetrical AI, which each have different implementations and assumptions made. Both categories, however, must retain the same core element - the takeaway should be that the AI is fun to play against, regardless of whether it is symmetrical or not. Read on to find out what the difference is, and what role each type of AI serves.

Asymmetrical AI is when the CPU doesn’t necessarily have to follow the same rules as the player. Monsters in a game of Diablo die by the hundreds, for example, and the player (almost) never feels disappointed that they don’t act or behave too intelligently. Even a scripted boss fight like the raids in World of Warcraft aren’t necessarily treated as particularly sensible. Asymmetrical AI should be something that the player can understand and predict. It usually functions like a puzzle to be solved, or an obstacle to be overcome. As such, it should be beatable - even easy, most times, since most players don’t want to have to solve a complicated problem each time they encounter a simple enemy. Concentration is a finite resource that players must spend, so it’s usually better to have the amount of concentration a player needs to put in wax and wane in order to keep things interesting.

The fun in defeating asymmetric AI comes from learning, understanding, and then performing the right skills to complete the puzzle. As such, the AI should broadcast clues to the player about how to solve the puzzle as presented. These clues can be visual, they can be audio, they can be sent via controller rumble, and so on. The player’s job is to read the clues, figure out the puzzle, and solve it. The player then earns the reward - experience, loot, moving on, and so forth. A savvy designer won’t dedicate too much to a single asymmetric AI, since they are essentially disposable, but the reward for solving the puzzle should be commensurate with the puzzle’s challenge. 

Symmetrical AI is when the AI essentially acts and functions as another player, but one controlled by the computer. When the AI is symmetrical, you expect behavior like a CPU opponent in Street Fighter or Mortal Kombat, an AI controlled enemy in a RTS game like Starcraft, or a FPS bot. This is a situation where the AI should feel sensible - the enemy should feel like somebody who is an actual opponent. In such a situation, having a puzzle-like opponent is actually a negative, because the whole purpose of the game is to be competitive. Above all, when fighting a symmetric AI, the player needs to feel that the AI is “fair” - not necessarily that the AI adheres to all the same rules that the player does, but that the AI appears to adhere to all the same rules that the player does. 

This means that the frustration ensues when the player sees the symmetric AI “cheating”. This could be by using an invincible attack move in a fighting game whenever you press an attack button, building units without apparent resource cost, micromanaging units to an extent that is nigh impossible for a human to replicate, or consistently killing with headshots from extremely long distances. These are all things that an AI can do, but tend to be things that players associate as being “humanly” impossible, and thus “unfair”. In this case, it is absolutely imperative that the AI is designed to emulate an interesting (and fallible) human player, rather than simply provide an unbeatable, cheating machine. The central goal of any design is to provide engagement and fun, and a nigh-unbeatable machine that simply does things better than a human is neither.

The big problem happens when designers switch these two categories up without also adjusting the player’s expectation. In the article you referenced, that’s exactly what happened - the AI designers tried to create AI that acted and behaved like sentient opponents, but in a situation where the players were expecting asymmetric AIs. The game itself wasn’t built with the symmetric model in mind - the player played as a larger-than-life hero who fought against many faceless mooks and thugs. All clues pointed to disposable enemies, but they didn’t necessarily act that way. This isn’t to say it couldn’t have worked, but the problem here was mostly that they went directly against the player’s intuitive assumptions about the game. I think it could have worked if the scenario had been changed - perhaps in a series of challenge maps, where the fiction was based around the player fighting an elite team of enemies. This way, the player would have entered knowing what to expect, and gotten that. The real takeaway is to make sure the players know what they’re going to get, and that fighting a player’s intuitive assumptions is a losing battle.