
Ask HN: Game devs, what is the hardest part about designing an AI for your game? - sgillen
I&#x27;ve been thinking a lot about different machine learning techniques applied to video game artificial intelligence. I&#x27;m wondering if any video game devs would like to share their experience about what parts of making a game AI are challenging, things that are difficult to get right, time consuming, or tedious.<p>Thanks!
======
Jasper_
Game AI is mostly pathfinding (which basically falls down to creating
navigation meshes for levels), and very basic decision making skills, which is
usually a weighted decision tree (e.g. if they're getting shot at, cover might
be weighted at 80%, shooting back at 20%). You don't want to actually create
something _actually_ smart to fight, you want something with understandable,
predictable logic for the player and something they feel clever outsmarting.
Anything too complex and players will get discouraged by the difficulty.

Machine learning isn't really a great fit for this, as it is neither
predictable nor understandable, and thus the player can't plan around it or
try to outsmart it.

~~~
dx87
That reminds me of a lesson developer's learned when making a stealth game,
Splinter Cell. Originally, when you were spotted the enemies would just try to
shoot you, but tester feedback said that players didn't like getting shot out
of nowhere. The devs added audible cues that you'd been found, like the guard
yelling "Hey you!" before they started shooting. It turns out realistic AI
behaviors aren't necessarily fun AI behaviors in a video game.

~~~
blattimwind
Splinter Cell also has cooldowns in some/most levels (like most stealth
games), which is of course an absurd notion. A guard on a military base
wouldn't back out of "DEFCON1" after two minutes just because the invader
didn't show herself for that long.

~~~
mcv
There's a cool isometric stealth game called Robin of Sherwood, where you can
follow what the guards are doing. A guard might accidentally stumble upon
someone you killed, be alarmed, look around, run off to warn their commander,
then the commander organises a search for the culprit, with lots of guards
explicitly looking for you everywhere. Eventually if they can't find you, the
whole thing dies down and everybody resumes their original duties, assuming
the dead body was just a fluke or something.

Also, if they find someone you merely knocked unconscious, they assume they'd
been sleeping on the job. If you tied them up, they free them, but I forgot if
they search for you or if they just assume they got accidentally tied up
somehow.

Also, you'd think they might raise the alarm if they find less and less guards
present at their assigned post.

------
krm01
I’ve been designing/coding games since I was 11yo and have a degree in
gamedesign & dev. The hardest part isn’t the technical side of the game’s AI.
It’s finding the balance of an AI that makes the gamer feel powerful enough to
be fun- but not too powerful to become bored - and not find it too hard that
he/she becomes demotivated. Obviously every player is different so finding a
balance that can keep the gamer excited enough to continue is extremely hard.
Getting the emotional-experience side of the player into the sweet zone is
hard. One would say ‘why not make the AI such that it adapts to the user’.
Which is 50% of the answer. But the challenge then becomes that you don’t want
good players to be at the same level as bad players when the AI creates a
similar experience for both. You want both players to enjoy the game, but not
in the same way.

~~~
Anil_Uttani
The Solution here would be rather than weighing the AI Difficulty for good
players and bad players.You must have different sets of AI.

------
azhenley
Making an AI that isn't "perfect", but rather it is fun and challenging to
play against.

My friend was working on a chess variant for years, and it was very easy for
him to make the AI be very, very difficult but that isn't fun at all! It would
always win.

He then tried adding an element of randomness to it to dumb it down. But then
the AI just looked perfect most of the time while occasionally making
bizarrely dumb moves.

He later added "personalities" to the AI, where each one would try their own
sets of strategies and attributes. He continued to make it better and better.

~~~
duggalr2
I'm curious if your friend's project is open-source at all, would love to see
the implementation details of this?

~~~
azhenley
It isn't open source, but you can still play it if you have Flash installed:
[http://www.kongregate.com/games/thegrandestine/chess-
evolved...](http://www.kongregate.com/games/thegrandestine/chess-evolved-
online)

Imagine chess but with 100+ different pieces and an element of
collecting/upgrading pieces.

~~~
duggalr2
Cool, thanks!

------
springogeek
The risk of overcomplicating. You can put a lot of effort into detailed game
AI which just makes for a hugely unfun experience. Something simple or
predictable often leads to be better gameplay.

------
formalsystem
I think any game with strategy is going to have to use reinforcement learning
to have any hopes of being understood by its players. Concerns around cheesing
strategies can be mitigated for real time games using constraints and cheesing
in turn based strategy games is a symptom that maybe the game you're working
on isn't that strategic.

Ultimately Chess, Go, Dota, Starcraft are compelling games to watch and play
because the skill cap is so high. No hand coded AI for such a game could ever
prepare you to enjoy playing any of those games professionally or get to a
point where you can understand professional play. High level strategy in those
games would never even be discovered had we not had a large population of
players playing those games for long hours.

Odds are most strategy games won't be as successful as any of the above games
but the newcomers could still have compelling strategic elements that gamers
could explore had they been able to find other players that are slightly
better than they are so they learn but at the same time aren't frustrated by
too big a skill gap. A small multiplayer community is why the strategy game
market is pretty bleak except for a few large players.

Reinforcement learning can provide a better experience to your players because
you the game developer don't need be the world expert at the strategies of
your game to create a top level AI. You just make sure the rules of the game
are there and self play will take care of the rest. You can constrain the AI
such that it's enjoyable by both your newest and most devoted players. You can
discover OP strategies and better guide your design process. Excel is great
for balancing but if you want to make a game complex enough then balancing
will be a tremendous challenge without a larger team.

If you're interested in leveraging reinforcement learning for your game, I run
yuri.ai which helps game developers do exactly that. So reach out to me at
mark@yuri.ai if I can help!

~~~
sgillen
Thanks for your response, I had actually heard about Yuri before and was
hoping you would respond to this post!

I have a lot of questions for you if you don't mind, maybe you don't want to
answer any/all of them that's fine.

1\. How do you handle constraining the AI such that it's enjoyable? If you
read through the comments here you'll notice just about everyone is saying the
hard part is not making a good AI but in making a fun AI. Do you know of any
research in this area? most examples I see are all about making the best AI
possible. It seems like it's really hard to find a good reward signal to use
for "fun". Have you seen this [0,1] paper on learning from human preferences?
does Yuri attempt to do something similar? It seems like it would be labor
intensive for the play testers.

2\. Does Yuri make use of imitation learning at all? I.E learning from lots of
human data when it's available to bootstrap the learning process?

3\. Do you let the game designers impose any structure on the AI? I.E. "I want
three phases for the boss where he progressively gets more aggressive" or use
RL just for certain small parts of the behavior? A well defined path finding
algorithm with the RL deciding where the agent tries to go? stuff like that?

I have a ton of questions about the technical details side of Yuri too, what
frameworks are you using, which RL algorithms etc etc, not sure if you are
willing to share those or not.

[0] [https://arxiv.org/abs/1706.03741](https://arxiv.org/abs/1706.03741) [1]
[https://blog.openai.com/deep-reinforcement-learning-from-
hum...](https://blog.openai.com/deep-reinforcement-learning-from-human-
preferences/)

~~~
formalsystem
1\. We use a couple of tricks for this, there's ideas from human preferences.
There's modeling a reward function to maximize fun instead of pure
performance, there's handicaps and a host of other tricks. Different tricks
work better for different games but game developers can easily configure them

2\. It can, imitation learning turns RL into a supervised learning problem
though from my experience it's not really needed. It's more of a if you
already have it then you can make the training a bit faster

3\. Yes all the RL part becomes tuning a config file where the game developer
needs to describe the task it wants to be accomplished

Regarding technical details, the infra is agnostic to the specific deep
learning library you'd like to use and the algorithms are chosen based on the
types of inputs (continuous vs discrete), size of the space (the end all of
strategy games or a casual puzzle game) and type of game (full information vs
partial information).

------
cpeterso
GDC (Game Developers Conference) publishes a lot of great talks on YouTube
about game design. Here are some of their popular AI talks:

[https://www.youtube.com/results?search_query=GDC+AI](https://www.youtube.com/results?search_query=GDC+AI)

------
danbolt
I programmed some new API functions for the AI scripting language in a popular
RTS game once. I think the most tedious part is exposing the game data in an
intuitive/helpful way for someone in a design role. It's a bit of an odd mix
of optimization, computational geometry, and finding the right level of
expressiveness.

For instance, a designer might say "It'd be great if I could put a tower
between myself and the enemy town, but kind of at the border of mine, but only
if I don't have enough money for a cool castle and I'm not overwhelmed."
There's a lot of fuzzy logic in statements like those. The API between the
game and the "AI" usually has to be general enough not to need spatial
information, but also specific enough not to do things that feel unintuitive
or silly to the player.

It's worth noting that machine learning helps avoid a lot of these issues, but
I find AI tends to be designed for an feeling or experience, so I'm not sure
if things like the Starcraft AI competition bots tend to make better products.
Perhaps if ML could make the building blocks for designers to design AI?

------
dleslie
F.E.A.R. remains at the top of its class for the feel of the NPC behaviour,
and it's worth understanding why.

[https://aiandgames.com/facing-your-fear/](https://aiandgames.com/facing-your-
fear/)

------
mrfusion
I’d love to do this as a career or sell Premade AI modules for games. Any
ideas?

Would game developers consider contracting out the AI to a freelancer?

------
cableshaft
No matter how dumb you make the A.I., you will have players complaining that
it's too hard or unfair. I literally had the computer make a list of all
possible moves and pick one out of the list at random for the default A.I. of
one of my games and I got accusations that the computer was cheating on that
difficulty level on a fairly regular basis.

------
ggggtez
Watch the talk about Civilization's AI. Seriously, game AI does not need to
use advanced ML techniques, and probably shouldn't if you want players to have
fun. The trick is coming up with fast algorithms, with a semblance of
personality to make it interesting. Difficulty is _not_ required for the AI to
be fun in most cases.

~~~
bryondowd
What talk is that? Last time I tried a Civ game, I was underwhelmed. That AI
just isn't fun to play against, to me. And the games run too long to play
against humans in a reasonable timeframe on normal settings. I'd love to play
Civ with competitive AIs.

~~~
ggggtez
"Playing to Lose" :
[https://www.youtube.com/watch?v=IJcuQQ1eWWI](https://www.youtube.com/watch?v=IJcuQQ1eWWI)

Personal preferences aside, they specifically talk about how to make the game
"fun". For example, they try to account for things like real-world leader's
personalities. "Gandhi" shouldn't just attack his neighbors unprovoked, but
maybe "Caesar" should.

And why do the AIs even bother to ally/trade/help the player at all, since the
player is the only enemy that really matters anyway? Because the player is
also the only person that matters in terms of having fun, and making the game
too hard isn't any fun.

~~~
bryondowd
Interesting talk, but it really pretty much confirmed a lot of my assumptions
about Civ AI. He pretty much admits that they are constrained on developer
resources and AI code only makes of a small percentage of the game. Because of
that, they have to make do with very naive assumptions about strategy. And
because of that, they have to cheat to get the AI up to a level that's even
considered fun.

At first I thought he seemed to be making a lot of false dichotomies, but
really, he was saying, these are my only choices, given the overarching
constraint on how much effort we can put into our AI. Basically, all of the AI
decisions he talked about were small scale heuristics, with no ability to
really look at the bigger picture, which is what the player is always doing
when making small decisions.

And of course, personalities are a great idea, and not mutually exclusive with
a strong AI. One thing I very much disagreed with was when he said an AI can
roleplay, but a player wouldn't do that, the player just plays to win at all
costs. But earlier on he even talked about one of the player types is people
who play for a narrative. Those are roleplayers. So, especially at lower
difficulty levels, it makes a lot of sense for the AI personalities to trump
optimal play. But that doesn't mean you can't have a strong AI in general,
that makes some limited sacrifices for the sake of fun.

Likewise with diplomacy. It isn't about making the game as difficult as
possible. If that were the goal, you could just make the player lose when he
loaded the game. It's about making it feel like you're on roughly even footing
with your opponents, so that success has meaning. To that end, I would expect
each empire to act in its own best interest, and therefore treat the player
the same as it would any other opponent. It's also why diplomatic options need
game mechanics to enforce them. As he pointed out, if there's no mechanic to
stop a player from accepting money for peace, and then declaring war again,
then there is no reason to every pay for peace. You should design these
mechanics with player behavior in mind, and the AI should use them the same
way players do. Think about how players behave with one another, make that
fun, and then make the AI behave similarly. This wasn't possible given their
constraints, but it's certainly possible in the more general sense.

As far as "making the game too hard isn't any fun" that's the problem. Too
hard is subjective. What's too hard for one person is perfect for another, and
way too easy for yet another. If anything, to please the maximum number of
people, you'd have to write (or train) a very skilled AI that can compete with
the brightest players. If you can do that, it should be much easier to tone it
down or penalize it to make easier difficulties for the average and casual
players.

------
mattnewport
I've never specialized in AI but from my experience on game teams most of the
challenges revolve around making the AI 'fun' to play against, which is not
generally very well defined. It's also about taking feedback from designers
and playtesters and figuring out how to adjust the AI to address that
feedback. Machine learning techniques aren't really used much in game AI,
partly due to these aspects. It's not necessarily obvious how to adjust some
black box ML model in response to feedback from playtesters on balance issues
or from designers on making the AI more or less 'agressive' or other
qualitative feedback. Usually it's more a collection of rules and heuristics
with various parameters that can be adjusted to tune the behavior and allow
for different difficulty levels etc.

------
ezekg
I always thought it would be an interesting to write an AI for a shooter like
Call of Duty or Halo, which is trained on real player data, according to your
skill level. Most FPS games already record match data, so the dataset is
available. What might happen is that the AI might evolve as the player meta
evolves, and would learn what types of maneuvers work in certain situations,
it would know average accuracy for a skill level to adjust itself, etc. It
would also allow the game to fill in low player population, e.g. during a low
traffic time or for a player with a ping that is too high, with bots that act
human-like. Maybe wishful thinking, but I always thought it was an interesting
thought.

------
bladedtoys
Like others have said, balancing play-ability is hard. Another problem is
users may perceive the AI cheats which hurts fun.

But I'd say another hard thing is that every game needs a different approach
to AI. Some need path finding, which is well documented. Others need resource
balancing which is less so. The newer and rarer the game mechanic, the more
novel the AI.

I would say Machine Learning is not useful except in a handful of games and if
yours is not in that category, you risk spending a lot of time for little pay
off. And even if in those categories, ML will be working to win not to
maximize fun so they could end up being too good or to cheaty looking.

------
setr
One big issue I can imagine with applying ML is that most games are buggy and
poorly balanced, but the majority of your players arent optimizing machines
(outside of speedrunners), and will follow “natural” paths, so its fine.

The only kinds of games that see enough time and resources to the same issues
that would trap an AI in some optimal but horribly boring strategy (or rather,
“unfair” strategy) would be competitive multiplayer games — and even then,
only those that can classify for e-sports. That is, those games who already
treat players (or rather, the wetware community optimization machine) the same
as they would treat your ML AI.

------
slimsag
I've been thinking about this a lot, too!

Honestly, I think the most challenging factor is that when you are working on
machine learning for your game.... you aren't really working on your game.
i.e., you're not writing storylines, creating visuals, audio, programming
general interaction, etc. Machine learning (if you don't have a ton of
experience) can be super time consuming and unless your game is 100% based
around the sole idea that it is ML.. all of that time is taken away from the
rest of your game.

~~~
sgillen
Yes I agree, I think the idea here is to provide applied machine learning as a
service to game devs.

------
tareqak
I actually think there would be value in having machine-learning / AI in a
game that would dynamically try to match the human player's skill without
overwhelming the human player. The goal would be to maintain parity such that
the human's player response would always be "I was this close, and I could
have done X, Y, and Z" as opposed to "I had no chance".

------
adamnemecek
A friend is working on a solution for this. Check it out
[https://yuri.ai](https://yuri.ai)

~~~
coolreader18
Why is it named Yuri, if I may ask?

~~~
adamnemecek
After Yuri from command and conquer.

------
jamesgeck0
IIRC, the developers of Final Fantasy XII said in an interview that the Gambit
system that the player uses to set the behavior of their computer controlled
party members is basically just a GUI representing the same logical structure
used behind the scenes for enemy behavior.

~~~
thedevindevops
I love that Gambit system and wish more games would adopt similar systems. The
Rules based approach allows complex behavior to emerge from a relatively
simple set of rules. Although this is a per-unit approach, in combination with
an overarching management AI, this could be very engaging.

------
z3t4
What I understand from machine learning is that the actual learning takes a
lot of compute power and learning data. So you can't have and AI that adapts
during gameplay. The AI will "just know the optimal move for any given play
state.

~~~
bryondowd
If your game doesn't give competitors perfect knowledge of the game state, it
would make sense for the AI to track previous activity of its current opponent
as part of the 'play state' for making decisions. For instance, if it doesn't
know where the enemy is, but the enemy has attacked from this direction every
time so far, it would prepare for another attack from that direction. Or if
the last time the AI made a given offensive move, the player countered
effectively a certain way, it may modify its next offensive to overcome that
counter. The longer the game, the more it could appear to be learning to
counter its opponent, even though, as you say, it's just making the optimal
move given the current state of play.

------
ionwake
Stopping it from making a base an Antarctica which players can’t invade due to
a bug

------
sjg007
My guess is you want it challenging but not too hard.

------
EtherDynamics
_Mixing Concepts_

The biggest surprise I have found on this subject is this strange entanglement
of ideas between "AI designed to kill the player" ("Military" AI), and "AI
designed to make the game fun ("Game" AI). This is like comparing a F-22
stealth fighter to a 747 commercial airliner -- sure, both are technically
"jets", but they serve ___completely_ __different functions.

The only AI we should be talking about here is AI designed to make a game more
engaging and enjoyable for the player. That may include __emulating __a __tiny
__bit of behavior that __resembles __military tactics, but only where games
use that as a topical facade -- unless you play The Sims _waaaay_ differently
than I do.

 _Easy Mode_

There are several posts here where people assert that "they don't want a game
to be hard, that would be bad". As a counterexample, I would invite these
individuals to view the awards, testimonials, and revenue from the
["Souls"]([https://en.wikipedia.org/wiki/Souls_(series)](https://en.wikipedia.org/wiki/Souls_\(series\)))
series (Demon's Souls / Dark Souls). I have spoken to psychology experts at
game development conferences, as well as professors that explicitly study
psychology in games -- and they are still struggling to define and quantify
the exact elements that came together to make this series such a hit. But
there is no denying that it is an absolute favorite, with a huge following,
and tremendous profits to prove the efficacy of the model.

Obviously, I'm not advocating that "every game should have a Souls-style
difficulty curve". That series caters to a specific audience -- and there are
an ocean of subtle differences between fans of the series. The point is: broad
statements like that go directly against a mountain of evidence. We need to
think about this issue with its full (tremendous!) complexity, and how to
leverage some immense computing power to address it. Speaking of which...

 _Based on your Purchase History..._

Right now, companies like Amazon and Google are mashing up the entire history
of everything you've purchased, every web page you've ever visited, along with
demographic and social media data about you. They can then deliver a product
recommendation / advertisement / etc. within milliseconds that targets _your_
specific needs, ___even for things you didn 't realize you wanted / needed
yet_ __.

If that can be done in milliseconds, then the same principles can be applied
to tweaking the difficulty for players in real time. You don't even need to
maintain a hot internet connection to constantly feed some huge cloud
implementation at all times -- some local processing combined with daily or
weekly micro-updates can do the trick. For a perfect example of local / cloud
balance, check out SnapChat -- that thing uses the local computing power on
your _phone_ to identify your face, and then attach various 3D objects and
effects to it __in real time __. That 's done at 20-30 FPS on a smartphone
_without_ any special dots or markers to help "anchor" the models. That's
amazing. And it's all possible because the [Convolutional Neural
Network]([https://en.wikipedia.org/wiki/Convolutional_neural_network](https://en.wikipedia.org/wiki/Convolutional_neural_network))
models that make that possible get trained on the cloud and downloaded every
now and then.

So, yes -- you _can_ code a game to dynamically customize itself for every
single player. This is the [exact point of my video
series]([https://www.youtube.com/watch?v=hTDSGU2vAow](https://www.youtube.com/watch?v=hTDSGU2vAow))
about this subject. Thus far I've made over a half-dozen Skyrim mods that
demonstrate some of these concepts, and our team is expanding into other games
that are finally opening up their engines to 3rd parties. If folks would like
to talk about the nitty-gritty of all that, I'm happy to chat.

------
expopinions
None of those (e.g. lighting, physics, graphics) are truly difficult today
with the advent of modern game tools (e.g Unity). Assuming you know basic
vector math and can think in 3 dimensions, most problems related to sound,
physics, graphics etc can be solved by some Googling.

