
The Outer Worlds: Fixing the “game thinks my companion is dead” bug - pkilgore
https://twitter.com/_taylorswope/status/1205252714680045568
======
ergothus
The real payoff to me is when someone asks why they didn't just try more ways
of making companions unkillable without/before going through all that
debugging. The response:

> that was our fallback, but

> -without actually knowing the cause, we couldn’t be sure that’d fix it

> -whatever the cause of the problem is might be causing OTHER problems, so we
> should find it

> -games near ship are fragile and it’s best to not make changes you don’t
> 100% understand

_THIS_ is someone that is practicing good software practices. I have no doubt
this is why Outer Worlds is so much less buggy than, say, recent Bethesda
_patches_. I saw a report this morning that the latest Fallout 76 patch causes
some armors to degrade when a weapon reloads. This sounds exactly like the
kind of bug you'd get from "fixing" something without understanding the actual
cause, and thus just kicking up more emergent bugs because of your fix.

Complex interactions (like modern games), filled with shortcuts and
assumptions (like modern games) means some bugs are inevitable. Requiring
rigorous understanding before making changes can keep those bugs from
multiplying.

...says someone with zero experience in coding games, but plenty of experience
both fixing and generating bugs.

[Edit: fix formatting]

~~~
mattmanser
I don't buy this at all.

This is the antithesis of defensive programming. It's a game-breaking bug
that's allowed to happen because of two things:

1\. They allowed companions to take damage because they marked them with the
wrong property

2\. They never checked if the companions went somewhere silly to warp them
back to where they should be

That's not defensive, that's like calling an API without a defensive try/catch
if something goes unexpectedly wrong.

If they'd have done either of these things (and sent back error reports when
it happened), they'd have caught it way sooner.

I don't write games and appreciate the interactions can be very complex. While
I can understand your delight in them finding the root cause, I can't
understand your support for failing to put in some defensive measures when
they couldn't recreate a frequently reported bug.

~~~
ergothus
> I can't understand your support for failing to put in some defensive
> measures when they couldn't recreate a frequently reported bug

From the tweet series: "There were one or two cases before launch where this
issue seemed to happen, but no one in QA ever managed to reproduce it and
despite our best efforts we couldn't learn anything concrete about it"

Assuming this is true, before launch, it was NOT a frequently reported bug.

After launch, they dove in and figured out what was happening AND what
triggered it.

Your complaint appears to be that they (1) had a bug, and (2) didn't resolve
boundary checking in the production code. Given the number of "fall through
the map" bugs in countless games, I assume #2 is, in fact, hard. Writing non-
trivial code without bugs (to avoid #1) is also hard, based on...all the smart
coders I've read.

Regardless of the source of the bugs, I'm cheering that they didn't just put
in defensive measures and called it a day. To the point that defensive
measures can be a source of bugs, consider how many "violent vibration" bugs
we've seen in, say, Skyrim, Fallout, and GTA. (While I don't play, based on
clips I've seen from GTA and Red Dead, those engine(s) spam duplicate entities
when the boundary overlap is detected, where Skyrim/Fallout just vibrate and
rattle). I find it very plausible that attempts to "warp" characters having
positional bugs without understanding the source of those bugs will end up a
source of new bugs.

~~~
wruza
>Given the number of "fall through the map" bugs in countless games, I assume
#2 is, in fact, hard.

Can’t gamedevs reinvent solidness and make underground/walls solid?
Viscosity/density/archimedes?

~~~
natpat
There's lot of great ways to ensure your collision detection is perfect. Sadly
none of them run at 60fps.

------
Danieru
This thread is speaking to how regular programing and game play programming
vastly differ.

In regular programming scopes are small, small enough for programmers to
accomodate all expected logic. In games interaction scope is unlimited, or
atleast that is what the game designers want.

If you try to program every expected system interaction you would never
finish. So instead game programming is about making systems flexible. For
example, as far as the gameplay code is concerned the inside of players ship
is just yet another level. The bounds of any level are defined by what the map
designers have layed out, not any pre-planned "ship area".

There are no meetings where project managers request "more playable area for
level 7".

Likewise the systems programmer who made the furniture system was not the
script writer who created the ladders. The script writer likely thought it was
brilliant and elegant to split the entrances and exits, because in game
programming it was indeed brilliant. It allowed the save system's existing
code to handling what would have otherwise been special handling. It allowed
the existing combat logic to handling what would otherwise be special
handling.

Of course regular programming is different. In regular applications if a
user's data is being sent to a server there is no secondary system which needs
to be allowed to kill this data. Nor is there any save system which needs to
be able to save and restore an in-progress action.

Games programming is all about figuring out ways to allow more varied and more
complex interactions without exploding the implementation exponentially.

~~~
konschubert
This was an extreme insightful comment.

Originally, when I read the article, I thought that the issues described may
be a result of bad software design.

But maybe it’s because applications are about strictness, and games are about
freedom.

In application programming, you should be in a defined state most of the time,
and make state transitions short and atomic.

In guess that in games, you are really in one huge, never-ending state
transition.

~~~
WORLD_ENDS_SOON
I would also add to this that in my opinion often the best games (from a
design perspective) are the ones that lean into this sort of freedom. This
line of thinking has very much influenced game design trends, at least within
certain genres. For example, simulation and survival games usually try to
expose the game's systems to the player in a way that lets the player be
creative in the game (Minecraft, Prison Architect, RimWorld, Factorio, etc).
Even in games that aren't based on simulation or user generated content,
designing the game's systems in this way gives the level creators freedom to
do more without having to write code specific for every level.

------
choeger
Brilliant. It shows how your code is never encapsulated enough. You _think_
physics should apply to all agents in your game and you _think_ climbing a
ladder is trivial and then someone _thinks_ they can easily stop all NPCs from
starting new Interactions. Boom.

In that particular case, what would be your favorite fix. I tend to say that
the second step of climbing a ladder should probably not count as a _new_
interaction.

~~~
NoodleIncident
I don't know if "encapsulation" is the right word, but the "undamageable"
state should definitely have included fall damage. No point in having more
than one way to reduce health points, damage is damage.

~~~
vvanders
I think you're giving a bit too much credit to game logic codebases.

They generally move at a pretty fast pace and change often(due to the
iterative nature of what's "fun"). I don't think I saw a single unit test
until I was well out of the game industry.

~~~
Filligree
My go-to exception for this has always been Factorio. They have not only unit
tests, but integration tests, regression tests, fuzzers, performance
regression tests, ....

Their blog regularly goes into details about how it all works. It's very much
worth reading.

~~~
hhmc
Sure, but you have to acknowledge that factorio is incredibly amenable to
those types of tests -- in a way that most games aren't.

~~~
worldsayshi
It seems to me if the physics engine and whatnot is deterministic I don't see
why it wouldn't be possible to record input sequences and keep those and their
effects as regression tests.

~~~
Fargren
Those tests can be written, but if you do it naively coverage is measurable
and very small. And the physics engine is not necessarily deterministic (or if
it is, you can't always control all it's parameters)

------
kevin_thibedeau
I had the same problem in a hardware product using a legacy wired protocol
that was sent over a ZigBee connection. People would complain that volume
would uncontrollably go to 100% when operating a remote.

Reproducing it was hard but the culprit was WiFi interference. The protocol
had separate on/off commands and no provision for lost packets since it wasn't
natively designed for wireless. If the off command dropped out everything
would act as if a button was still pressed.

~~~
AceJohnny2
stateful volume control...

>_<

~~~
CGamesPlay
Probably the design constraints here make this pretty necessary: you want the
user to be able to hold the button down to adjust the volume by large amounts
and release when it gets to a comfortable level; sending individual volume
packets this quickly causes too much traffic over the slow connection; and the
remote doesn't have any way to receive packets from the controlled device so
it can't just send absolute volume levels.

~~~
pshc
You could send another volume-up packet every say 200ms. No way that would
congest a local wireless connection.

~~~
CGamesPlay
This might not be the actual constraint this system had:

> I had the same problem in a hardware product using a legacy wired protocol
> that was sent over a ZigBee connection.

~~~
pshc
Oh _wired_ , wow, I had to read that five times to understand it fully. That
is an unreconcilable situation, ouch.

------
fernandopj
That'a well-worth read to everyone that enjoy reading about QA finding bugs,
even if not interested in games Finding quest-related bugs is a nightmare in
open-ended games...

------
theshrike79
Readable format:
[https://threadreaderapp.com/thread/1205252714680045568.html](https://threadreaderapp.com/thread/1205252714680045568.html)

------
gambiting
So I work in games as well, and one thing you learn really quickly is that
this "one weird bug that QA has seen once and no one has been ever able to
reproduce or even come up with an idea of how it could happen" will be
reported thousands of times and cause uproar on forums once you sell 10+
million copies. It's just a statistical inevitably.

~~~
loeg
Ditto in enterprise, but that doesn't mean you're ever able to reproduce it in
house before it ships. You do the best you can when you have a signal to
follow. And there are probably other open bugs that impact more customers that
deserve higher priority anyway.

------
viraptor
For all the flak people usually give the telemetry systems, this bug sounds
like it could be made trivial if devs could push custom reporting to users.
"When the companion dies, send me last few events and a minimal world-state".
Having ready data collection/reporting when things go weird is super helpful,
whether it's an own, server side software, or a remote client.

~~~
aaronbrethorst
It doesn't sound like that would've worked here.

 _One reason it was so hard to pin down is that it was impossible to tell when
the bug actually happened -- all of the cases we had were essentially "hey
something bad happened in the last ten hours and now my quest is broken"
(5/18)_

~~~
cjbprime
But they knew that the proximate cause of the quest failure was a dead
companion, so adding logging around companion death would have figured it out.

~~~
aneutron
You generally do not enable the level of logging that I imagine would be
necessary to find this bug, on the enduser version of the software. I could be
wrong.

~~~
girvo
I wonder if that level of logging would have a performance impact, especially
on consoles?

~~~
wyldfire
"logging"/telemetry can be flight-data-recorder style like XRay uses [1]. This
can be limited to a smaller window because it writes to a circular buffer,
then again you could fit a lot of entries in a ~512MB buffer.

[1] [https://llvm.org/docs/XRay.html#flight-data-recorder-
mode](https://llvm.org/docs/XRay.html#flight-data-recorder-mode)

------
lordleft
CRPGs are my favorite genre of video game, and the complex interactions they
feature make them both super fun and especially prone to glitches.

I will say that TOW was very stable for an Obsidian Game. I don't think I've
experienced a bug yet.

------
nitwit005
Some may remember Todd Howard's Quakecon interview where he talked about
Bethesda failing to implement ladders due to AI problems:
[https://www.ign.com/articles/2010/08/14/why-there-are-no-
lad...](https://www.ign.com/articles/2010/08/14/why-there-are-no-ladders-in-
fallout)

------
AdmiralAsshat
Should've called this bug "Stairway to Heaven".

~~~
iscrewyou
But it ruins the story since that’s the punchline.

------
pkilgore
Don't skip the gif at the end!

------
remarkEon
Funny read.

The Outer Worlds is an unexpected joy, too. Definitely an open-world game I
had no idea I needed. If you haven’t, check it out. It’s basically Fallout 4
but in space.

~~~
thrownblown
...basically Fallout New Vegas in space... (obsidian made that)

~~~
remarkEon
Ah, yes. This is the better comparison.

------
oniTony
This reads a lot like the central premise in "You" by Austin Grossman --
unkillable NPCs are occasionally found to be dead and that bug is threatening
the entire game studio.

------
MattyRad
Only tangentially related, sorry: all of the the companions in The Outer
Worlds were uninteresting, I found it much more satisfying (and challenging)
to beat the game solo. Effectively, each companion was an ordinary human, with
some arbitrary specialities, joining your crew because the current mission
required it. That said, The Outer Worlds was a lot of fun (albeit short-
lived).

~~~
francislavoie
I wholeheartedly disagree. I loved the companion missions and the dive into
the characters and their psychology. They all were interesting in different
ways.

Notably, I found that the Vicar had a really interesting progression as a
character and his relationship with religion. It lead to some interesting
philosophical dialogue, especially if you bring him when you meet the
Iconoclasts. He has a healthy debate with their leader and they both come to
the conclusion "let's agree to disagree". It was great.

------
doublement
I don't play video games, but if someone created one that was _entirely_
glitches like this, I might start.

~~~
drewrv
Check out Goat Simulator:
[https://en.wikipedia.org/wiki/Goat_Simulator](https://en.wikipedia.org/wiki/Goat_Simulator)

"The game, initially developed as a joke prototype from an internal game jam
and shown in an early alpha state in YouTube videos, was met with excitement
and attention, prompting the studio to build the game into a releasable state
while still retaining various non-breaking bugs and glitches to maintain the
game's entertainment value."

~~~
Groxx
It's full of awkward physics, I sort of love it. E.g. goats go up ladders by
walking straight into them and continuing to walk with absolutely no other
changes... so of course the body physics get a bit excited
[http://i.imgur.com/lNfzaDj.gif](http://i.imgur.com/lNfzaDj.gif)

------
squar1sm
Gamedev is full of interesting problems. The industry scares me because of a
few friends. Fun to hack on, on the side and I’m jealous of the story and
mystery this team got to chew on for a while. I know it’s hard but still
jealous of these gems.

------
floatingatoll
This is an excellent post-mortem, thanks for posting it.

------
bitwize
I imagined the MGS3 theme softly playing as the companion character climbed an
invisible ladder off the map and into infinity.

------
lowbloodsugar
If anyone is reading, can you guys fix the "There is no new game+" bug please?
=)

------
db48x
Their QA should have been using a record-and-replay debugger. rr-project.org

~~~
alex7o
I thought the same, but there might be a good reason they can't.

~~~
db48x
Yea, using rr would certainly require that they test on Linux. That doesn't
seem like a very high hurdle though.

------
bipolar_lisper
why is the most upvoted thing on hackernews right now a story about a bug that
came about as the result of bad code

> The only logical culprit was a bit of scripting that runs when a companion's
> health reaches zero: if they're in the party, it waits for combat to end and
> revives them; otherwise it marks them as dead "for real

The code should never have been written this way if the companion is not
supposed to die. There should have been an `is_dead()` function that always
returned false.

This site has declined dramatically and I think it's a result of the quality
of programmers we have today vs what we had 10 years ago before the internet
became mainstream.

None of these comments nor the tweets talk about how bad that code was, but
instead kiss the guy's ass.

I expect to see more and more posts about games and other trivialities and
less and less posts about important topics as the years fly by.

------
INTPenis
I haven't bought this game but I watched a 12 minute speedrun of it with two
devs commentating and got the feeling that QA didn't perform as well as they
could have on this project.

First of all the fact that the game could be finished in 12 minutes was a
surprise to both devs.

And at several points during the video dev#1 expressed surprise over a
shortcut the player was taking, while dev#2 said he did that in testing all
the time.

Seems like a lack of both QA and communication.

Projects are often rushed to completion for shipping, and open world games
can't be the easiest thing to test. You're balancing player freedom and bug
hunting. It's a challenge I can only imagine.

~~~
schoen
On the other hand, if you watch speedrun records on YouTube or watch the
Summoning Salt documentaries about them, expert speedruns (especially those
that allow glitches) are often 10% or less the time that an ordinarily skilled
player might take to finish the game. For example, the Super Mario Bros.
record is under five minutes, while the Castlevania record is under 11.5
minutes.

I don't think the fact that developers are surprised by speedrun times means
that a game is necessarily bad! (And even many of the most extraordinarily
beloved games have glitches, often because realistic physics simulation is
hard.)

I think I saw a video of Bennett Foddy in which he noted that other people
could complete "Getting Over It with Bennett Foddy" considerably faster than
he could, and maybe faster than he imagined anyone would be able to.

~~~
bipolar_lisper
programming quality and program quality are two very separate things. a great
game can have terrible programming. and an awful game can have exceptional
programming.

