
Confessions of an Unreal Engine 4 engineering firefighter - lfowles
https://allarsblog.com/2018/03/16/confessions-of-an-unreal-engine-4-engineering-firefighter/
======
martin_ky
I'm sure this article resonated with many engineers, not just from the game
industry. What resonated with me the most, was the part about hiring a
consultant, only to tell management what their employees have been telling
them all along (for a ridiculous hourly rate of course). The simplified
executive line of reasoning, why to rather trust an external entity over their
own people, goes usually along these lines: "We do not take advice from our
employees. We hired them for grunt work and we only expect them to follow
orders. What could they possibly know that could benefit our high-level
decision making. Afterall we pay them so little".

I consider the case, which the author describes, where the consultant actually
talks to engineers, a happy day scenario. I was witness to a fast growing
startup hiring a well known global consulting firm to help them set up company
structure and processes. The startup was unable to do this on its own because
of lack of leadership and unwillingness of the owners to empower their own
people. The consulting firm sent over a couple of their junior consultants who
consulted their 3-ring binder of best practices for the nearest matching
industry category and gave their recommendations, which were followed quite
strictly by a newly hired junior management. All without talking to a single
engineer.

More often than not, employees are considered a necessary means to an end and
treated as such, with the least amount of reward and care for which they are
willing to work. Their sense of professional honor, responsibility and guilt
is often exploited in an attempt to achieve higher output. Does the company's
products or services suffer because of this? Of course, but nobody at the top
cares as long as there is a metric that tells a good story. And when not, the
lowest ranks are the first to be pruned and restructured.

~~~
direfungasaur
> The simplified executive line of reasoning, why to rather trust an external
> entity over their own people, goes usually along these lines: "We do not
> take advice from our employees. We hired them for grunt work and we only
> expect them to follow orders. What could they possibly know that could
> benefit our high-level decision making. Afterall we pay them so little".

It's not always that leadership ignores line-engineers just because they
consider them stupid. Even obvious recommendations get tangled into the web of
politics and personal favors. Sometimes the 'obvious solution' is also a play
to get more power or influence for someone you're opposed to. That factors
into how the advice is taken.

Part of the fiction management sells itself is consultants can cut across
those organizational boundaries and find the 'right solution'. Sometimes
having an 'impartial' voice echoing what others have already said is enough to
convince you it's not just someone trying to poach your turf, it's actually
good advice.

Further, our tendencies towards tribalism influence this process. We tend not
to think of other employees as part of the same group - they are devs, or ops,
or product. That builds an 'us vs. them' mentality that's hard to break.
Bringing in an outsider tends to unify the group into us (the company/team)
vs. the outsider (the consultant). That psychology helps us to justify bitter
medicine like our team getting more work (which will help the company).

I tend to have a negative opinion of consultants, but someone pointed this
aspect of group psychology to me in the past and it resonated.

~~~
koffiezet
> Part of the fiction management sells itself is consultants can cut across
> those organizational boundaries and find the 'right solution'.

Being a consultant, and knowing multiple others, I feel that not being part of
the company "tribes", really allows you to more easily cut across those
organizational boundaries you mention.

How many times I've heard of 2 technical divisions complaining about each-
other where the main issue is the lack of communication and simply not knowing
who to talk to in the other "tribe". Consultants however are part of their own
little 'consultant tribe' across company tribe boundaries, who more easily
bond together and talk about stuff like that. This means this group is more
diverse and has more contacts spread throughout the organization, making them
ideal to cut trough the bullshit. You also have the fact that when a company
was happy about a consultant, chances are that when they need someone
somewhere else, they'll hire the same consultant again, dumping them in a
completely different group of people.

Yes this is 100% a social problem, but it's there, and a group of outsiders
will address this more efficiently.

------
georgeecollins
The amazing thing to me is that now people can make games without a bunch of
great engineers. Back in the day when we made Battlezone we had at least four
really amazing engineers (one was a Masters in CS from MIT, one wrote early 3d
workstation code, UCLA, Cal Tech CS degrees). In those days you needed people
like that to just 3D to work in Windows. Direct X helped a lot. Unity and
Unreal have made things we couldn't have dreamed of possible.

~~~
ythn
> Unity and Unreal have made things we couldn't have dreamed of possible.

Such as mountains of bloatware/shovelware template games clogging up every
single online store and platform.

~~~
georgeecollins
There has always been mountains of books and huge libraries of music being
created and we have always managed. Find people you trust to recommend things
to you. I recommend Tom Chick at Quarter to Three.

~~~
zrobotics
I've never understood why there's so much rage about crappy games on steam. I
certainly wouldn't have the gall to charge money for a unity tutorial game,
but I absolutely love some of the insane games people like zachtronics make.
If we want to go back to publishers controlling access, who in their right
mind would approve Schenzen IO? There's a lot of crappy products on EBay as
well, but that doesn't mean the platform is useless.

~~~
krapp
>I've never understood why there's so much rage about crappy games on steam.

It's because there are _so many_ crappy games on Steam that it makes it all
but impossible for indie developers with decent games to be discovered. The
perception that Steam is a cesspool of con-artists and incompetents depresses
the perceived value of any game on the platform not released by a known
publisher, meaning any independent game is more likely to fail merely by being
associated with Steam to begin with. Also because the last time the video game
market was drowning in mediocrity, it crashed.

I mean... when I was in high school I used to doodle Mario levels on my
schoolbook jackets because I was just that kind of nerd, and now I live in the
glorious future of Unity and Godot and Game Maker and C++ and SDL2 if I want
to be a masochist. It's fantastic that anyone can more or less make something
that can at least theoretically be called a "game." But that doesn't mean all
of it deserves to be on the open market.

>If we want to go back to publishers controlling access, who in their right
mind would approve Schenzen IO?

You're right, if it were up to publishers no one would be making anything but
MOBAs and first person shooters and maybe Candy Crush clones. Independent game
development allows niche and non-profitable genres to thrive. However, again,
it's still a problem that the sheer bulk of garbage on these platforms makes
it difficult to _find_ that good content without some kind of gatekeeping.

~~~
pjmlp
Similarly, it is not because one can play a guitar that they are entitled to
turn into musician, or writing something doesn't turn one into a writer.

There is crap in all kinds of artistic work, for some reason devs seem to
think they should be special and get everything for free.

~~~
krapp
> Similarly, it is not because one can play a guitar that they are entitled to
> turn into musician, or writing something doesn't turn one into a writer.

I disagree.. playing a guitar _does_ make one a musician, and writing _does_
make one a writer. What either is not entitled to is success in the
marketplace, but then, no one is, regardless of skill. Once you put your work
into the marketplace, the market decides its value, and your value by proxy.
But for that to work successfully, the marketplace has to efficiently sort for
quality, and one way to enforce quality is having the barriers to entry be
sufficiently high. In practice, that doesn't often work because entrants don't
want to be sorted efficiently, they want to be sorted to the top to maximize
exposure and revenue. It's why SEO exists - to optimize the Googlebot for _you
and you alone_ by any means necessary. Unfortunately, Steam also wants to
maximize the size of its library and revenue, so it has an incentive to
engineer a marketplace that favors quantity over quality.

But even amateurs put in more effort than the developers publishing much of
the garbage onto Steam - using the term "developers" loosely. These are
incoherent rage games or weekend projects, barely functional tech demos and
tutorial games, asset flips, etc. Games that don't even _run._ If the bar for
quality on Steam was merely "amateur" then I would be happy, but it isn't.
There seems to be no bar, and that makes it bad for everyone, except maybe
Valve.

------
lfowles
FTA, a take on entry/mid/senior level distinctions I haven't heard before:

"If you are the guy asking for help from people below, to your side, and above
you, you’re probably an entry-level engineer. If you frequently need support
from those above you (excluding C-level and executive support, that my friends
are a whole new level of hell), there’s a good chance you are a mid-level
engineer. If you find yourself supporting and being supported by those at your
side, you are likely a senior or lead level engineer."

~~~
abritinthebay
I've heard a similar thing but phrased as what frustrates you in you work
most. It's a bit more cynical.

Basically: If you're frustrated by your work... probably junior. If you're
frustrated by your colleagues... probably mid-level. If you're frustrated by
too much management... probably senior. If you're frustrated by too little
management... probably lead.

~~~
josephg
I think about it in relation to how much responsibility each person in the
team is comfortable taking.

If you're responsible to solve tickets, you're probably junior. If you're
responsible for implementing features, mid. If you're responsible for making
sure the whole project will work in time for delivery, senior. And if you're
responsible for making sure your team can deliver the project on time, lead.

------
AndyNemmity
"Usually an outstanding problem or issue is already known by your staff, and
often they have tried to make you aware of this problem, but usually it gets
prioritized into nothingness, or filtered into white noise by your chain of
command where no action gets done."

This is the same as being a consultant. You come on site, you listen to
everyone and make notes of it, and then you make a presentation and plan for
direction.

They pay you way more than their own people, to tell you what your own people
are telling you. It's quite crazy.

~~~
pas
As a consultant, they never change.

~~~
jeremiep
As an employee happy with his current employer, those who can change will and
won't hire consultants in the process.

~~~
pas
Not that kind of consultant. :) And sort of agreed. Though, those who can
change and will, should stop being a problem "soon" anyway.

------
Kapura
As somebody who writes code for games for a living, I found this article
fascinating. Something that was coming up in my mind again and again as I was
reading through it is the question of who has the power to affect the
necessary changes. That seems to be the crux of a lot of the issues; is the
studio leadership wrongminded, or have they put the wrong people at the gate?

The parable of the producer ignoring engineering specialities when assigning
tickets seemed especially gregarious. Why was that person unable to learn,
even after the engineers raised the issue? Why were they still there? Were
they friendly with the higher-ups, or what? Why was there no will to improve
the process until external help had to be brought on?

It also reminds me of stories I had heard from inside Microsoft in the pre-
Nadella era. Tales of power brokers and entrenched lifers who could make
things easy or impossible on a whim. What is the nature of the organisations
that allow people like these to take root?

It's funny how small scale engineering problems are rooted in code and large
scale engineering problems are rooted in people.

~~~
Allar
I think I'mma have to use this quote somewhere, if permitted.

'It's funny how small scale engineering problems are rooted in code and large
scale engineering problems are rooted in people.'

~~~
Kapura
Take it!

------
shoo
tangentially, the "blueprint spaghetti" link is pretty great:

[https://blueprintsfromhell.tumblr.com/](https://blueprintsfromhell.tumblr.com/)

at a former job i knew a team of devs who built and maintained a framework to
help teams of researchers put together pipelines of simulation (often fluid
simulation), animation, rendering, etc. This framework had its own
blueprinting functionality.

One of the cute features they had was nested blueprints -- e.g. you could
define some blueprint, and abstract it behind an interface as its own
component, then re-use the component in another blueprint.

So i guess in a "blueprints from hell" scenario that lets you have "fractal
blueprints from hell".

edit:

i'm also reminded of Chris Domas' defcon talk "Psychological Warfare in
Reverse Engineering", where he sets out to construct binaries that cause
despair when someone tries to reverse engineer them -- in particular he asks
the question: "could i make a binary that draws a particular picture when
someone views the binary's control flow graph in IDA"

[https://www.youtube.com/watch?v=HlUe0TUHOIc](https://www.youtube.com/watch?v=HlUe0TUHOIc)

------
gambler
Good article, but I'm not sure who it is targeted at. Companies/teams that do
bad things the author recommends against (or don't do good things he
recommends) are usually ran by people who either don't care or are
_pathologically_ clueless. They're not going to read this. If they do, they
are not going to change the right thing. They will instead go to some business
conference, listen to overhyped process consultants and force-implement more
"agile" techniques that have nothing to do with what actually hurts their
productivity.

Maybe, strip out most of the game dev stuff and write a small book? That might
help. I don't know why, but books work on managers much better than articles.
At least I can _give_ or recommend someone a book without making it seem like
an insult. "Hey, I've read this book by an engineer who worked on many large
game projects. You should check it out."

~~~
Allar
You're right. I wrote this article knowing this, which is what I tried opening
with.

I'm not qualified to write a book however because this article itself is proof
my writing skills need work.

Otherwise I'd be able to be more clear.

~~~
joshu
Are you kidding? It was a great article.

~~~
daveslash
Agreed - great article. That said, I think most authors of books like the one
we're all imagining have support (in the form of friends and colleagues) in
addition to an editor. I don't want to pressure you to write a book, but
please don't let your own criticism of your writing abilities get in the way
of a book.

------
vvanders
> Overtaxing your engineering effort is a surefire way to end up with hard-to-
> maintain work. Unless you are developing a throwaway prototype, hard-to-
> maintain work is one of the fastest ways to cause engineering fatigue

Sadly this was still all par-for-the-course back when I was in the industry
and all my old contacts still affirm that this is the case today across most
studios. Easier to burn people, ship something and then re-hire/train a new
batch.

It's a shame the incentives of development aren't aligned with reasonable
employment, the industry would be a lot better for it.

------
wyldfire
> They immediately asked who was giving up this information and wanted to
> crack down on this perceived insubordination, rather than trying to address
> the issue. I was then dismissed for not specifying names, and my services
> were no longer required.

Wow! That sounds terrible.

~~~
pjc50
This is horribly routine in all hierarchical organisations across the world;
those that manage to escape from it usually become runaway successes.

------
ajeet_dhaliwal
This was a great article and I recognize most of the issues described but I
feel it doesn’t criticise leadership enough (even though it does somewhat).
This is the biggest failure in the industry. A lot of the best people are too
busy putting in the hard work to be put in leadership roles due to
mismanagement from the current leadership. Why is crunch always a thing in
games dev? It’s poor planning that endures for years and years.

------
hypertexthero
Great article that reminded me of John Cleese’s wonderful talk about
creativity.

> So, here's how to stamp out creativity in the rest of the organization and
> get a bit of respect going.

> One: Allow subordinates no humour, it threatens your self-importance and
> especially your omniscience. Treat all humour as frivolous or subversive.

> Because subversive is, of course, what humour will be in your setup, as it's
> the only way that people can express their opposition, since (if they
> express it openly) you're down on them like a ton of bricks.

> So let's get this clear: blame humour for the resistance that your way of
> working creates. Then you don't have to blame your way of working. This is
> important. And I mean that solemnly. Your dignity is no laughing matter.

> Second: keeping ourselves feeling irreplaceable involves cutting everybody
> else down to size, so don't miss an opportunity to undermine your employees'
> confidence.

> A perfect opportunity comes when you're reviewing work that they've done.
> Use your authority to zero in immediately on all the things you can find
> wrong. Never never balance the negatives with positives, only criticize,
> just as your school teachers did.

> Always remember: praise makes people uppity.

> Third: Demand that people should always be actively doing things. If you
> catch anyone pondering, accuse them of laziness and/or indecision. This is
> to starve employees of thinking time because that leads to creativity and
> insurrection. So demand urgency at all times, use lots of fighting talk and
> war analogies, and establish a permanent atmosphere of stress, of breathless
> anxiety, and crisis.

> In a phrase: keep that mode closed.

> In this way we no-nonsense types can be sure that the tiny, tiny,
> microscopic quantity of creativity in our organization will all be ours!

> But! Let your vigilance slip for one moment, and you could find yourself
> surrounded by happy, enthusiastic, and creative people whom you might never
> be able to completely control ever again!

> So be careful.

[https://www.youtube.com/watch?v=Pb5oIIPO62g](https://www.youtube.com/watch?v=Pb5oIIPO62g)

[https://www.brainpickings.org/2012/04/12/john-cleese-on-
crea...](https://www.brainpickings.org/2012/04/12/john-cleese-on-
creativity-1991/)

------
laythea
Good article and I agree with most points, however I am not sure I like the
articles tone. Maybe I am a precious butterfly but it was a bit "all-knowing",
with too much emphasis on how you save the day. I am sure your phone will
start ringing after publications too :) Thanks anyway though, I read it :)

------
banachtarski
I don't understand why Unreal Engine 4 is in the title. It feels click-baity
since you're expecting to read something about the "dark side" of UE4 or
something.

~~~
egypturnash
First sentence of the article: "I have been working professionally with Unreal
Engine for over 9 years."

Second paragraph: "If a game company in Los Angeles is having an Unreal Engine
4 problem that no one can solve, I usually end up getting that call. I am
writing this article to explain why I get that call, how one would prevent the
need for that call, and what I normally do after getting that call."

So while a lot of these things are pretty useful strategies for _any_ large
project (we had asset naming conventions for parts of the Flash cartoons I was
working on around the turn of the century), they are all common problems he's
encountered in his career of being That Guy You Call When Your Big Unreal
Project Is Broken.

~~~
banachtarski
Yea I read the article >.>

I still read it as him trying to suggest what better practices you can adopt
specifically pertaining to UE4 that would "prevent the need for that call"

~~~
nightfly
Have someone who has a clue how to design and integrate complex systems from a
high level on your team. Seems to be the point of the whole article.

------
golergka
Wow, this hits way too close to home. On the last few projects I've been hired
as a lead Unity Dev, my main job was to untangle the mess left by previous
maintainers and finally put some good engineering practices in place. Project
I'm working on right on had it's multiplayer "just added" on top of single
player gameplay mechanics, and now my team finds itself in the middle of the
complete architectural rewrite instead of making any new features.

------
_bxg1
Most of this just sounds like standard software engineering best-practices,
though from what I've gathered it seems like standard SE challenges get
multiplied five-fold in game development. It takes an engineer braver than I
to work at a triple-A game studio.

------
lkrubner
About this:

“ _responsibilities than they can handle, and development suffers from this
person being fatigued, or worse, leaving altogether. Neither the engineer nor
team can grow because of this, as new features or changes get increasingly
more resource-demanding to implement_ ”

This is said about mid-level engineers who burn out, but I think it happens to
senior level engineers as well.

------
FollowSteph3
One of the most common things I’ve seen over and over is under powered
hardware. It’s so cheap in comparison but since it’s easy for an accountant to
total compared to lost time it’s very frequently done. It’s not a solve all
but in so many cases it would make a significant difference ignoring all else.

------
johnmcarthur
"If you're reading this as an engineer (but this may apply to many fields), or
as someone looking to grow an engineer, the most important thing you can do is
identify current issues and weaknesses."

I learned a very different things, but thank you nonetheless: there _are_
companies that care about these things! I just got a full-time job that is
basically like this, a hell on Earth (I've been freelancing before). So the
lesson I learned is, I should be looking for a new job :)

My coworker (we are 2 in the team) is already jumping ship even when he joined
later than me.

------
nudpiedo
Since I've lived, almost to the letter, a few times many of the scenarios
detailed here, I cannot upvote more this post.

My question now is: is there any company mid-big size not guilty of those
sins? Does it even exist or it is an utopia wished but not achievable? (just
asking)

------
EliRivers
_When I got the call to fix this dilemma literally a couple of weeks before
release, the simple act of cleaning up the source control server, and purging
these unneeded cache files from source control, fixed this audio issue that
was plaguing them for months and had costed the company tons of money and
time._

This sort of thing drives me insane (or at least used to; my current
colleagues, from junior to senior, are all actually pretty good). This is
something that everyone knows, just by looking at it, is a bad thing. Everyone
who saw this must have known it wasn't a great idea, even if they didn't know
why. Even without being able to elucidate why, everyone (should) get the
feeling that having these things in the repository is asking for trouble; I'd
expect a junior to have at least a bad feeling about it, and a senior to be
able to specify what kind of problems it could cause. I don't know about the
exact circumstances, of course, but it's even surprising that nobody found
this problem while they were investigating. I've certainly gone as far as "are
we definitely using the exact same binary files on the different systems?"
when chasing odd problems that only manifest on some installations and not
others, so what went so wrong that nobody got that far in their debugging?

So unless the company in question had by chance a room full of people with
this particular massive blind spot, people saw this happening and they knew it
was a bad idea and they didn't do anything about it. Culture failure. I don't
know why. Maybe they were afraid to mention bad things because pointing out
possible problems is seen as not being a team player. Maybe they were just so
sick of it all they didn't care anymore. Maybe there were a couple of
forcefully strong people in charge of the repository who saw any question as a
personal attack and people were just sick of dealing with the egotistical
prima donnas. Maybe the process for doing anything about it was laughable
painful and went through someone who would shoot it down on the grounds that
we're so busy firefighting things that are broken and you want to waste time
tinkering with something that looks fine to me?

If Allar is reading, could you say a few words about this? How many times do
you get called in to fix something that could have been averted with a decent
programming culture, even if the specific technical chops weren't there? We've
all seen teams with good culture deliver software that they really didn't know
how to build when they started through good culture, and we've all seen teams
of people who all know what they're doing deliver a flaming wreck because the
culture was a disaster. How often are you called in to deal with something
that the existing team really couldn't have foreseen without some particular
piece of specific knowledge or experience, and how often is it something that
they could have dealt with months ago if they had just been allowed and guided
to work sensibly and professionally?

~~~
Allar
Literally no engineers were allocated to the issue and majority of the
project.

20% of the time I work on novel unforseeable issues. R&D and future tech
stuff.

80% of the time its culture, professionalism, and generally "are you fucking
serious" issues.

I've dealt with literally every situation you've described however. Just
because this particular reason was that there were no engineers, you could
swap in any other situation and still have a true story.

------
dang
Url changed from [https://80.lv/articles/confessions-of-an-unreal-
engine-4-eng...](https://80.lv/articles/confessions-of-an-unreal-
engine-4-engineering-firefighter/), which points to this.

