
What senior engineers do: fix knowledge holes - mooreds
http://www.mooreds.com/wordpress/archives/3232
======
codr7
I once had a boss who asked me to spend my last two weeks writing down
everything I knew after working there for 13 years and basically creating the
entire 2Mloc system they were happily selling from scratch.

That's when I finally realized how much of a clue he really had.

And the reason I quit is that he gradually started chasing me around and
micromanage my time even though I had been left alone to do my thing since
forever, and pretty damn good too. But you can't measure the effect, and suits
tend to measure what they can.

The effect of leaving was easier to measure, the company folded about a year
later.

~~~
godelski
I had a similar situation. Had won a small contact and was the only one
working on it. Then turned it into a much bigger contact. It was like 30% of
the company's revenue so I asked for a raise (I was at the bottom of the
ladder). Said no so I started applying to grad school. They started micro
managing me and asking for a 10 page report each week detailing what I did.
When they caught wind of my leaving they asked me to write everything down
about how to do what I was doing.

Company is still there but they lost more than half their engineers. They
didn't realize why people were leaving. Sometimes it's suits. But these were
engineers. I learned a lot about how not to manage people though.

~~~
krageon
> They didn't realize why people were leaving.

I think this is a standpoint that is not defensible. Seeing how it is close to
their core competency (people), it is more reasonable to assume that what they
were doing was a conscious choice. Maybe you (and apparently a lot of other
people) think that is the wrong choice, but that doesn't mean it was not made.

~~~
godelski
Well when I'm grabbing beers with my manager and he's like "I don't get why X
left. He was doing well. Guess it must have been <unsubstantiated reason that
is a flaw in their character>" I fully believe that they just didn't
understand. Frankly because they told me. I'd say "no, they left because
<x,y,z,...>" And they'd respond "well that's dumb. Were about to get funded.
They're missing out on a lot of money" (we had no equity and had asked several
times, gotten bs answers). They still don't have funding and it's been 1.5
years (they had been working on with that investor for a year prior). It
really was just delusions of grandeur.

Maybe they do realize now (I was the last to leave, even though the first to
secure my ticket out). They thought I was leaving because I _just_ wanted my
PhD. I do want it, but there were a lot of reasons I left. Big part was being
micromanaged on a project I won, I did all the work on, and I was the only one
that had even basic knowledge in the subject (like knowing the difference
between a proton and alpha particle. True story),this was the largest contact
we currently had, and I was paid the least. They didn't see how insulting this
was.

Sometimes people are blind. It's probably much more common in small companies
too. We were <20 people. 5/12 engineers left within a 3 month period. All but
1 E1 left and all E2s left. When people are dropping like flies maybe it's not
the best to continue blaming those people and find out what you can do to have
higher retention.

------
EliRivers
I work on some software of which the oldest parts of the source code date back
to about 2009. Over the years, some very smart (some of them dangerously smart
and woefully inexperienced, and clearly - not at all their fault - not
properly mentored or supervised) people worked on it and left.

What we have now is frequently a mystery. Simple changes are difficult,
difficult changes verge on the impossible. Every new feature requires reverse-
engineering of the existing code. Sometimes literally 95% of the time is spent
reverse-engineering the existing code (no exaggeration - we measured it);
changes can take literally 20 times as long as they should while we work out
what the existing code does (and also, often not quite the same, what it's
_meant_ to do, which is sometimes simply impossible to ever know).

Pieces are gradually being documented as we work out what they do, but layers
of cruft from years gone by from people with deadlines to meet and no chance
of understanding the existing code sit like landmines and sometimes like
unbreakable bonds that can never be undone. In our estimates, every time we
have to rely on existing functionality that should be rock solid reliable and
completely understood yet that we have not yet had to fully reverse-engineer,
we mark it "high risk, add a month". The time I found that someone had
rewritten several pieces of the Qt libraries (without documenting what, or
why) was devastating; it took away one of the cornerstones I'd been relying
on, the one marked "at least I know I can trust the Qt libraries".

It doesn't matter how smart we are, how skilled a coder we are, how genius our
algorithms are; if we write something that can't be understood by the next
person to read it, and isn't properly documented somewhere in some way that
our fifth replacement can find easily five years later - if we write something
of which even the _purpose_ , let alone the implementation, will take someone
weeks to reverse engineer - we're writing legacy code on day one and, while we
may be skilled programmers, we're truly Godawful software engineers.

~~~
tus88
Legacy code from 2009. I'm getting old.

~~~
WalterBright
I'm still running/maintaining code I originally wrote in the 1980's.

[https://github.com/dlang/dmd/tree/master/src/dmd/backend](https://github.com/dlang/dmd/tree/master/src/dmd/backend)

~~~
tus88
Wow, I didn't realize D went back that far. I thought C++ came in the early
90s and D naturally followed as a "better version" after that. Very cool.

~~~
WalterBright
It was originally for a C compiler, and work started on it in 1983 or so.

------
jorblumesea
I've often thought that the need (and compensation) for senior engineers is
partly based around their historical knowledge of systems and business logic.

If you lose a senior, not only do you lose a highly skilled individual, you
also lose institutional memory. Documentation can help some of this, but is
often not enough or does not capture reasoning behind technical
implementations.

Senior engineers are often living repositories for business logic and company
knowledge.

~~~
mooreds
> Senior engineers are often living repositories for business logic and
> company knowledge.

I've definitely seen that. I agree that documentation can help with this, but
you can't document _everything_ (unless you're working for NASA. I have a
friend who works for an aerospace company and they do a lot of documentation).

This also has an effect on your compensation. If you've been at a company for
years and have deep knowledge of the domain and the software, you're more
valuable than you might be on the open market. It's interesting to me that
(typically) the company doesn't recognize this and compensate you
appropriately.

~~~
dilyevsky
Companies like to play chicken with engineers bc going to interviews to prove
your true market value is not something most people are comfortable doing.

~~~
mooreds
wish I could upvote this 10 times.

------
tracer4201
I agree fixing knowledge holes is something I expect a senior engineer to own.
Overall, I think what stands apart senior from junior engineers isn’t just the
skill or the act of fixing the knowledge hole but -

1) having the right judgement 2) being a force multiplier

Good judgement is important. For example, a knowledge hole could exist, but is
it the right thing to be working on now given other knowledge holes or other
problems? Is it something another person on the team could pick up while the
senior engineer focuses more on another, more ambiguous problem?

Being a force multiplier I think is critical. Yes your experience and
expertise matters, but how well can you scale yourself? This can actually be
very difficult and even demoralizing. When I first reached a senior level, one
of the teams I was the lead on at the time had an issue where other engineers
literally were not motivated. They don’t want to move on because it’s an easy
environment but they don’t put in even 50%, and management was focused more on
managing upwards than managing downwards. Heh some interesting lessons learned
there.

~~~
mooreds
> Being a force multiplier I think is critical.

Totally agree. At some point you spend less time doing the work and more time
unblocking other people, doing spikes and research, and coordinating across
the organization. For someone who like to be in "flow", all of these can be
frustrating.

~~~
perl4ever
I got to a point where "actually working" seemed worthless because everything
that determined the ultimate value of my work was a management/communication
issue. My boss would tell me not to worry about that stuff, but it was
demoralizing because I couldn't stand any longer to think about spending year
after year doing technical stuff that had no positive, cumulative effect on
the real world.

I don't want to be a "boss" or have more power or prestige, but I've gotten to
the point where the only problems I am really interested in are management
problems because everything else is a waste of time if those aren't solved.

~~~
mooreds
"The Second Law of Consulting: No matter how it looks at first, it's always a
people problem." \- Gerald Weinberg (
[https://en.wikiquote.org/wiki/Gerald_Weinberg](https://en.wikiquote.org/wiki/Gerald_Weinberg)
)

------
kd5bjo
Overall, the sentiment is right, but this line seems off:

>Note, he was a software engineer, not QA or a network engineer. But he didn’t
let the role definition stop him.

QA is about determining when things are broken, which they weren’t (yet).
Network engineering is about making sure all of the packets get to to where
they’re supposed to go, and not what’s inside them.

This is about a private protocol between two pieces of software that are both
exclusively developed by the same company, and probably the same team; its
details fall quite squarely in the purview of software engineering, or at
least they would at every company I’ve ever worked for.

~~~
mooreds
I wrote this piece. Thanks for the feedback.

The point is that he was was a software developer who was responsible for the
back end systems. I don't think he was directly responsible for the
client/server protocol and the client was, I believe, outside his purview. I
was trying to think of a couple of different roles that might have had
incentive to take care of this. (QA because of a desire to ensure the software
was working correctly, network engineering because they might have helped
design the protocol.) But he saw a knowledge hole and he acted to remedy it.

But I take your point that the line is a bit off. I'll edit.

------
Niksko
I think the other key part of a senior engineers role is to ensure that
product doesn't trump quality. When you have no senior engineers, what often
happens is without an advocate for doing things right, product 'wins', and
teams turn into feature factories with little time for quality.

Yes, quality is everyone's responsibility. But without an advocate with the
right title and respect, it becomes secondary or even tertiary (behind
velocity and features)

~~~
ehnto
I worked at an agency who's manifesto was built on quality over all. It
permeated the culture and decision making. If we thought a particular way was
the best approach but the client couldn't afford it, we would sometimes even
split the cost as to avoid the cost of maintenance down the road. The clients
were usually long term clients so this made sense.

Eventually we grew to the point of needing some project managers and I learned
so much about project management during that period. We had some that
absolutely understood the ethos, advocated for the engineering team and
communicated to the client why we do things the way we do.

Then we had the PMs who saw the client as their paymaster, wanted to please
them by making big promises, saw engineering as their personal work force, and
were happy throwing engineering under the bus when it didn't work out.

The worst work was done under the latter PMs and it always came back to bite
us. Half baked features needing rewriting, third party modules hacked into the
codebase, bugs and untested code needing weeks of client back and forth.

------
jvandonsel
That’s an excellent, terse description of what I’ve found my role as a senior
engineer to be. It’s not in the official job description but it’s what I end
up doing day to day.

~~~
mooreds
Thanks!

I think this is why job descriptions for senior engineers are often hard to
write. You often want a general purpose problem solver, but all you have is
examples of types of problems that have been solved in the past, and so you
put that in the job description (the general you, not you specifically, of
course).

Reminds me a bit of "Profession" by Asimov:
[https://www.abelard.org/asimov.php](https://www.abelard.org/asimov.php)

------
proj-14
The link is broken, I’m seeing a 500 error.

~~~
mooreds
Sorry, hug of death on shared hosting. Just enabled caching, please try again
(I couldn't find a cached version on google, doh!)

------
xchaotic
There is job title inflation and my company hires at Senior Eng as the lowest
level.

------
Noumenon72
Seems to me that being curious about things and documenting how they work is a
very junior-accessible thing to do. But a junior probably wouldn't have
identified this as needing to be done.

~~~
notus
Juniors also generally do not understand how a piece fits in a larger system
nor are they are expected to.

~~~
drdeadringer
I have always had a "problem" of needing to understand how something fits into
a larger system and "not being allowed to" ... because "Junior" or "new guy".

Yes, I do need to know "why" this is or "why" that is. "Just because" has
never sat well with me, ever.

This could be my own "problem" but really: when I ask "why", give a real
answer -- I'll even take a Cliff Notes answer, but just give a real answer.
"Don't worry about it" doesn't do me any favors.

~~~
notus
It's hard to give good advice because I don't know what the culture is like
where you work. Sometimes this behavior is because they are feeling you out
for a few months and making sure you stick around before investing a lot in
you. Other times it could be because the more senior devs like being in
control and making decisions. You have to learn to work around these behaviors
because directly confronting them won't always go so well.

The reality of the situation is they are just making their own lives more
difficult by not enabling more developers to help out but that's how people
are. Do you do 1 on 1's with your manager? That is the best place IMO to
express what you want to do and if you're feeling left our from knowledge or
conversations. A good manager will help you out and make the senior devs be
more inclusive it makes sense. If they are not helping then go to the
director. If the director doesn't help then you should reassess your future in
the company. Note that I'm not saying quit, because sometimes, especially when
just starting out, you have to work a less than ideal job to build up some
experience.

As long as you're still learning and growing you can benefit from it, but the
second you feel like you're stagnating and you're sure it's not you it's them,
then leave. Next time you will have something to discuss at the interview too
about what kind of place you want to work at.

------
flatfilefan
You can call it anti-dailyWTF.

~~~
trilila
Up until the tech lead comes along and calls it anti daily wtf for the
seniors. Then the cto on top of the lead. Then ceo on top of the cto. And the
rate race saga that never ends and loads of software workers sharing rooms in
overpriced areas, and doing voluntary work to put on their cvs calling it open
source, and mindlessly believing what every single company will tell them: we
hire the best in the world. Rinse and repeat each generation.

~~~
flatfilefan
What you wrote is almost a half of the "meander WTF-manifesto"!

