
How I operated as a staff engineer at Heroku - craigkerstiens
http://amyunger.com/blog/2020/09/10/staff-engineer-at-heroku.html
======
delete_more
A thought I had recently is that staff/principal engineers who got their
position by long tenure (not hired recently) should delete more code. Deleting
code is actually one of the most difficult things to do in a project. To do it
properly and without risking breaking things, requires not only vast and
detailed knowledge of the system, but also knowing the context and decision
around when the feature was added.

A long tenure engineer is the perfect fit for this. They know how to
surgically remove the useless code. They remember all the hopes they had for
the feature and when it failed to produce those results. They know well that
code is not free, it's an ongoing cost to maintenance and complexity. This is
not easy work to give to a junior. It's not even for senior engineers because
they often lack the context and broad system knowledge. Only the long tenure
staff engineer can confidently say, that feature failed, and I can remove it
without breaking anything. If someone brings up sunken cost fallacy and how
some users use the feature, only the long tenure staff has the background,
respect, and authority to say, no this feature failed, it has 1% of the
results we wanted, it's not worth the ongoing maintenance burden.

A staff engineer is supposed to a team multiplier through code. People often
think of designs, refactoring, helpful tools, architecture discussions and
documents. And it's true, these are all code related things that can multiply
a team's efforts when done well.

But what better way to multiply a team's productivity than to reduce the size
of the system? Think of how green field projects start. They feel so fast and
progress is so quick. Then the features come, more and more, slowing down
development with expanding system complexity and maintenance of code. How much
faster and quicker can the team move after the staff engineer has reduced the
system by 50%? It's like turning back the clock on an aging system. Suddenly
it feels years younger, able to move quickly again, but this time with the
right features in place from the experience of past years. And the team is
happier, less useless code to wade through and understand, less strange bugs
caused from code that supports a useless feature.

~~~
PopeDotNinja
I don’t disagree that purging unneeded code has value. That being said,
software systems are so complex, isn’t it better to just leave code alone if
it’s not negatively impacting the business? How many orgs have a micro service
architecture that is so well tested that you can delete code from one service
and demonstrate that won’t cause a failure in some other service? I haven’t
seen one yet.

~~~
aszen
Deleting code shouldn't be this hard, by logging how often a piece of code is
being invoked in production one can start planning purges of unused dead code.
It has to be done carefully but by no means it is so complex that it shouldn't
be done.

Dead code is often a negative for maintenance, understanding and performance.
With version control one can always go back and recover deleted code that
proves to be useful later on.

------
etimberg
Wow! As someone who has one of these nebulous titles ("Software Architect"),
this is one of the most accurate descriptions I've seen of what the job
entails. My employer is nowhere near the size of Salesforce, so a lot of
things scale down, but the basics are the same.

I would say the biggest challenge with roles like this is the loneliness and
what seems like constant conflict. One needs to work hard to keep up personal
relationships with product/sales/project management/etc to ensure that the
level of trust is there to be able to have the hard conversations. It is
really easy to forget that, and a lot of those other folks start to feel like
you're always just shooting down their ideas.

~~~
rarecoil
> One needs to work hard to keep up personal relationships with
> product/sales/project management/etc to ensure that the level of trust is
> there to be able to have the hard conversations.

In the heat of the moment, on deadline with a bunch of bugs and yet multiple
compliance hurdles to clear, with a short-staffed team and conflicting
management priorities, it's pretty easy to lose sight of the common goal.

Microsoft has a nice cliche you can use when you catch yourself on the
adversarial end of this type of conversation: "assume positive intent". It's
written on most of the Redmond campus whiteboards. If we are all assuming
positive intent, it can give us the right empathetic mindset with which to
build this trust even when it seems difficult. I've found working with other
teams through the hard conversations with compromise instead of "no" leads to
better results. They're assuming positive intent as am I and it's OK. If I'm
feeling something is adversarial I need to understand why.

~~~
jaggederest
I like to frame it as "not me versus you, us versus the problem" when things
get too heated. Otherwise things degrade into ad hominem "Those goddamn X
people", instead of "We're having a lot of problems getting our project to
work with X"

~~~
rarecoil
I like this one, too. The moment the _people_ are your adversary in a
technical discussion, you have probably lost sight of the actual problem. Much
of the time I find it's misaligned incentives or, as Amy wrote here, sometimes
it's people working with obsolete context or old grudges. Reframing yourself
first, and then the discussion, is very much the way to succeed.

------
spenczar5
What a great writeup. A lot of it sounds very familiar to my experience as a
principal engineer at Amazon. There’s a certain point at which you realize the
hardest software engineering problems for a large company are _social_
problems, and that’s what you devote most of your energy to as a PE (or, I
guess, a Staff Engineer).

Personally, I really didn’t have the stomach for the work, eventually; I have
gone back to active programming, this time for scientific projects, and I am
much happier.

~~~
cyrux004
Are you still working as a Principal Engineer ? Edit: just saw that you are at
UW now.

------
amerine
As someone who worked with Amy for years at Heroku, I can confirm that this is
very accurate. Amazing write up, Amy. Damn.

~~~
vishvananda
Less years, but I concur.

------
georgeoliver
>Everybody wants to hire senior engineers; nobody wants to make them.

Gold.

~~~
humanlion87
I have been struggling with this right now. I am not good enough nor
experienced enough to figure out things on my own (to get to senior engineer).
I feel I just need some guidance or push to get to the next level. But it's so
discouraging to get no support from my management. And other folks around me
also keep saying that depending on management to support me in my career
growth is not a good idea. I feel it's ridiculous and distorts my image of the
role of management. I just have to accept that good managers/management is
hard to find.

Anyway, it's kinda good to know that it's not just my company/team that seems
to have this issue.

~~~
scollet
Sounds like you need some meat (analogy aside of dietary preference) to sink
your teeth into.

It's not a push because a push comes from behind. It's you grabbing onto
something elevating and interesting because it's juicy.

A reasonable senior will recognize your talent, but a stagnant organization
will simply extract your current value without pulling you up with that bite.

I know the analogy is bit inhumane, but the human part comes from those very
human people who just want to solve problems.

Whether they be lower in pay/'rank' or higher, find the people who appreciate
your strengths immediately. Don't ever be afraid to express your ideals in
your capabilities and learning process.

In any organization from 10 to 10,000 you will find those folks who can take a
genuine appreciation for your skillset and apply it.

The failure state should always be: "I'm departing to company Y, I wish you
the best."

The success is an expression of your ideals.

The 'meat' of my argument is - at your level - don't look for the push, find
the pull.

You've got this.

Edit: an -> and (Para. 5)

------
throawayyyy
So here's a curious question that is perhaps _incredibly naive_ and seemingly
impolite, hence the throwaway, but I really don't know, which is why I'm
asking.

Why must one be a senior engineer in tech before they can become a staff
engineer? To me, this seems much more like a communication role. Moreover, the
tech that is needed to know is at a high level. Is it really needed to know
the technical finer details of systems when high level software architecture
and communication seem to be 90% of the whole role?

Or put in a more blunt, albeit unrefined, way: why can't someone who
specializes in communication become a staff engineer? I mean people like
consultants. They get so much experience with communication and high level
tech (some of them at least) that this seems to be what they'd be good at.

~~~
questionableans
Another thing to keep in mind is title inflation.

The role that most companies these days call “Senior Software Engineer” is
actually an intermediate-level role (e.g., 5 years of experience). Staff is
arguably where the actual senior-level roles start. This corresponds with the
transition from being told what to do to being told to figure out what needs
to be done and make it happen.

~~~
disgruntledphd2
If you are a Senior anything and being told what to do, you're not really a
Senior.

Maybe that works for you, if you're in the right place getting paid the right
amount, but we do ourselves a disservice when we allow this to occur.

My personal understanding was: junior: lots of tactical guidance

medium: less tactical, lots of strategic guidance

senior: no tactical, very vague strategic (normally coming from business
partners).

Maybe that's just me, though.

~~~
learc83
>senior: no tactical, very vague strategic (normally coming from business
partners).

>Maybe that's just me, though.

That's just you. I agree that's probably closer to how the terminology should
work, but it doesn't. I'm unaware of any large company where senior engineers
are getting direction directly from business partners.

You're lucky if "senior" engineers have 3 years of experience.

------
jrott
This is really good. It's one of the best descriptions I've seen of what being
a staff engineer is like.

When I came into the role I was stunned at the amount of time that I ended up
talking to people and just trying to figure out what is going on.

------
selfinvariant
This article is incredibly well written and rings true to me and my experience
being in a similar role / working with people in such roles.

I like the notion of articulating a career path in tech as of: 1) Learning the
ropes 2) Being a producer / writing code / driving design decisions 3) Being a
multiplier - working across different teams to unblock / amplify other peoples
work

3) is pretty much communication centric, while 2) is deep knowledge of tech
involved

There is a lot of nuance to that of course (which I think this article
captures well within specific organization). I'm curious if this is something
any one else has the same experience with

------
kenhwang
So glad the term "hopes and dreams" was used. That's exactly how I describe
half my staff level role. I move mountains to enable and materialize "hopes
and dreams", or I destroy them without a sliver of doubt, in equal parts. That
means doing at least all the things the post describes.

But the other half was only lightly touched on. Staff is as much spearhead as
it is rearguard. So that means caring for legacy codebases, aging tech, and
anything else that's falling behind. It's search and rescue and postmortems
after things have gone catastrophically wrong.

Great great write up!

------
koolba
I do not know this person at all but after reading this I think I’d enjoy
being her coworker.

~~~
exolymph
Right? She sounds so competent and practical, but also kind. Love it.

~~~
bjeanes
I worked with her (I was on the team she was initially hired onto at Heroku)
and she is one of the best hires we ever made, hands down.

------
sciurus
If you enjoyed this, check out [https://staffeng.com/](https://staffeng.com/)

It's composed of interviews with Staff-plus engineers as well as guides for
reaching and succeeding at Staff-plus roles

~~~
nojvek
I love this site. Thanks for sharing.

------
sethammons
> You may be furthering existing misconceptions or old grudges that really
> should die off. Actively invest in meeting new folks as a counterbalance.

This one hits home. I've been at a rapidly growing org for nearly a decade and
have grown a massive amount in turn. Things that I thought were the case
sometimes ceased being so, literally, years ago.

------
ChrisMarshallNY
Great write-up. Not sure I'd want the job. Sounds like she was a good fit for
it.

I was a manager for most of my career, and spent most of that time pining for
the fio- er, my days as a coder.

So, after I left my career position, I reverted back to a coder. It's been
nice.

------
orliesaurus
Damn really nice read! I wonder: is this what enterprise software engineering
really feels like up there in the high ranks? I don't know if I would ever
enjoy doing any of what she listed!

~~~
cmwelsh
I had a similar reaction but there are a few archetypes that are very similar
to the duties of an IC. She mentions “Deep Diver” and that’s similar to my
current role. I enjoy having systems thrown onto my lap and becoming an expert
in them as quickly as possible. Then the organization uses my analysis to
decide what to do next: keep it or pivot to a different strategic solution in
the long-term. Ideally my deep dives result in useful documentation, code
improvements, or process automation to allow the future deep diver to come up
to speed quicker than I did.

~~~
spdionis
As a staff engineer, you have multiple responsibilities you need to cover, and
you can delegate some of them.

The reality is that in most teams you can delegate the "Deep Diver"
responsibility most of the time. "Context sharing" and "Coaching" not so much.

So yes, as some comments in other threads mention, your coding/hard tech
contributions diminish with time.

------
paganel
> If we made the wrong bet, some blame is with product, but it’s also on you,
> and the manager probably won’t look great.

Why would any manager in his/her right mind choose to take any major risks
while working for a big company? Maybe I'm looking at it wrong, but "looking
not great" because you failed makes sure that next time you'll have to make
the choice you'll always going to pick the safe bet. And safe bets aren't good
for any technological company long-term.

~~~
shankhs
I am with you on this. There is some positive incentive if the technical
manager gets the product right maybe a strongly exceeds for a year. But if the
product fails TLM gets equal share of negative press and a low rating which
means greatly low bonus and bad reputation which will make hiring even more
difficult. I don't see any reason why a TLM will take any major risks while
working in a large company.

------
sabujp
What is the salary incl bonus for a "staff engineer" at heroku? I find that
titles are meaningless and in the end it also boils down to show me the money.

~~~
nfriedly
Heroku is now part of Salesforce. They don't appear have an official Staff
level, but per [1] the levels above Senior are Lead at ~$287k (total comp),
Principal at ~$378k, and Architect at ~$557k.

[1]:
[https://www.levels.fyi/?compare=Salesforce&track=Software%20...](https://www.levels.fyi/?compare=Salesforce&track=Software%20Engineer)

~~~
secondcoming
US salaries are insane

~~~
nfriedly
Yes, although to be fair, that's on the high-end even for San Francisco. Total
compensation in the range of $100-200k is much more common.

~~~
yibg
That's not right. Base compensation of 100k - 200k for engineers is more
common, and even that is usually for up to "senior" engineers or so.

------
spdionis
Really good article. Context sharing is really powerful.

Your job as a staff engineer involves going into a cross-team meeting and
explaining in technical detail a given piece of work, thanks to which
your/another team re-estimates a given ticket from 5 points to 1 point. Those
4 points of difference are your hard work, done over the course of a 30-60
minute meeting.

This is probably what people mean by "force multiplier".

------
beams
Great questions and answers. Admittedly, I can’t help but feel there aren’t
very many pain points mentioned, which makes me ask what’s being left out.

Additionally, how often was coding actually involved in this role? I hear this
claim often but in my experience it’s rare. One-line fixes don’t count.
Specifically, what projects were you a code contributor to?

~~~
bradlys
I'm going to guess quite infrequently based on this first sentence of how she
spends her time.

> While I do write code from time-to-time, it’s only after I’ve delivered on
> my obligations on these four functions.

------
User23
>> How do you keep in touch with how things really work as you spend less time
on hands-on development?

> As someone who still does write code occasionally, this is something I
> struggle with. Our codebase shifts underneath me constantly, in ways I often
> don’t see. It has been my goal for the year to make sure I say “I’m not
> sure, let me check” even when I feel confident.

This is not at all consistent with Staff Software Engineer at any of the
several Big Tech Companies You've Definitely Heard Of I've worked at. It would
be more of a Distinguished Engineer level sinecure[1]. Staff is usually
expected to spend at least 50% of their time actually writing software.

[1] Not pejorative. If I were the relevant hiring manager I'd be happy to let
someone like, say, Leslie Lamport, to pick a name out of the hat, spend as
much or as little time as he felt like writing software just to have him
around.

~~~
spenczar5
Definitely extremely consistent with Amazon’s Principal Engineers. Internal
surveys on time spent coding support it.

------
unixhero
I really liked the perspectives on the four obligations every day.

Planning

Context sharing

Coaching & Culture

------
pier25
For the past years it seems Heroku has kinda stagnated. When was the last time
they announced a new feature?

It seems all the talent is focused on Salesforce.

------
aprdm
This is truly great! Work in an organization with ~200 sw engineers and some
of it is pretty accurate to my role as well.

------
moron4hire
...

What is a staff engineer?

~~~
kenhwang
The level above lead/senior.

~~~
noir_lord
Not every org has them though depending on size, I’m a lead who does a lot of
what she describes and there is no level beyond mine that isn’t pure
management (which I don’t want if I’m honest).

When it comes down to it my role is a lot of ad-hoc dealing with bun fights
before they come actual fires.

------
fizixer
Solver/Deep diver FTW

All the way.

