
Trying to sneak in a sketchy .so over the weekend - ingve
https://rachelbythebay.com/w/2020/02/09/horizonta/
======
PragmaticPulp
I expect this post will resonate with a lot of HN readers. Every tech company
I’ve worked for has had at least 1 manager who tries to ship features over the
weekend with a ragtag team of developers who don’t understand the system or
why that’s a problem.

If they succeeded, the manager would have pointed to the feature as an example
of their “hustle” and ability to get things done where others couldn’t.

If they shipped the feature and it crashed the website, the manager would
blame the front end team for making a fragile system that couldn’t handle a
simple feature.

If they failed or were blocked, they’d point out their working proof-of-
concept and blame the front end team for onerous process or politics.

The real litmus test is how the company reacts to that manager after this
stunt. If the company sides with the hustle manager, it doesn’t bode well for
engineering in the long term. When management rewards shows of effort instead
of long-term results and fails to acknowledge negative externalities or
selfish managers, you breed more of that behavior.

However, if management sides with engineering and shuts the hustle manager
down, you’ve found a good workplace.

~~~
zild3d
> Every tech company I’ve worked for has had at least 1 manager who tries to
> ship features over the weekend with a ragtag team of developers who don’t
> understand why that’s a bad idea.

Maybe I've been in startup land for too long, but seems super normal and fine
to ship a feature over the weekend if it goes through the regular CI gates -
it's tested, peer reviewed, been QAd in staging, etc. Is this not accepted
outside of startups?

~~~
randomdude402
I'm at a fintech in NYC and my team generally doesn't release stuff to prod
between like 10AM Friday and 10AM Monday.

I haven't had specific conversations with anybody about it, but I think we
have all been around the block enough times to have been burned on a few
weekends when it really wasn't necessary.

Not a start up at all though, and not a team of twenty somethings with
anything to prove by moving fast and breaking things.

Agree with you though that I have seen this at a lot of places. I did a number
of phone interviews looking for a more relaxed place in order to end up here.

~~~
pnako
Deploying stuff over the week-end is standard practice for trading systems,
because that's when markets are closed. But obviously those deployments are
discussed and planned during the week.

In fact most trading platforms have this huge advantage of not being 24/7
operations.

------
hyperpape
Funny that everyone leapt to the conclusion that it was a junior developer. I
immediately thought of a senior developer who "gets things done" and gets
rewarded for that, but leaves a flaming pile of rubble in their wake. I've
seen bad bugs written by junior developers, but I've spent more time cleaning
up issues introduced by experienced people for whom time had just reinforced
bad habits. I don't know which is more common in general, that's just my
personal history.

Who knows which it is. The article says they haven't touched the frontend
servers yet, not that they haven't done anything substantial yet, so I don't
think we can know unless Rachel decides to comment on that point.

~~~
ObsoleteNerd
The worst experiences I ever had at a job were due to the most senior
developer there. He was basically given this god-like status from management
where he could do no wrong, and had free reign of all systems, because he'd
been there for so long..

Yes, he was an incredibly capable software developer, but that had led to an
ego issue where he thought he knew EVERYTHING, not just developing in his
particular stack... so he'd find his way into various systems (software and
hardware) he had no experience in, and assume he could just figure it out.

We had half a dozen full-on-disaster level situations in the couple years I
was there thanks to this 1 person. Each time it was shrugged off as "an
accident" because heaven forbid you actually upset him or hold him
accountable.

~~~
throwaway777555
This sounds a lot like my current situation at work. We're a decently-sized
company of 100+ people. About a quarter of them are developers. Developers are
divided into teams who are responsible for specific components of our system.

However, there are also a few "architects" who are basically free agents with
god-like status. They don't architect anything. Rather, they do a lot of
counter-productive things such as attending meetings for specific teams
without those teams knowing. Then they make large decisions about the
direction of a given feature without documenting anything, let alone letting
those teams in on what's going on. It's always a fun surprise and works
wonders for our ability to deliver a solution on time /s

Beyond that, they'll go off and develop whatever they feel like and force it
through the system. We have a code review process, but if you bring up that
problems are present in their code, they will often say that it doesn't matter
and that the code just has to go through to meet some unspecified deadlines.
They'll get management to skip code reviews if you try to hold them
accountable. And of course, this results in things exploding. But then they
can clean up those exploded bits, knowing what they were, and come out
heroically since they "put out so many fires".

I'll stop ranting. But it really is frustrating, especially when you get shut
down by management for bringing up any issues with this chaotic group.

~~~
shrimpx
Sounds like every startup ever.

------
tialaramex
Week patterns have a long history, and humans have to rest some point, so I
respect the "Nothing ships on Friday [and implicitly Saturday or Sunday]" plan
outside of very young (often self-funded) startups where basically all that
the half a dozen people doing it are thinking about 24/7 is that one project
so there aren't really "weekends" anyway at first.

However, bigger organisations can get really out of hand with this. They start
out with small carve outs, oh, nobody really works from the start of Christmas
week until the first full week of the New Year. Soon it's from Black Friday
although good luck scheduling anything the week before that, and so before you
know it they're declaring that nothing can even be _planned_ for the months of
November, December and January each year.

Now, if you really must have this habit, which I don't recommend, what you
need to understand, just as much as the "No shipping on Friday" folks is what
this does to your perception of time. If you have "Don't ship Friday" then the
question in the Tuesday meeting "Will it be live on Monday?" must be re-
phrased "Is it ready to ship by Thursday?" and not every manager seems to
understand that. But when you have "Nothing can be scheduled for our 90 day
Winter Risk period" as policy then the question in that meeting "Will we be
ready for the January 31st legal deadline?" is actually "Are we 100% sure
we'll have fixed this by the end of October this year?" and I have yet to see
anyone who recognises that despite having a policy which makes it so.

~~~
EdwardDiego
We have a "don't deploy stuff that could go sideways between Dec 1st and Dec
25th" policy because that's when we have the highest a) traffic and b) spend.
But the converse of that, is that in Jan/Feb, we can sorta go nuts, because
all our customers blew their budget in the lead-up to Xmas.

It's really a case of knowing your business and customers.

~~~
azernik
I have a similar situation with similar timing, and have found it's a good
time to get people going on long-running projects. They're not going to be
deploying anything to production within a month anyway, so it doesn't block
them.

------
azernik
"There's a time and a place to dig in your heels and say NO to something. It
usually comes when someone else has been reckless. If you do it for
everything, then either you're the problem, or you're surrounded by
recklessness. I think you can figure out what to do then."

The magic is in telling the difference.

~~~
opless
There is a rule of thumb to not ship anything on a Friday.

This was a Sunday!

And it didn’t seem to have been tested, so where was the Dev/UAT/Prod
separation?

So many red flags, and I think anyone who has been in development for a while
has had some product owner try to break procedures to force whatever pet
project that is falling behind to hit some target to get their bonus/gain
brownie points. It almost never goes well. But by this time the PO has a
scapegoat because it’s no longer their fault it’s not working, it’s
production’s fault. (Also it’s not always a PO, it can be a developer or
anyone up to c-suite level)

Sigh

~~~
K0SM0S
_“No but you don 't get it, my son is really good at those things, he said he
would speed up our Windows machines 10x if I gave him the codes and physical
access to the servers during weekends!”_

That would be a very bad, very cliche joke if it weren't a true war story
(2005, good times). Turns out the 15 y.o. had convinced his father that I was
"lame" at my job because I didn't tweak regedit. Important lessons were
learned by the three of us that day!

~~~
ineedasername
Sounds like a "Dude, do you even regedit Bro?"

------
texthompson
One of the funniest stories I've ever heard was about how a junior developer
asked a more senior developer a creatively terrifying question:

How do I install half of an RPM?

~~~
WrtCdEvrydy
> half of an RPM

Well, I'm stumped... I wonder if he just needed one tool out of suite?

~~~
Const-me
Maybe trying to workaround a dependency issue. Not sure what's in rpms, but in
deb the manifest may require libsomething version exactly 1.1, will fail if
something else requires libsomething >= 1.2.

------
snlnspc
This comment thread originally consisted of one single reply by someone who,
apparently, was not aware of Rachel by the bay and her wonderful blog. That
post concluded that calling a fellow worker a "rando" was toxic, and that they
wouldn't want to work with the author.

While I agree with Rachel here at a high level, and have been a dedicated
follower of her blog, I completely agreed with that comment. You shouldn't be
shipping things in the manner described in this post, and you shouldn't be
considering your coworker "some rando" and looking down on them for not having
the same schedule as you.

~~~
GavinMcG
People are free to project "looking down on" or contempt onto the phrase "some
random person" and the (subsequent) use of "rando" but I felt like it
communicated precisely what was going on: this is a large organization, and
the person is totally unknown to this sysadmin.

Even aside from whether "rando" is contemptuous, the issue isn't that they
don't have the same schedule: it's that they are not respecting the _company-
wide_ schedule, nor are they respecting fairly obvious norms of professional
software development.

I'm a teacher, and I very much believe in educating people rather than putting
them down, but jumping on a single phrase/term when there's nothing else to
suggest contempt here strikes me as odd. It's especially odd when the entire
culture of sysadminship has a reputation of eye-rolling and begrudging
wizardry to protect users from themselves.

~~~
snlnspc
I think this is a fair point, and I cannot disagree with what you've said.
But, I must reiterate, the tone of this piece changes dramatically if "some
random person" becomes "a junior developer".

As far as the company wide/normal schedule goes, at my previous place of
employment, major changes were routinely performed (by me) off hours on a
Sunday with only relevant personnel on hand. This was primarily for B2B
reasons where the vast majority of our clients were doing mission critical
things from Monday to Friday. I don't feel that this was the case in these
circumstances, which is why I completely agree with what was really said here,
and with your reply was well.

I suppose my personal experience in this industry leads me to believe that
small snipes like that uncover much deeper contempt than is revealed on the
surface.

------
jrockway
Sort of tangential to the article, but... Setting aggressive HTTP timeouts can
be quite informative in finding "interesting" HTTP endpoints. Set them, enable
tracing, and search for timeouts every morning. At my last job, I found an
interesting "charge_cards" endpoint and learned that someone logs in every
day, clicks "charge cards" and all billing is handled in the frontend web
process instead of the event queue. It was idempotent so it wasn't a problem
(just click reload until it returns), but was an interesting bit of visibility
into a large system. Knowing that that's how it works makes it less surprising
when someone wonders "I noticed cards didn't get charged". Is the person who
normally does that on vacation? Aha, that's why. This never caused us any
problems so we didn't rewrite it, but it's always nice to have weirdness
exposed front and center so you are less surprised when it starts causing an
issue.

It was also interesting to read about a wide deployment of circuit breaking.
That's a relatively new concept to me (never ran into it at Google, for
whatever reason), but it's another way to prevent weird outages. Last week I
upgraded something in my k8s cluster and the API server stopped responding.
With the API server unresponsive, I couldn't kill the workload (or even
determine if that's what was causing it). I had to kill all the nodes before I
could get back in. If the job were communicating to the API server through a
circuit breaker, it would have popped after requests started taking too long,
and I would have been able to just kill the job instead of having downtime.
But, I guess it's still a relatively esoteric concept.

I know this article is about not doing dumb things in production. I just
assume those are a given. I'm more interested in the nuts and bolts of day-to-
day operation ;)

------
ATsch
> That's why some clever person had added a client-side "gate" to the
> transport. If it saw that too many requests were timing out to some service,
> they'd just start failing all of them without waiting

I believe this is commonly called the "circuit breaker pattern"

~~~
Rapzid
Si. If you already have a distributed rate-limiting service, it could be used
to create a more intelligent circuit-breaking service as well I would
imagine..

------
7ewis
Was half expecting these people to be rogue actors trying to install some
malware into prod.

Glad I was wrong!

~~~
cortesoft
It is really better if they are coworkers trying to install malware?

~~~
lmkg
"Sufficiently advanced incompetence is indistinguishable from malware."

~~~
NikolaeVarius
I remember an instance where someone updated and deployed the strong_passwords
ruby gem since "it had an update". This was more or less right after it was
found out that the v0.7 gem was compromised. Cue rushing at the commit and
double checking that it was using the "fixed" 0.8 and not the 0.7. Developer
had no idea that 0.7 was compromised, and just updated the gem because they
could without looking at patch notes.

------
z3t4
The interesting question is why the ODBC driver needed to be updated and why
it was such a hurry. Probably because he/she wanted to see if it worked before
business opening... And when he/she was not allow to do that maintenance on
off hours, then he/she tested it on a less important internal system. The
thing with ODBC drivers is that they deal with things like bigin support,
which cover edge cases that you only see in production.

~~~
asdfasgasdgasdg
Odbc/SQL newb. I know what begin does, but I don't know why you'd only see it
in prod. Surely you should test the same codepaths in staging that you plan to
execute in prod?

~~~
NikolaeVarius
Generally speaking from experience, most non-production facing testing does
not accurately reflect what is actually happening in production in terms of
actual queries run and volume of it.

A codepath which is perfectly fine in testing/staging could kneel over
immediately against a production workload,

~~~
asdfasgasdgasdg
That's all well and good, but begin is pretty basic? Like, that's how you do
transactions in SQL, right? We're not talking about a codepath that's working
in testing and falling over in production. The person I responded to was
suggesting a codepath that's not even tested in testing, and I was hoping to
understand why begin would be such a codepath.

~~~
z3t4
I meant biginT, for example the prod might have over two billion of something,
where local dev have not.

------
arctangent
This is why we have release management processes

[https://en.wikipedia.org/wiki/Release_management](https://en.wikipedia.org/wiki/Release_management)

------
technion
I need blogs like this to remind myself that there in fact are actually sane
organisations in the world. It's a glimmering ray of hope against my own
experiences, which more and more would suggest the writers pushback in that
chat room would just get them considered a problem employee.

------
rawoke083600
My ex-boss use to say... "Don't you like your weekends ?" whenever somebody
wanted to deploy something on a Friday :D

------
nojvek
It’s about process + team and not about individuals. Far too many times I’ve
seen companies over praise/reward certain individuals at the expense of
demotivating the team around them that makes things possible.

I’ve found a lot of things solved by a decent review and deploy process.

My precious company just went through SOC2 certification. SOC2 is mostly about
change management. Who can make changes to prod? Is every change approved by
the right personnel? Is every change auditable? Etc

Mostly boils down to using git(hub) properly and having a sane
deploy/monitoring/revert system.

Doesn’t matter if you’re the CEO or a VP or a yolo manager. You can’t deploy
things without oversight. If you want to deploy on the weekend you need
approval from the opslead for that system who is responsible for keeping
things alive.

On the weekend, no one should be working, even opsleads should only work if
they are paged. Only broken things that really affect customers are fixed on
weekends. No new features. Period.

------
crtlaltdel
ha! last year i was at a shop that had a dev manager who snuck in a stored
procedure into a prod db. the arch called for an instance per customer, with
each using the same sprocs. when one customer suddenly had prod fails it
caused all the teams who had pushed code to burn cycles tracing the issue (the
effects were system-wide and this was the biggest customer). we diffed the
sprocs and saw what was up. the correct code was restored and the problem got
fixed. in the end the dev manager in question didn’t have much of a reason for
making this change without telling anyone.

------
klausjensen
...I'm probably missing something obvious, but what is ".so"?

I have read the post, btw (liked it - I usually like Rachel's posts) - but
still don't understand what .so is.

~~~
wickedwiesel
In Unix these are shared object libraries.

In this context, I believe .so is the file extension for extensions that can
be loaded by web servers during runtime. I think the person in the story was
trying to change the Apache2 configuration of all FE servers to load the
wanted client library / .so file.

\-
[https://httpd.apache.org/docs/2.4/mod/mod_so.html](https://httpd.apache.org/docs/2.4/mod/mod_so.html)

------
darepublic
Bringing up problems only shortly before launch sounds damn familiar

------
neya
On another note, I hate posts from Rachel. My opinion is from reading a dozen
posts from her blog by now. It's usually about complaining of fellow
colleagues and her employers. I would really hate to be her colleague. I've
never seen a post where she admitted she fucked up. It's always about how
everyone else is awful and how she is the victim in all this. I really hate
this toxic attitude and I would really hate to have a toxic colleague who
instead of grabbing her teammates for coffee and giving them constructive
feedback (and keeping company events confidential), shames her colleagues
publicly.

~~~
throwaway6845
While I don't enjoy the blog either (partly for the reason you cite, and
partly for the unspoken assumption behind many posts that FAANG-like companies
are a good thing and a good working environment), I wouldn't phrase it like
that. There is enough misogyny on the internet that starting a critique with
"I hate" is not going to get your point across lucidly.

~~~
magduf
>partly for the unspoken assumption behind many posts that FAANG-like
companies are a good thing and a good working environment

Sorry for sidetracking, but are they not good working environments (note this
is a separate question from whether they're a good thing)?

I've been planning to try to get a job at one of the FAANG companies, and have
basically assumed it would be a good working environment based on the
reputation, the perks offered, and my prior experience at another very large
tech (but not exactly FAANG) company, which was that the working environment
was generally very good there though I didn't really like how that particular
company operated in the marketplace.

~~~
ThrowawayR2
> " _Sorry for sidetracking, but are they not good working environments (note
> this is a separate question from whether they 're a good thing)?_"

Depends heavily on which team you join within a company. FAANGs are not
monoliths; they are collections of business units cooperating (or sometimes
not) with quite a bit of variation in culture and work-life balance.

------
enchiridion
Why are the comments here so hostile? Seems awfully critical, even for hn.

~~~
dijit
I (cynically) believe it's due to very low attention span.

At least, I'm guilty of that, I skim articles and read comments. But this post
is not skim-able(?) and thus feels a lot more dense than usual.

However, I went back and read the whole thing and it's certainly worth the
read.

~~~
rachelbythebay
Pretty sure it’s the domain name.

Also the time of day/day of week.

But mostly the domain name.

~~~
wgerard
For what it's worth, I personally thoroughly enjoy reading your articles. I
wouldn't consider myself adept at operations by any means, but your articles
are extremely approachable and I always enjoy reading them.

I always click on the article when I see it's from you.

------
iamleppert
Just reading that, the tone of this person made me cringe. I definitely would
never want to work with him/her. I can feel the negativity and attitude in
almost every sentence.

The right thing to do would be to help this person achieve their goals while
maintaining reliability of the operation. Calling a co-worker who is just
trying to get something done a rando is the epitome of toxicity. She wouldn't
last 5 minutes with that attitude in my organization.

~~~
dang
Please don't cross into personal attack here. Besides being mean and against
the rules, it makes for boring reading.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
craftyguy
> it makes for boring reading

I didn't find this readers interpretation of the author's behavior to be
boring. I can see how she may come across as being overly combative (though I
agree with her goal).

Your frequent comments about what you find to be 'boring' _is_ boring.

~~~
dang
I mean the word 'boring' in the HN sense of not intellectually interesting.
Something can still be exciting even if it's boring in this way. It's boring
because there are only so many ways to denounce something as "the epitome of
toxicity", "this person made me cringe", and so on, which means the patterns
of denunciation get recycled a lot. That makes them predictable, and this site
exists for things that aren't predictable.

Another way of putting it is that indignation tends to melt down to a few
formulations that get repeated a lot, where intellectual curiosity looks for
what is new and different in a situation, and then has a new and different
reaction to that.

Since we want HN to be a site for intellectual curiosity, it's important not
to feed indignation too much. It quickly overwhelms curiosity if allowed to
(especially on the internet); therefore we need to moderate it. You're right
that the moderation comments are boring; that's because they're also
predictable. If it helps at all, they're even more tedious to write than they
are to read.

------
iamleppert
Calling your co-worker a rando? Very unprofessional.

Why not teach people the proper way to release things to production and avoid
the name calling and negativity?

~~~
shrimpx
I agree that it might be offensive to the actual person who is being singled
out in this article, if they’re reading the article. It also occurred to me
that this person is fictitious or semi fictitious in which case it’s not just
inoffensive, but accurate, to call them a rando.

