
Don’t Do This in Production - pattrn
https://stephenmann.io/post/dont-do-this-in-production/
======
wpietri
I want to underline this bit:

> they hired a team of developers without having a technical person on staff
> to vet them

For any non-technical entrepreneurs reading this: please don't do this. If you
aren't competent to hire developers, please borrow or rent a few trusted
technical experts and use them as a hiring committee. Otherwise you are less
likely to hire the _best_ technologist, and more likely to get the _most glib
and appealing_ technologist.

Over the years I have met far too many smooth-talking consultants and would-be
employees. And I've seen them cause enormous garbage fires when managers hire
beyond their ability to evaluate. The size of these garbage fires are a direct
result of what got them hired: if someone is good at telling managers what
they want to hear, they can go a very long time saying, "Yup, we're building
it and it will be great! Just you wait and see!"

And then, no matter who you hire, demand a "ship early and often" schedule.
Ship to internal users. Ship to alpha users. Ship to external testers. Heck,
ship to just person for just one narrow use case. The earlier you can start
seeing real-world success or failure, the earlier you can course correct.

~~~
ChuckMcM
I was going to call out this quote : _If you read this far hoping for an
answer, then I’m sorry: I don’t have a simple one. This is a difficult problem
to solve._

And yet there is an answer, it is _hire an experienced engineer_. And the
author glosses over the problem by not realizing they "solved" the problem by
hiring him.

It is the bad calculus where one "experienced" engineer is going to cost you
maybe $150K - $200K and yet you could get anywhere from 3 to 10 junior
engineers for that depending on how far out you go.

It would be awesome of there were some sort of certification for "experience"
sort of equivalent to the guild master status, it isn't an education thing it
is a "oh yeah I've seen that before, lets not do that here's why" kind of
thing.

~~~
yen223
If you aren't technical yourself, how do you separate an experienced engineer
from one who, well, isn't? This was the fundamental problem that the company
failed to solve, and frankly I don't think there's any easy solution around
it.

~~~
memset
Agree with this. Same applies to any expert: how do you identify an
experienced marketer? Or accountant?

~~~
jedharris
To hire someone with expertise to solve a problem in a domain where you aren't
an expert: You find people who claim to understand the domain, and then
discuss the problem with them in depth. Your goal is not to evaluate them in
general but to understand how they analyze the problem and how they'd deal
with it.

You may have to hire them as consultants. You have to dedicate a day or more
to discussion with each of them -- maybe considerably more if your problem is
complex. You have to reveal information about your plans and/or the condition
of your systems that may make you anxious or embarrassed.

After you've gone over the problem separately with five or ten putative
experts, you will have a pretty good idea which of them understand it (if
any), and you will also know how to ask much better questions going forward.

If necessary repeat this process, using your better understanding to find
better experts and gain a better map of the territory.

Once you can rank expertise in your domain reliably, you might hire one (or
more) of the known experts, but the best ones will be hardest to hire. Or you
may use your new meta-expertise in the domain to look for more junior hires
who have an unusually good grasp of your issues. Also, you may be able to get
the best of your experts to act as advisors.

This takes a lot of time and may cost some money depending on the domain. But
not doing it will take more time and cost more money in the medium to long
term (and even in the short term sometimes).

~~~
hyperpallium
AKA become expert.

I wonder if the people who become fantastically successful simply have the
ability to evaluate expertise, without having any of it. i.e. by "reading"
people, in some sense.

This relies on the "expert"'s own evaluation of their expertise, which
mightn't be accurate, because of Dunning–Kruger, or conversely, self-doubt.
So, one would need to "read" those too.

~~~
jedharris
You definitely can't trust people's self-evaluation either way. Lots of really
skilled people are self-critical. "Reading people" of course helps but it is
nowhere near enough.

But you don't have to become an expert. If I hire a plumber or carpenter, I
_cannot do_ what they can. But with enough work I can understand which
plumbers or carpenters will do a good job.

------
jseliger
_“Move fast and break things,” they said. It turns out that’s a pretty bad
idea when your business relies on a small number of large customers. Broken
products tend to scare them off, which in turn tanks your business. There’s a
lot to be said for building things that work, but “move slowly and steadily
towards a goal” just doesn’t have the same ring._

 _In reality, there’s a balance between moving fast and and moving slow. It’s
difficult to communicate that balance because every type of product demands a
different balance. I suppose that intuition comes from experience, which is a
terrible answer for someone trying to learn._

I'm guessing this continuum also depends on the nature of the business one is
in. In a fast-moving, consumer-oriented world like social networking, "move
fast and break things" is probably very good advice.

In a slower-moving, enterprise-oriented world, "move slowly and steadily
without breaking things" is probably the right orientation. Those kinds of
companies probably die in fast-moving consumer land.

A while ago I read an article about the programming teams that work on the
space station, on satellites, and that sort of thing. Alas, I cannot find the
link right now, but the gist was that those teams move very slowly and work
very hard towards correctness. Any bug found is generalized and their entire
code base scoured for similar bugs. That's a different environment than social
networking.

~~~
tetha
Most enterprise customers I've dealt with don't even care about things
breaking that much, they care about control. That's the much bigger thing.
Things break. Let's just make sure we don't break things while our customer is
dumping silly amounts of money into an EU-wide advertisement campaign. Or we
tend to freeze updates during christmas, due to the christmas business of our
customers.

With those guys, it's fine to execute risky changes on productive systems, as
long as those changes are tested, communicated, scheduled to a non-critical
time of the customers and include backoff-plans.

But yes, this naturally slows things down, or makes systems more complex and
expensive due to redundancy and staged rollouts.

~~~
sbinthree
Enterprise wise, they are very forgiving in my experience unless it affects
_their_ customer. Then they are very unhappy, understandably so.

------
peterwwillis
Bugs will always happen in production. The important part is not developing
everything correctly. "Move fast and break stuff" is a decent philosophy, _as
long as you can learn from mistakes and fix recurring problems_.

When a bug in your application presents itself in production, something very
specific should happen: 1) identify the symptoms, 2) isolate the causes, 3)
address the issue in production, 4) perform a postmortem to identify the next
steps, 5) fix the problem's causes throughout _the entire supply chain_ so it
cannot happen again.

A lot of organizations get stuck only doing steps 1-3. That's not good.
Operational processes are just as important, if not more so, than development
processes. They're what ensure that when you find inefficiency or bugs, that
they not only get fixed, but that they never happen again. This means actually
making policy changes, training changes, etc in addition to just fixing code.

This is the heart of lean manufacturing that supposedly influenced a lot of
modern software development methodology. But some people may have forgotten
that the lean manufacturing chain doesn't end at development.

~~~
ryandrake
> Bugs will always happen in production.

This is not some universal law. It’s true only when the project/team is not
willing to make the trade-offs (like development speed) needed to deploy bug-
free code. For most software it’s better to release with bugs, but not so for
all software. For some software, like safety critical projects, things
launched into space, etc, it’s better to take your time and release without
bugs.

Besides, good development practices, good requirements, good change control,
and good testing can prevent the vast majority of bugs from reaching
production. Sadly, this idea that bugs in production are inevitable is
pervasive in industry.

~~~
ljm
In order for that to happen, you would need perfect practices, perfect
requirements, perfect testing and perfect ‘change control’, and an inhuman
ability to never make a human mistake in the understanding of any one of those
things within your team of equally perfect people, so add perfect
communication on to the list.

Anything less than perfection at any stage will be a vector for bugs, whether
that bug is a syntax error, unexpected behavior, or a fully tested and
functional feature that doesn’t meet expectation.

Mistakes are inevitable and not necessarily bad when you handle them well
(move fast and break things says nothing about what you do with all the
breakage, so feels immature). Calling them bugs doesn’t change that.

------
dsr_
"Move fast and break stuff" is a decent philosophy if and only if the
consequences of breaking stuff are survivable.

If breaking stuff means that your website looks weird, that's survivable.

If breaking stuff means that performance sucks for a while, that's survivable.

If breaking stuff causes unavailability during a critical period of end-user
demand, a few incidents might be survivable.

If breaking stuff causes your company to have a terrible reputation for
privacy, security, or competency, that might not be survivable.

If breaking stuff causes your company to divulge financial information, that
might not be survivable.

If breaking stuff ends up costing your customers any significant amount of
money, that probably will not be survivable.

~~~
ryanlol
>If breaking stuff causes your company to have a terrible reputation for
privacy, security, or competency, that might not be survivable.

I reckon this stuff is particularly survivable. You can be like Linode, fuck
up everything over and over again and still do just fine.

~~~
coldtea
> _You can be like Linode, fuck up everything over and over again and still do
> just fine._

Source? Because every time I've read about Linode they had glowing user
reviews, including on HN...

~~~
freehunter
It blows my mind that people still give glowing reviews to Linode when in 2013
hackers breached their systems and stole credit card numbers and customers
reported their credit cards were being used maliciously and Linode got away
with saying "nuh uh":
[https://www.theregister.co.uk/2013/04/16/linode_breach/](https://www.theregister.co.uk/2013/04/16/linode_breach/)

Or again in 2015 when Linode's poor security lead to PagerDuty being hacked:
[https://www.securityweek.com/how-attackers-likely-
bypassed-l...](https://www.securityweek.com/how-attackers-likely-bypassed-
linodes-two-factor-authentication-hack-pagerduty)

You can find some more discussion in previous HN threads about it:
[https://news.ycombinator.com/item?id=11136399](https://news.ycombinator.com/item?id=11136399)

------
_hardwaregeek
This something I don't understand about certain startup founders. If you're
starting a company, you should have at least one absolute monster of a
developer. Someone who is capable, communicative, and with a deep pool of
knowledge. Someone who can attract, hire, and lead other good developers. Not
your buddy Steve who made an iPhone app or two. Not that developer who has
some PHP experience and built a React app once. A real, terrifyingly good
developer. The aphorism "first class people hire first class people; second
class people hire third class people" comes to mind.

Granted, I'm a pretty inexperienced developer, but I see companies being
founded by people who are even less competent and less knowledgeable than me.

And sure, finding a competent person is hard. But it's like the movie Seven
Samurai. Can't afford to pay a samurai? Find a hungry one. There are plenty of
developers who are young, but still good. And if you can't even do that, then
take any and all money you have, and pool it to find one or two (or seven)
developers who can find other competent people.

~~~
eropple
_> There are plenty of developers who are young, but still good._

I was young, and I was good. (Now I am moderately young, and I'm very good.)
But I am a better talker than I was a developer. Only recently have I really
become as good a developer as I am a talker. How are those startup founders
supposed to evaluate me?

------
i000
So a team of inexperienced developers made a system that mostly worked with
some hiccups, could be debuged by an expert in 1 day, and fixed on a tight
launch timeline. Looks like a success story!

~~~
pattrn
It was definitely a success story. The company, their customers, and my
company all benefited. It wasn't all rainbows (the story is a bit longer and
more complicated), but it worked out well for everyone in the end.

------
altitudinous
Many developers are commenting in here trying to fix a problem. I am a
developer but I now run a business. From a business point of view, there is no
problem here. Code has been delivered and it is running in production
successfully. It might not have arrived at that point in the most efficient
manner, but the point is that the business has executed successfully here.
There is no issue. This is a lot further than most businesses get. If you see
a problem to be fixed here then you are looking at a small picture of the
business.

~~~
alecbenzer
Are you discounting that technical debt is a thing/matters?

What gets delivered isn't just a set of features, it's a piece of software,
which implements those features, but might have a whole bunch of baggage
attached which can make it really hard (or easy) to change those features or
add new features in the future, or to allow the software to adapt to changing
external conditions.

I don't think this is specific to software. E.g., I'd imagine it'd be possible
to build a building a lot faster if you didn't need to worry about it not
collapsing in five years.

\---

Though to be fair, the mere presence of technical debt on its own is not a
problem; there is some amount of technical debt that's appropriate, and it's
hard to know exactly where that is. You certainly can't get by just ignoring
it completely.

------
ballenf
There's a great book _Apprenticeship Patterns_ which gives real answers and
advice to developers. It addresses the situation described in a few different
ways and offers different strategies. Most of them boil down to how to pick a
team where you will learn the most quickly while not drowning.

[https://www.amazon.com/Apprenticeship-Patterns-Guidance-
Aspi...](https://www.amazon.com/Apprenticeship-Patterns-Guidance-Aspiring-
Craftsman/dp/0596518382)

~~~
mazatta
Of all the books I recommend for junior/intermediate developers, this is the
one I demand they read. I wish the the free online version was still
available, but it doesn't appear to have survived O'Reilly's switch to the
subscription model.

------
wiz21c
One thing I see amongst many of developpers, is that they write code like it
will work in production. OTOH, I tend to write code that will crash nicely :
good error reporting, ability to continue after a crash, write code that
handle only the data it needs, make sure queries address the smallest possible
set of data, log a lot. Interestingly, that's a lot of code I have to write
anyway because it helps me to write my programs (I mostly don't use
debuggers).

~~~
Raidion
I agrret with you, but want to say I think it's important to write code that
is "brittle" enough to just satisfy the requirements. I've seen a lot of "edge
case that probably never happens" code that ends up causing problems down the
line because it does get hit, but for a unexpected situation and ends up
mangling data.

Especially in the paradime of black box/microservices, if something comes in
that doesn't line up with a known senario, there should only be enough code to
log the data and raise an alert/error. Having something bad happen, and
handling it so it becomes 'less bad' and doesn't get flagged, escalated, and
tested properly is going to cause issues.

~~~
wiz21c
>>> I think it's important to write code that is "brittle" enough to just
satisfy the requirements

100% agree and the idea behind my post was that writing code in such a way
requires a lot of experience and an appropriate mindset.

------
hashrate
> all of these developers lacked experience.

This is nothing knew unfortunately.

Working for large corporations I've often notice this :

Either the devs on the project don't have the required experience and will
deliver poor quality code creating immediate "legacy" , or either it'll be
outsourced to another company with no governance plan to review the code
quality creating a "black box".

Because there is very little standard and regulation in software industry
about competence, you can have two devs claiming they are "Software Developer"
with one able to deliver a full project by himself the and the other not
capable of creating a simple web page in HTML.

Hence , I've noticed very little company understand the importance of "tech" ,
"IS Governance" and investing in their staff in general, they often see "tech"
as a constraint like : "We have to do a website for our customers , otherwise
we'll loose market share" and not "We have to seize the opportunity to create
a platform for our customers, it will drive growth massively"

This is sad , but this is pretty much the standard in corporation these days.

------
tlb
There's no reasonable way to write code suitable for "production" in general,
because there are so many conflicting requirements. Just among internet
services, components need different kinds of robustness for content websites,
communication, e-commerce, email, banking, social networks, internal corporate
services, and more.

There do exist components suitable for most environments, like Postgres, but
usually at the cost of 10x complexity over what'd be needed for the pure
functionality. That engineering is worth it for things like databases that get
used everywhere, but not for any custom component.

------
crazygringo
Can someone give an example of a blog post that gives example code yet warns
"not to do it in production"? I've programmed and read about programming for
decades and can't recall ever seeing anything like this.

It's hard for me to take this article seriously without even a single example
-- I can't tell if this is a strawman or not.

Is it about code that isn't thread-safe? That relies on undefined behavior?
That doesn't scale? That has race conditions?

"In dev" and "in prod" can mean so many different things (is it about scale?
or running on different OS's? or running unattended? or running on different
hardware? or running compiled?), out of any particular context they're
essentially meaningless, and in case particular case, it would seem to be the
particulars that matter...

~~~
thinkingemote
First result from DDG.
[https://blogs.msdn.microsoft.com/canberrapfe/2013/02/16/powe...](https://blogs.msdn.microsoft.com/canberrapfe/2013/02/16/powershell-
web-access-very-quick-start-yep-powershell-in-your-web-browser/) the thing in
that example is that a thing is not properly configured but good enough to get
started with in development.

Second result: [https://stackoverflow.com/questions/1475297/phps-white-
scree...](https://stackoverflow.com/questions/1475297/phps-white-screen-of-
death) describes how to put debugging statements in the code which is good for
local development but which shouldnt be kept for production.

------
pattrn
This blog post describes my motivation for writing a blog. It's mostly a
written memoire for myself to remember in detail what motivated me to start
writing, but I thought HN readers might have some good feedback, so here it
is. Would love to hear any stories you have of production fiascos.

------
honkycat
> If you read this far hoping for an answer, then I’m sorry: I don’t have a
> simple one. This is a difficult problem to solve. The solution is too large
> for a single blog post, changes every day, and differs subtly for every
> project.

Crazy idea, what if we found one of these people who writes these blog posts,
an especially well vetted and knowledgeable one. Maybe even one who writes the
frameworks and software these people are using. _cough_
[https://pragprog.com/book/phoenix14/programming-
phoenix-1-4](https://pragprog.com/book/phoenix14/programming-phoenix-1-4)
_cough_

What if we took this person, and paid them money to write a bunch of blog
posts about on topic, in a way that it read in a linear fashion? We could even
maybe hire someone to help them, to proof read and make helpful suggestions.

We could give the knowledgeable developer a year or so to write the blog
posts, then print out the blog posts, bind them together with glue, and SELL
IT!

People could pay money for the well researched, well edited series of blog
posts about a specific topic written by the expert. I think I'll name this
blog-printed-out-onto-paper... a honkycat. After it's inventor.

\---

Sarcasm aside, I find people's insistence on googling and sifting through
half-thought-out blog posts infuriating. I am a HUGE proponent of a well
written book about a software product. Compare "Programming Elixir" to their
free online book and documentation. It's night and day. And I can always look
in the same place for reference, the book. Written by Dave Thomas.

A book has: ( At least ) one author, one editor, and a publisher. All of these
people work hundreds of hours to produce a useful manuscript. A blog post is
something a script kiddie shits out because they're trying to get their first
programming job.

Yet people say the same thing: "A book?!? I can just find it on the internet"
and do the same thing: "I'm sure I'm smart enough to just start writing this
software without any prior experience in the language, or forethought" and
make the same mistakes over and over.

Skip the blog. Read a book.

~~~
aidenn0
Finding a good book is not particularly easier than finding a good set of blog
posts. While the quality of books is on average significantly better, finding
books where the material was written in the past year is much harder.

Particulary in the trendy JS world, where there might not be a good book for
the stack you are using, and even when there is, you ask questions online and
get responses that are roughly "It hasn't worked like that for 6 months, why
are you trying to do it this way?"

You give the example of the Phoenix book; I don't know if that's what this
team was using. If it was not, the fact that there is a good book for that
stack is not particularly relevant. If they were using Phoenix, then the fact
that there is not a good book for most stacks would explain why they wouldn't
consider looking for one for the stack they are using.

~~~
shoo
> Particulary in the trendy JS world, where there might not be a good book for
> the stack you are using, and even when there is, you ask questions online
> and get responses that are roughly "It hasn't worked like that for 6 months,
> why are you trying to do it this way?"

[http://boringtechnology.club](http://boringtechnology.club)

------
lmilcin
I find this a result of pervasive command and control structures which
inevitably lead to low trust environment.

Something breaks, so the manager naturally thinks he must add a control that
will prevent the person or a from ever being able to do it.

Very soon nobody can do anything without approvals by managers so far removed
from actual decision that they don't even know the names of components being
subject to change. The change then must travel from the only people that know
what the change is but are absolutely barred from ever seeing production
environment to people that being so focused on operations don't have time to
actually understand what the change is.

Once you are being treated as untrustworthy it will show in the rest of your
work. Why code correctly from the start when you are not deemed trustworthy to
do it correctly and there is lengthy testing and approval process to make sure
you are not going to bomb your company?

The correct step is to recognize, that introducing more controls can very soon
lead to low trust between employees which is self-fulfilling prophecy. When
you are not allowed to do something because you are not deemed trustworthy to
do something, then soon you have no experience and then this serves as a proof
that the initial decision was correct.

~~~
tensor
On the flip side, it seems that many young devs think that they should be
allowed to do anything they want, without oversight whatsoever. I don't know
of any other industry where junior people have this sort of arrogance. Usually
people recognize they are junior and work to both learn from and gain the
trust of their seniors.

In development land, the attitude is "seniors are old, useless, and outdated"
and any form of oversight at all "removes trust." E.g. A young dev suggests
building their own database. After being told politely that this is an
enormous task full of seriously hard problems and that it's not feasible to
try, they yell about their "ideas being shot down" and that the work
environment "lacks psychological safety."

~~~
lmilcin
That's generally true.

What is important, though, is how the very young devs are molded the first
time they come to an organization.

I have long experience working for various enterprises. One outlier company I
worked for is Intel. Intel or at least the organization I worked for seems to
be aiming to hire almost exclusively fresh out of university just to be the
first one to "mold" the young engineer. Young engineers are immediately
treated as adults trusted with their projects. Managers (not all, but I would
say majority) seems themselves as shepherds rather than commanders. They will
ensure projects are assigned to engineers and that engineers create value and
they will try to debug if there is any cause of concern but will not assume
the typical "single point of contact, single point of decision and authority
on everything" role seen in most corporations.

This of not true of all managers at Intel but it was very striking to me when
I joined the company with some 15 years of experience in dev under my belt.

Junior engineers seem to adjust to this immediately and seem to genuinely try
their best to do best and try to seek wisdom which there typically is a lot of
in the close vicinity.

------
hyporthogon
Copypasta from other codebases happen to work better in non-production
environments because the input space is smaller in virtually any non-prod
runtime space than in prod.

This makes for a larger intersection of inputs to the copypasta that produce
outputs that are expected by the blogpost/googling developer and inputs to the
copypasta that produce outputs that are generated at non-prod runtime.

When you don't quickly see what's semantically wrong with the code (the
problem mismatch) because you pasted it and it just worked (maybe with a few
small tweaks), then you quickly forget about it and black-box the snippet in
your head. Then, when input space gets real-world and things break, because
the copypasta never had much cognitive gravity (because you didn't think much
about it), it doesn't attract log dump elements in your log-scanning brain as
much as stuff you had to think through yourself does.

I guess you could automate some static checks for this by some massive trawl
of snippets from a few websites (at least popular SO posts in the same
language?) and finding verbatim matches in your codebase. Sounds massively
impractical to do this at scale though. Does anyone know tools that do this?

------
exabrial
"Move fast and break things" only works if your company has a stranglehold on
a monopoly and your user base is used to bugs in production. Terrible advice
for just about everyone else. You want your users to have the best experience
possible, rather than do your QA process for you.

EDIT: I should say my point is: "moving fast" and "testing things" are goals
that aren't contradictory. You can and should do both.

------
sattoshi
This post was completely not what I expected it to be.

The "do not use in production" is the programming equivalent of taking a
strong stance on an issue but then appending "but hey, I don't know" or "don't
take my word for it".

They completely defeat the purpose of the entire statement and in the
production comment's case, the code samples.

Let's ask a different question, what am I supposed to use in production? I had
an email discussion with some blogger who wrote about a very narrow problem
which I needed to solve. Their code samples worked, but poorly. The entire
thing was a dumpster fire but it solved the problem and were the only one I
could find after days of Googling and Stackoverflowing.

I did not care about their "do not use in production" comment. I needed their
solution in my production system. What was I supposed to do?

Sorry for ranting, but I believe it is irresponsible to put code out there
which shouldn't be used except for when it is the point.

------
arendtio
Some time ago I was writing some software in Go in my spare time (just for fun
and I could spend my time how I liked it). As I wasn't completely sure what it
should look like in the end, so I thought it would be reasonable to skip
writing tests in the beginning and just write the code and add tests afterward
where they would be necessary.

Fast forward I had to write the whole code twice. The second time I wasn't
completely sure how it should look like either, but I put a much greater
emphasis on interfaces and modularity. That way I could guarantee that parts
of the program were actually doing what they were supposed to do, which made
the bug hunt so much more effective.

So sometimes the longer way is actually the faster one and experience matters
when it comes to recognizing such situations.

------
Yokohiii
Building a good team is hard. Yet companies think they can hire an engineer
which had a senior position somewhere and be fine. Senior or 10y experience
can mean literally nothing, it doesn't qualify for a central role in a team,
nor does it tell anything about quality. Expecting something grand from a
bunch of rookies is simply dumb. Writing better blog posts or smart essays
won't change anything. A rookie needs to follow the wrong path and fail with
it. Good mentoring can speed things up, but you still need to let them fail.
Team building is vastly underdeveloped in software engineering, it is not even
a serious thing. Yes if you start out with a good guy that carries the
newcomers, then you're golden. But that is mostly just luck.

------
anon1253
Even for an experienced programmer this stuff happens, and it's the most
frustrating experience I've ever had. It's the stuff that happens when
management and C-level just "wants stuff" to be buzzword compliant. And, every
day/week/month the requirements change. This weeks feature X has "top
priority" and then amnesia kicks in and now feature Y is the next best thing.
As a programmer-now-manager I try to shield people from this, trying to find
the gems among the rubble of business meetings. But it's tricky, so this
really resonated with me:

"I have an overwhelming empathy for developers in this position. They have
more information than they will ever need, but it’s completely disorganized.
It’s like trying to build a ten piece puzzle, except you have to find the ten
pieces within a pile of 10,000,000,000 pieces, all of which are square, and
you don’t know what it’s supposed to look like at the end. Good luck."

Even with a team of highly experienced developers things quickly turn to a
mess if it's not perfectly clear what to work on. I don't mean things like
"button should go 1px north to align with header". That's trivial. In fact, I
would go so far as to say: most production-ready programming is trivial.
Trivial in the sense that someone, somewhere, solved it in a way that's
correct. But, when it is not /crystal/ clear what the requirements are, what
the business case is, or what the edge-cases look like: oh my. I've been in
the incredibly unlucky position of only being in settings like this. Start-
ups, academia, R&D positions. And, as a result even with the best intentions
and the best teams you'll still produce production-quality garbage. Mostly
because incentives were misaligned and there was no coherent vision. Agile
story-driven development only works when there is leadership that has the guts
to say "no", and when there is something well defined to work on. Sometimes I
yearn for some CRUD webstore ... at least then the things are knowable.
Anyway, just my rant. I'll probably go the meme-way of becoming a
plumber/welder/etc at some point ;-)

------
mmt
> helped mentor their developers

This is typically missing from startup culture.

Anecdotally, mentorship improves everyone involved, but I don't know if
there's data out there showing if engineering mentorship within a startup
improves its chances of success (or is otherwise beneficial [1]).

To some extent, groups like YC participants and alumni end up being mentors to
each other, but I doubt that extends beyond the founders. Might the OP have an
opinion on something like a rent-a-mentor program/startup?

[1] Not that certain benefits, like productivity, would be enough to motivate
a change, as evidence by the open plan office situation

------
ddebernardy
> If you read this far hoping for an answer, then I’m sorry: I don’t have a
> simple one. This is a difficult problem to solve. The solution is too large
> for a single blog post, changes every day, and differs subtly for every
> project.

 _[Cough]_ \- Hire a few more (expensive) experienced developers? It might not
be a silver bullet, but it goes a long way towards avoiding the situation
described in the blog post. Also, get help from a trusted technical friend to
hire the initial one.

FWIW I see something similar with almost every client I get in my own neck of
the wood. Growth is flat. Turns out nobody is worrying about whether they're
building a product anyone wants - they're doing it on paper but nobody is
actually talking to end-users. Marketing is under-delivering. Turns out it's
all juniors worrying about bringing traffic without wondering if it's the
right type of traffic; or they're not measuring anything, plain and simple.
Sales aren't selling enough. Turns out it's all juniors who aren't selective
or systematic enough; or sales are doubling as pseudo-project managers and/or
pseudo support-engineers because nobody is managing operations. Projects take
forever to deliver. Turns out the project "manager" holds nobody accountable
to deliver any specific task let alone by a certain deadline. The list could
go on, and on, and on.

------
viburnum
Naw, you can totally write garbage code and still get acquired for $100
million. Happens all the time.

------
senderista
MSDN used to make their code samples as simple as possible for educational
reasons, until they realized everyone was just pasting them directly into
production applications. So they started adding error handling, etc.

------
platform
Doesn't this highlight a continuous issue,that folks in our trade, do not have
any meaningful and up-to-date professional certification?

~~~
alecbenzer
Is there a clear indication that certification helps in other fields?

(Admittedly this seems like it'd be very hard to verify, since most fields
either legally require certification or don't have it, and there's probably
tons of other variables that would get in the way too.)

------
brian-armstrong
Another angle to look at this through is that they hired a team in spite of
only having business, non-technical minded people. It all worked out in the
end when they added consulting to clear up production issues near launch. From
a business perspective this is surely a win in all ways, even if not as
efficient as it could be. Not every business is a Google.

------
yeukhon
> They hired the first developer, and he vetted the second developer, and so
> on until they had a development team

This is hard for a startup in their early stage. Not sure how mature the
company was, but let's assume it was young enough to have this very problem.

Direct team hire is a double-edged sword. It works well when your team is very
competent and good at weeding out false positive. Basically, when you have a
bunch of senior people with maybe a couple junior and mid-level teammates. A
normal distribution is actually perfect for a team. Having a direct contact
with your future teammate allows you to know the person ahead of time,
especially for a very team. Furthermore, direct hire means you are hiring not
just a generalist. Your SWE may not know what to ask a network engineer, even
from a coding point of view. The downside is what happened in the story.

The way I'd like to go about it is to make sure we have two interviewers per
round, even from the same team would be far better than one person vetting.

------
janwillemb
> Before I go any further, I’ll continue my story.

It seems to me that that sentence has no significance at all and can be
removed safely ;)

~~~
pattrn
Point taken, and sentence removed. Thank you!

------
jahewson
If you’re an amateur or a hobbyist then these excuses are all just fine. Stuff
is hard. But a professional should not being doing this sort of thing - we’re
literally talking about copying something labelled _do not do this_. The
moment you ignore that warning is the moment you’ve forsaken your professional
development. This problem only compounds.

That said, a professional can still cut corners as and when appropriate (and
clearly label and explain such things) but that’s not what’s happening here,
is it?

The advice I would give in this situation is simple: _stop and think_ about
what you’re doing. If you can’t do that, you’re not a professional, you
certainly shouldn’t call yourself an engineer and without that skill you stand
no chance of improving.

------
memset
This is interesting - many of the comments here are correctly asking "what if
you don't know an expert to help vet the engineers and the system?" I feel the
same way when trying to find dentists or accountants. I simply don't know how
to know who will perform completely!

So, here is my pitch: if you don't have technical expertise, and you realize
that you need someone to help understand just how the heck to get started with
technology, what to build, and determine if you even need engineers, then hire
me! I'll help!

[https://www.jaygoel.com/consulting/](https://www.jaygoel.com/consulting/)

------
ohiovr
I usually leave a URL for stack overflow answers or other advice as a comment
in what I'm working on. I know some areas of my code are more sensitive than
others. It is good to have an understanding of what you are doing of course.
Sometimes a stackoverflow link is removed simply because the mode of thinking
is wide spread in the application and I no longer need the note to it. Might
be worth searching for my links in the code when it is time for release incase
there is any funny business. Stackoverflow usually tells you if you "shouldn't
do it in production".

------
jarym
I dunno.. whenever I read ‘don’t do this in production’ I spend ages trying to
really understand why - only when I feel I’ve got a real handle on what the
pitfalls are would I go forward. I’ve done that ever since I started
programming over 20 years ago and I still do it now. Sometimes it takes me
longer to code a solution but I do feel I spend less time figuring out why
stuff breaks.

It’s not just experience. It’s about hiring curious and diligent people.

------
lisper
> At the end of the day, the best way to get more money is to have other
> offers.

Actually, the best way to get more money is to already have a job that you are
willing to leave.

------
Double_Cast
> _There’s a lot to be said for building things that work, but “move slowly
> and steadily towards a goal” just doesn’t have the same ring._

I'm reminded of one of my math professors. When a student asked him "How do I
do this problem?", the prof would always respond "Carefully and correctly".
And then he'd laugh at his own joke and give the student a real answer.

I thought it was funny. And I liked the alliteration.

------
daveheq
Copying and pasting internet code into production that says “Don’t do this in
production“ is not a lack of experience problem, it's a lack-of-reading
problem or a rushing-things problem; the latter is typically caused by
management putting deployment dates on features without understanding the
technical cost.

------
z3t4
When you publish some code you can be pretty sure someone will use it in
production, whether you advice against it or not.

------
paulie_a
I work with a consultant that dealt with emergency situations that had the
motto, "be brave, be bold, deploy"

In a different situation three of us figured out a solution for a critical
situation. We voted on deploying with the thought "well we can't guarantee it
will work, but it won't make things worse"

~~~
walshemj
They have not recently been working at the TSB :-)

The TSB a medium size UK bank had a very well publicised disaster - when doing
a one way no roll back possible switch from an legacy IBM system to a Java
based one

------
exmicrosoldier
This was a problem with people. Specifically, people who force people to
commit to a known deadline on something they dont know how to do.

Someone got promoted for delivering business value, and they were smart, they
left before it all blew up.

Get 'er done without context and fire them for failure is bad management.

------
stcredzero
_But what other option did these developers have? They had to deliver
something, and they had never released a production application before. So
they did what any sensible person would try to do, and they learned on the
job._

And yet, so many people push back on my complaints about the shallowness of
the education many recent CS grads are getting. No, it's not an irrational
fear of the next generation. It's an awareness that some large fraction of
recent grads sporting high GPAs don't have enough basic background knowledge
to know what they don't know and keep themselves out of trouble.

What would we think if companies let non-engineer/architect managers put
together teams of completely mechanical engineers and architects, who then
went on to build faulty bridges and machines that started to call apart?
People would be calling foul. Given the societal, privacy, and financial
consequences, it's way past time for the programming field to grow up!

~~~
user5994461
It's the same in every industry and every domain. A new grade needs a few
years of practice before he can contribute well to any real project.

However companies don't always have dummy projects for years, just to practice
and drop afterwards.

~~~
stcredzero
_It 's the same in every industry and every domain._

No. In most industries, groups of novices don't get to do something that gets
pushed out to production.

------
kristianp
"... fix issues as I went along. Without unit tests or automated deployments,
this took almost a year."

One of the first things he should have done was to get them Joel-test
compliant, otherwise he's wasting the money they pay him.

------
humzakh
"most fast and break things" only if you have an idea about the implications
of your actions and you are confident you will be able to fix the broken parts
which in production without losing revenue or customers.

------
syndacks
Stephen Mann: the subscribe button on your site is broken (at least on mobile
web)

~~~
pattrn
It should be a link to an RSS feed. I'll change it to link to my newsletter
signup form. Thanks!

------
daveheq
"The only clear issue was lack of experience"

"In this case, however, it had a nefarious twist: all of these developers
lacked experience."

Am I missing the twist here? It was already mentioned as the primary problem.

~~~
pattrn
Good catch! macns is correct: "all" should be bolded.

------
angersock
"If it's stupid but it works, it isn't stupid".

I liked the post, but I'm not sure that I agree.

~~~
detaro
To take one of the classics of bad copy-past examples: "If the 'use-PHP-with-
MySQL' example you copied is stupid, works, and also causes your production
database to be hacked due to SQL-injection, it probably was stupid"

The problem is to tell the difference between "ugly stupid" and "dangerous
stupid"

------
pytyper2
So, what shouldn't i do in production? This article was unhelpful.

~~~
pknopf
The snippet of code copied from a blog into production software was only a
small detail. You obviously missed the entire point of the article.

------
Ftuuky
The solution? Blockchain. /s

------
quickben
Summary: The author clickbaits the title with "don't do X". Proceeds with a
life story of his struggles ommiting what X is.

~~~
pattrn
Sorry about that. I guess I didn't really think much about the title sounding
like clickbait. The phrase is what motivated me to write the post, so I used
it as the title. Perhaps I should put quotes around it?

~~~
turdnagel
Quotes would probably be more "accurate," but I don't think they'd quell this
person's (unwarranted) complaints. I believe sharing your story was valuable
and I look forward to reading more of your blog.

~~~
pattrn
Updated. Thanks!

