
You Don’t Need All That Complex/Expensive/Distracting Infrastructure - kristianp
https://medium.com/@dmfutcher/you-dont-need-all-that-complex-expensive-distracting-infrastructure-a70dbe0dbccb
======
reacharavindh
Good old KISS policy.

Sure, Kubernetes, fully automated CI/CD, autoscaling, lambda functions are all
cool and very useful if you have a problem and paying users in that scale.

Majority of the applications even today in 2019 would be served well by the
reliable Nginx + (Python, C++, Rust, Go or other language of choice),
PostgreSQL and backups. If you keep things simple, you can even have a standby
mirror of the same infrastructure to take over if needed, and it would still
cost you less than spinning up nodes after nodes for the complexity of running
your own Kubernetes.

~~~
scarface74
Or just set up Elastic Beanstalk with AWS or something similar, hand it a zip
file of your code and let AWS configure your load balancer, autoscaling,
patching your OS and runtime, managing your database and backups, etc.

~~~
reacharavindh
What you suggest may be "simple" in terms of the amount of work the user needs
to do, but my comment is about reduced complexity and $$ because you don't
need them yet.

When you set up such an infrastructure like you proposed, there are still too
many wheels in motion(complexity), and even worse they are abstracted away
from you further and further. And, I'm sure the AWS bill is not going to be
pretty considering they do all those bells and whistles.

~~~
scarface74
You literally click a few buttons and it sets up all of the infrastructure.
Then you give it a zip file. The EC2 instance that it creates is a regular VM.

Elastic Beanstalk itself is free.

~~~
reacharavindh
It is true that Elastic Beanstalk is free, and is very easy for a developer to
use to begin with. However, If I were to advice a new startup on their
infrastructure, I would not recommend it for the following reasons. It is my
personal judgement. Others may prefer to go all in on Amazon.

1\. It is clearly a vendor lock in. You build your business upon all AWS
services, using an AWS orchestrator - good luck ever getting out.

2\. Abstraction. It is a controversial opinion. I'd rather have less magic
when possible and know what goes on underneath, and possibly have control
over. Now, a startup may choose to offload the duties of operating a
database(RDS), storing files(S3), performing backups(S3/Glacier) in the
interest of time and trade-off control, optimisation and $$. However,
considering the generalisation of simpler startups, IMHO operating a small
scale Postgres on a standard EC2 instance is not really much of a hassle...

3\. Cost - however small it may be in terms of actual dollars, AWS managed
services(S3, RDS, SES, SQS etc..) come with added cost over simply running
their open source equivalent yourselves.

Again, Keeping it simple - not in terms of what the user needs to do to deploy
their application, but in terms of what components do you need to get your
application running in production.

~~~
scarface74
_1\. It is clearly a vendor lock in. You build your business upon all AWS
services, using an AWS orchestrator - good luck ever getting out._

If you are starting a business, the least of your business risks is lock in to
AWS.

Besides that, you're handing a regular old zip file with your code to Elastic
Beanstalk, it creates a VM with the standard web server package installed
depending on your language. If later on, you need to go to another environment
(and in the grand scheme of things, few companies ever switch out their
infrastructure no matter how many Repository classes they write to abstract
their database), you create your same webserver set up in the new environment.

Every cloud vendor has a hand-me-your-code-in-a-zip-file-and-I-setup-
everything-for-you service.

 _However, considering the generalisation of simpler startups, IMHO operating
a small scale Postgres on a standard EC2 instance is not really much of a
hassle..._

Why would I want to manage my own Postgres database and worry about backups,
patching, etc. when I can pay someone else to do it?

 _Cost - however small it may be in terms of actual dollars, AWS managed
services(S3, RDS, SES, SQS etc..) come with added cost over simply running
their open source equivalent yourselves._

And that's more for your developers to manage, I'm sure if you can find some
gullible young developers to work 60+ hours per week pulling double duty as
developers and infrastructure operators it will be cheaper. I for one am not
willing to do that. Every hour that your developers are spending maintaining
infrastructure and doing the "undifferentiated heavy lifting" they are
spending time not creating features adding business value.

You're talking about adding more servers (and no redundancy) to save a few
pennies. I would avoid any company managed like that like the plague.

I actually have walked away from offers where the company was "on AWS" but all
they were doing was running a bunch of EC2 instances and hosting everything
themselves and expected their developers to be on call for infrastructure
failures.

On the other hand, I did take a job where the new manager's marching orders
were to move as far away from managing any infrastructure that could be done
by AWS. We have to have a serious justification to put anything new on EC2
instances instead of using Lambda or Fargate.

~~~
krageon
> If you are starting a business, the least of your business risks is lock in
> to AWS.

You are locking yourself into a cloud provider that is a part of a really
hostile country in terms of digital privacy. Your customers may not care right
now, but they really _should_. If you can sell that kind of behaviour to
yourself because you're "starting", then your customers deserve the kind of
business they are getting from you. To me it sounds incredibly unethical.

~~~
scarface74
So which provider should a US based start up use? If they use a colo which
country do you think it’s going to be in?

Seeing that all of the most valuable tech companies in the world besides the
ones in China are based in the US, do you suggest they host with Alibaba
Cloud?

~~~
krageon
There is more to the world than the US and China (both of which are terrible
choices). Outside of those places there are also cloud providers that give
good service at a reasonable price.

~~~
scarface74
Where would that be where there is a better startup scene than the US?

------
carlisle_
>Your users to don’t care how your code gets onto your server.

I don't like arguments like this. This is true for any number of tasks that a
layman wouldn't care to understand. You still will do them anyway, regardless
of whether or not your user cares. It doesn't take a lot of experience to see
the correlation between poor engineering practices and poor user experience.

Now, I don't think the author was using this argument to pitch poor
engineering, but there are good reasons for setting up resilient and
consistent build deploys, as he addresses. I think the only real takeaway is
"dont over-engineer your infrastructure" which is just an extension of
generally "don't over-engineer."

Every time infrastructure gets talked about on HN the conversation gets so
sidetracked. The real problem is that people don't think about infrastructure
in the context of a holistic view of a project. The hours you spend setting up
your project with resilient and consistent CI builds are many times more
valuable down the road.

I think people need to not be so afraid of spending time on infrastructure.
People already err on the side of spending less than more time on
infrastructure. I don't think it's wrong at all that the author spent _hours_
building an automatic build process for his project, especially because now
that he's set it up he can refer to it in the future. Maybe with his newfound
knowledge and experience on the problem means that he can leverage it with
little effort in the future. None of this is a bad thing especially if you
think about your engineering career on a macro scale instead of week to week.

~~~
0db532a0
I agree that CI is a brilliant investment. Also it can initially be as simple
as a git hook. However, I agree with the author’s point that you do not need a
Kubernetes distributed NoSQL megagalaxy when you have a number of customers
which you can count on one hand. I’ve seen so much time wasted on this sort of
thing.

One obvious benefit of the whole single/simple server idea is that you can
make changes to the system so quickly and easily, which is very important at
the beginning.

~~~
davidjnelson
Ya nosql can be useful at scale, but it's not a great choice for most new
projects. It really locks you into your schema and makes querying the data at
scale difficult outside of use cases you originally designed for. I've seen
aws architects recommending just starting with sql, and adding nosql later as
needed. Definitely agree with this approach for most use cases.

------
jdietrich
There's an obvious and overlooked middle ground between "massively complicated
infrastructure" and "a cheap VPS" \- charge more and use PaaS. Heroku and
Engine Yard are expensive, but they deliver a huge amount of value. You've got
better things to do than patch Nginx or deal with a crashed server at 4am,
just pay someone a sensible amount of money to run your infrastructure for
you.

~~~
pseudoramble
I agree. There are also options to make some of the things from the article
relatively easy. Off the top of my head, a CI/CD pipeline can be set up using
Gitlab , Azure DevOps, Bitbucket Pipelines, and I'm sure tons of others. They
do some of the heavy lifting and you still get the benefit of an organized
build that's easy to deploy to one of the services listed above.

And yeah, this may be layered on later when it seems like the idea has legs
and you'd benefit from this. It most likely doesn't need to be the first step.

------
m90
While I think you can definitely get too fancy with all of that sweet infra, I
feel this is missing an important point:

Once you have a working pipeline in place, you can focus on the features,
that's the whole point of the exercise: enabling developers to ship features
for users. If I have to SSH into some machine and restart a set of services
manually in the correct order, it will keep me from releasing features. I am
afraid / lazy / whatever. Spending time on this is not something that will
never ever reach the user in any way.

The user does not care if something is using Heroku, K8s, or something else,
yes, but the user definitely appreciates a product that can reliably iterate
on things without major downtime or similar problems.

It's kind of like saying, "well unit tests are nice and stuff, but does the
user ever notice?". I would strongly argue the user will notice unit tests in
the long run as they ensure the software keeps being maintainable.

~~~
collyw
> Once you have a working pipeline in place, you can focus on the features

Bullshit, its just another layer of complexity to be managed and that may
fail. My guess is that it will take more time maintaining it than it will save
you in the majority of cases.

~~~
m90
> My guess is that it will take more time maintaining it than it will save you
> in the majority of cases.

From my experience, this has almost never been the case. Exceptions were when
there was infrastructure for the sake of infrastructure.

~~~
afarrell
If a leader is worried about their organization having infra for the sake of
infra, then they should also worry that their user-facing product has features
for the sake of features—- features that complicate an interface and don’t
help a user get their job done. The solution is PMs, UX designers, and
engineers who know to ask “why?” when a project is proposed.

~~~
davidjnelson
That's a good point. Many things can be overengineered/overdone, including UX
design.

------
warp_factor
I saw so many places falling for this fallacy.

Instead on working on the core products, engineers would have "fun" adding
more microservices, redundant Kubernetes clusters, istio controllers etc etc.
(all of that with less than 10 users).

~~~
lykr0n
I had an argument with a co-worker about this. He wanted to use Kubernetes, so
I asked him what problem he was trying to solve. No answer- just wanted to
because he could put it on his resume.

You deploy tech to solve a problem. Don't go solving problems that you don't
have yet.

~~~
miki123211
That, actually, is a bigger problem. Some people are using things not for the
benefit of the thing they're building, but for their own (AKA resumes).

------
cdoxsey
Please at least use a managed database. Treat that linode box like it might
disappear at any moment. It is so easy to shoot yourself in the foot with a
database and lose everything.

Also FWIW configuring Linux machines manually over SSH and building and
deploying your code too it, isn't always easier. Make sure you write
everything you did down, because in 6 months you will completely forget how
you built this thing, where it was deployed, etc...

~~~
Cthulhu_
And whatever you choose, TEST YOUR BACKUPS. Spin up a new instance and
practice recovery. This needs to be done at least quarterly, both to test your
backups and train yourself and/or your people in this process.

------
lbill
I agree with the author : start simple, then use more complex tools when and
if you need to.

I believe it is true about infrastructure, and about features and code as
well.

When my team needs to release a functionality for the "product"[1] that we
maintain, we have a very simple strategy : we go step by step. The users tell
us what they need, and we start to deliver as soon as we can. Our first
release only solves a portion of the user's problem, but the second release
solves more, and so does the next, and so on. I devised this strategy with the
users to make sure they are fine with it: they expect a perfect product...
eventually. But in the mean time, they having only some portions of their
needs met, and we avoid the feeling of being overwhelmed by a great apparent
complexity.

[1] The context is a bit complicated... and irrelevant. We don't exactly
maintain a product, but it's close enough for the argument here.

~~~
lbill
Oops! Sorry for the huge typo in my comment. "they having only some portions
of their needs met" should be read "they are fine with having only some
portions of their needs met".

(It is a bit more than a typo)

------
perfunctory
> ‘Engineers get sidetracked by things that make engineers excited, not that
> solve real problems for users’

The real question is why business allows them to. I find it entertaining to
try to substitute another profession for engineer in the sentence above and
see if it still works. Lawyers, teachers, doctors, civil architects...

~~~
cturner
The precious commodity with our work is cognitive load. We can build tooling
that reduces future cognitive load. As a result, we have access to
multipliers. There is no analogy to this with the other fields.

~~~
philwelch
The idea that tooling "reduces" cognitive load is based on the assumption that
tooling lets you work in terms of abstractions rather than worrying about what
the tooling actually does. This assumption breaks down as soon as the tooling
itself breaks down. At that point, not only do you have to worry about the
underlying problems that the tooling is intended to solve, but you also have
to worry about the implementation details of the tooling itself.

Tooling breaks down more often when it's more complicated, and you need more
complexity when your tooling abstracts over a large amount of complexity, so
the more that your tooling is supposed to _reduce_ your cognitive load, the
more likely it will break in such a way that it actually _increases_ your
cognitive load.

------
jpttsn
I run a small team, but we do a large number of projects. Each project is
developed fast and then maintained at low intensity for a long time. Have
found great use for a nifty complex DevOps pipeline.

Before, when I built the same app for two years, the nifty pipeline was
overwrought.

I think the context switching inherent in what I do now makes the nifty
pipeline useful.

------
davidjnelson
> Kubernetes clusters? Load balancers across multiple cloud regions? Zero-
> click, Blue-Green deployment based on real-time metrics from a continuous
> deployment pipeline?

Kubernetes, multi region, canary deployment, immutable infrastructure, even
infrastructure as code is overengineering for a brand new indie project, for
sure.

BUT, there is a certain minimum amount of quality, availability and
scalability any product needs, especially a new product that might show up on
hacker news or product hunt and get 100-300 concurrent requests a second for
multiple hours at unpredictable times.

To not use fargate because you have to dockerize your app and build it during
a codepipeline stage is silly. That takes what, a week, max, the first time
you ever do it? Once you've done it, you can do it faster for future products.
To not use an autoscaling and high availability database when it costs the
same and is so simple to set up makes no sense either. To not use ci/cd to
test your db migrations and new code on staging before pushing to prod is just
dumb and asking for pain and suffering. Why put yourself through that when
products have been built to make these things _so_ easy and inexpensive now?

------
mark_l_watson
Right on! I am enjoying the book “A Company of One” right now and the author
makes the same point of only doing what it takes to provide great service to a
few customers. Another point is to spend more effort servicing existing
customers than spending time trying to get new customers.

For deployment: if you can tolerate rare service downtimes then it is
difficult to beat Linode or Digital Ocean. I am probably in a minority, but I
also really like GCP’s AppEngine: turn on billing, set the minimum instance
count to one and the maximum to two or three, and you have a resilient system.
That said, I run my primary web site on a single tiny GCP VPS instance with no
fault tolerance.

For heavier compute requirements, I also really like a physical server from
Hetzner- for about $100/month you can get a GPU, i7, and lots of RAM and
bandwidth.

~~~
robodale
Thanks for the "A Company of One" book recommendation. I just ordered it.

------
mdekkers
You don't need it if you are a one-man band doing a small side project.

> _I spent hours setting up a simple and free continuous deployment pipeline.
> Commit to git, Semaphore CI builds the Docker image, uploads the image to a
> container registry, then pulls & restarts the container on my server. It’s
> really cool. I felt like a wizard once I got it working … and it was free! I
> actually sat down here to write all about how to do it yourself, until I
> realised how long I spent on it and how much value it delivered to my users_

If you have a team of some size, the time you gain over the lifecycle of the
project can be very significant when you have a smoothly running automated
dev/test/UA/live pipeline.

> _Not build the fanciest deployment pipelines, or multi-zone, multi-region,
> multi-cloud Nuclear Winter proof high availability setup._

If your downtime can be measured in appreciable lost revenue or reputational
cost, then yes, you need all of that, as well as a solid SRE or 5 to keep it
all running.

~~~
sleepychu
> _Obviously if you’re FAANG-level or some established site where that 0.1%
> downtime translates into vast quantities of cash disappearing from your
> books, this stuff is all great. You have the funds to do things “right”._

~~~
scarface74
You don’t have to be a FAANG. You can be a B2B startup with one or two major
clients who if they see you aren’t highly available will jump ship taking a
lot of your revenue with them.

I’ve been in enough meetings working for small startups where our clients
grilled us about our resiliency, availability, redundancy etc. and where they
forced us to put our code in escrow as part of our contract.

------
AstralStorm
Usually,. the thing that you lose is scalability, availability any
reliability.

You will need a remote backup and a secondary located service, just in case.
Potentially a third development one to not push breaking changes directly into
production.

Skipping any of this will make your service brittle in addition to not being
scalable.

And it already hits the rule of three... (of "don't repeat yourself") that
makes a lightweight Kubernetes deployment worthwhile. (Perhaps without all the
bells and whistles.)

Remember, downtime is usually lost money. A day's worth can be thousands of
dollars at least, and that's two nines already. (99.99%)

~~~
chii
Or, you ship 2 months earlier, and the trade off being suffering a day's worth
of downtime.

It makes no sense to start with a more complicated setup, until there is clear
evidence of a need.

~~~
AstralStorm
The main problem is that usually low quality infra bites you when you get
exposure. Which is exactly when you don't want it to fail...

For a start, Heroku or AWS instance will do and they scale. (As the other
commenter said)

If you're doing your own infra, you need at least basic fallbacks and safety.

------
q-base
I see this "Push for new and shiny solutions"[1] across everything from
startups to large corporations. And I am genuinely surprised at how often I
see people falling for this good old trap of over-optimization/engineering. I
genuinely cannot fathom why people want the complexity.

It is by no means constrained to infrastructure, it is architecture,
technology, management the whole lot.

[1][http://jesperreiche.com/the-push-for-new-and-shiny-
solutions...](http://jesperreiche.com/the-push-for-new-and-shiny-solutions-to-
old-known-problems/)

------
sbhn
It depends on who your customer is. If your customer is the investor, then
there is no limit to the amount of hardware, infrastructure, and experts
sitting behind a desk and multiple monitors in every concevable separation of
concern you can think of in every major city across the globe. Your product is
amazing, and its going to change the world and how people percieve things. You
need everything you can throw at it. Investors want to be the best. And thats
if you customer is the investor.

------
SnacksOnAPlane
One of my web sites is run by a "bundle exec rails s" in a tmux window on an
EC2 instance. New deploy? Run "git pull" (and maybe "rake db:migrate") in tmux
pane and restart the "rails s" command. Easy peasy.

I'd make it more complicated and resilient but why?

------
jb3689
The problem with Kubernetes (and microservices in general) is getting a team
productive with it, and that requires lots of time, dedication, and tooling.
It's not the right solution for most teams, but it could be if we streamline
it more

------
alfonsodev
When you are building a startup, there is always the question, "will this
scale?" or "in the event of an investment or going viral are we ready to scale
automatically ?"

I think that there is a lot of worries around this, it conditions technology
choices, sometimes pushing early optimisations, and it is hard because you
don't have the data on how many concurrent users your product will have. The
best you can do is to extrapolate from you current data or other products
data.

So many people end up following a recipe, it could be Firebase recipe, AWS
Fargate, Serverless... you name it. Recipes that promise limitless autoscale.

I think indeed is distracting and premature, but again is hard to avoid it, if
the premise is that there will be a massive growth overnight, which is
imposible to predict that will potentially kill us, and we have to be ready.

~~~
jcadam
I've wasted a lot of time during my attempts as a solo developer to build a
product with a scalable backend architecture for the hordes of users that
never materialize.

Next idea I have, I swear I'm going to throw all my effort into the frontend
and go with as minimal a backend as I can get away with (heck, depending on
the application, human-in-the-loop might be fine early on) until I get some
validation. It'll feel "wrong", but whatever.

------
ratling
This is a pretty dumb article.

Like, no shit don't build more infrastructure than you need. If you're a one
or two man team you don't need anything more complicated than 'push container
here, pull down here' or 'push code here' et. al. Once you get above that
having a repeatable process (repeatable by someone besides the person who
built it) becomes valuable.

The 'every second spent on infrastructure isn't spent on features' line makes
me think I'm going to have to blackmail into working on security updates or
paying down technical debt. And the picture you picked is cheeky.

I will agree with this statement, "You need the most simple, least time
consuming infrastructure that gets you to that point." But that WILL become
more than just 'throw it on a linode instance' after you get some customers
and more than 2 people working on whatever uber for dogs project you have
going on.

