
Cheap Kubernetes Cluster on AWS with Kubeadm - unkoman
https://github.com/cablespaghetti/kubeadm-aws
======
pstadler
I'm running a three node cluster on Hetzner Cloud for less than $10 a month.
Comprehensive guide and automated provisioning available here:
[https://github.com/hobby-kube/guide](https://github.com/hobby-kube/guide)

~~~
jimmy_ruska
Hetzner lists VMs that are ultra cheap but then has separate listing for more
expensive VMs with dedicated cores. Alibaba does the same.

With these shared core machines, I wonder what the performance reliability
guarantees available and how it compares to the dedicated core machines.

~~~
chrismeller
I don't have any actual statistics to back it up, but I've hosted a variety of
things in Hetzner's cloud (and dedicated) options, and... it depends.

If you're hosting a website (or a database backing one) there is going to be a
natural kind of ebb and flow as traffic comes in, gets processed, and
answered. Since you're not utilizing a large amount of CPU consistently this
actually fits really well into the shared core model - that's exactly why they
can offer it. Even if a subset of requests take twice as long to fulfill this
is usually not even noticeable to the user.

On the other hand if you're routinely running a large ETL process, resizing
video or images, or any of dozens of other things I'm sure you can think of
that are using a lot of CPU for prolonged periods you're going to notice it.

Specifically I have two situations I've run into: My TeamCity build agent will
be unpredictable for longer builds - one might finish in 5 minutes, the next
might take 10 - and when I had a Windows Server running as an Amazon Workspace
alternative (RDP in, run Visual Studio, etc.) things like building and
debugging an app were noticeably slower than if I did it locally or on my
dedicated server (even with similar specs).

So if CPU usage isn't normally your bottleneck or you're scaling horizontally
and CPU performance isn't as important it's a great option that will save you
quite a bit. If your workload is very CPU sensitive you probably shouldn't be
using a VM anyway and should look into a more dedicated infrastructure, but
obviously there is also a middle ground to be had...

~~~
merb
they also have a dedicated vcpu option but it is not that cheap

------
crb
> If anyone has any suggestions on a better way of doing this without shelling
> out $20 a month for an ELB, please open an Issue!

kubehost was designed for this purpose on GKE:
[https://github.com/GoogleContainerTools/kubehost](https://github.com/GoogleContainerTools/kubehost)

~~~
acd
You could use route53 health checks and do dns failover

[https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dn...](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-
failover-configuring.html)

------
013a
You can totally install Minikube on AWS [1], which removes the need for a
dedicated master and drop the price to that of a single instance.

Not sure why you'd ever want to do this, given that GKE or DO will always be
cheaper and AWS's core services aren't all that special, but as a thought
experiment it's interesting.

[1] [https://www.radishlogic.com/kubernetes/running-minikube-
in-a...](https://www.radishlogic.com/kubernetes/running-minikube-in-aws-
ec2-ubuntu/)

~~~
zwerdlds
New to the k8s scene, but does Minikube support multi-node systems?

~~~
crb
No. We discuss this exact topic with Minikube author Dan Lorenc here:
[https://kubernetespodcast.com/episode/039-minikube/](https://kubernetespodcast.com/episode/039-minikube/)

~~~
shitloadofbooks
Your podcast is great! You two make a fairly dry topic engaging and I look
forward to every episode.

------
lifeisstillgood
I love the attitude here - because in my privileged western world worrying if
my monthly hosting costs are one latte and crossiant or two seems quaint.

But even in the "rich" EU Avergae monthly wages can be only 1,000 USD, making
this a tenth of a day's work, and we don't have to go much further afield to
see 6 bucks becoming a significant chunk of a workers day.

So thank you, some kids somewhere will be able to afford to develop skills
because you started penny pinching.

cheers

~~~
skybrian
Even for people who can afford it, this makes a difference for preserving low-
traffic hobby sites you only mildly care about.

I have a couple of silly sites that I've run on App Engine for many years,
using its generous free tier. If they were on Digital Ocean I'd have shut them
down by now, as I eventually did for a Go search engine that I wasn't using
much.

------
fridgamarator
Google Cloud and Digital Ocean both offer managed kubernetes clusters, you
only pay for the workers.

~~~
bryanlarsen
Not only that, Digital Ocean's fully managed cluster starts at $10.

~~~
k__
AWS Fargate starts at $0

If it's only a hobby project with a small cluster, the Fargate costs could
very well be under $10.

~~~
yebyen
That's not really an apples-to-apples comparison though, is it? I mean yeah, a
cluster that you never power up costs nothing, that's what I would have hoped.

[https://aws.amazon.com/fargate/pricing/](https://aws.amazon.com/fargate/pricing/)

Assume that you are actually _doing something_ with the cluster, then there
will be charges for vCPU and memory. The pricing on this page indicates that
if you only need your containers for 10 minutes a day, every day, then you
will pay $1.23 for the month at 2GB/1vCPU. That's pretty modest usage at a
pretty modest cost.

Sure, you can do it, but... you may need to re-architect your product to take
advantage of transient workers. (Now I think we all should do that, but that's
another conversation...)

Compared to DigitalOcean's $10/mo (single-worker, managed) cluster, which
provides one full-time node with the same specs, that you can slice-and-dice
to run as many tasks as you can fit in 2GB of memory and a single vCPU. Now it
becomes clearer that Fargate is priced at a premium. If I'm doing the math
right, you'll pay $177.12 for that same task space with your Fargate cluster.

If your hobby project needs to run for more than a couple of hours a month,
you will usually pay a lot more with Fargate.

~~~
k__
I had the impression that the $1.23 are for 10 workers?

~~~
yebyen
My mistake:

> For example, your service uses 5 ECS Tasks, running for 10 minutes (600
> seconds) every day for a month (30 days) where each ECS Task uses 1 vCPU and
> 2GB memory.

5 workers, at 10 minutes each. That's still $35 / $10 = 350% of the cost, and
they're not full-time. The point was that Fargate is not priced to be the
cheapest option, unless your product is designed specifically with Fargate in
mind.

(And ok, not specifically Fargate, really any architecture that permits the
workers to come and go, but mostly go.)

Fargate is a tool for a job, and in the suite of Amazon's offerings that
includes Lambda, S3, CloudFront, you really can make some cool stuff for
cheap, if you pay attention to what kind of resources it needs, and if it
_really needs_ those resources, or maybe there's a cheaper offering that is a
better fit, or at least that could do the job just as well.

Don't get me wrong! But if I want to take, say, my Ruby on Rails app and host
it somewhere without reinventing my whole stack from scratch, I definitely
won't use Lambda because any Rails app just won't fit the model very well, and
I won't use Fargate because it's [N] integer times as expensive as comparable
offerings.

If Fargate was capable of auto-scaling tasks to zero workers during periods of
inactivity while remaining mostly responsive, like Lambda "cold start" vs
"warm start", then it would be a much easier sell for me.

~~~
k__
Sure, I never had the impression it was cheaper for the same workload.

My take was, that hobby projects probably don't run 24/7, and at some point,
you pay while not using them.

~~~
bryanlarsen
Yup, you're both right. I use my $10 kube cluster for 2 apps, one of which has
3 users, the other has a dozen. So it's idle >99% of the time, but it needs to
be available 24/7\. And they're rails apps, so it's neither easy to "scale to
0", and if it was, the startup times would probably be horrendous enough that
it wouldn't be worth saving that fraction of $10...

~~~
yebyen
Keep an eye on Knative and/or Project Riff. They _can_ scale pods to zero
during periods of inactivity, that's one of the bigger selling points, and the
startup times might not be as horrendous as you think. (YMMV depending on your
runtime and your requirements.)

Riff is really a Function-as-a-Service library, built on Knative, but Knative
can run any arbitrary workloads. Riff will soon have support for Ruby again (I
promise, I'm working on it[1]) and Knative has this capability now[2]

So, there's a convergence coming and it's going to make a big difference in
this cost/benefit trade-off. You need a cluster to run your Knative workloads
on, and that cluster has to be persistent and always available to make the
magic trick work.

But say you have a small to medium enterprise that runs 75 custom apps, and
any 10-30 of them could be in use at any given time. You'd like to take
advantage of office hours and turn things off when they're not in use, but you
can't guarantee that nobody is going to need some app, some time after 5pm.

The cluster itself can autoscale up and down to a smaller or larger size,
depending on how many balls are in the air. Your apps remain responsive even
at nighttime, when your cluster footprint is a tiny fraction of what supports
the company's daytime operations.

(By "I'm working on it" I really mean, they've made it as easy as they
possibly can for new runtimes to be added with their buildpack v3 framework. I
riffed on Ruby back in v0.0.7 and it was possible to reuse the work in v0.1.0,
after they ported their stack over to Knative. Now in v0.2.0, the work I did
on my Ruby invoker is not wholly reusable, but I'm hoping it will be a pretty
smooth transition.)

And you didn't say you're using Ruby, but the point again is that anyone can
add support for any language, and it's no big deal. I'm basically nobody here,
and I can do it...

So that may sound a little crazy, but keep in mind that there are also K8s
"virtual node" solutions in the works or already out there. So binpacking your
containers into nodes could soon be a thing of the past, and as clunky as what
I'm proposing sounds, it may not always be that way.

Sure, AWS could do it with Fargate too, and it might turn out to be just as
good, but right now that's speculative. This is pretty much stuff that is all
out there now. It's just parts that need to be put together.

[1]:
[https://github.com/projectriff/riff/issues/1093](https://github.com/projectriff/riff/issues/1093)

[2]:
[https://github.com/knative/docs/tree/master/serving/samples/...](https://github.com/knative/docs/tree/master/serving/samples/helloworld-
ruby)

------
zegl
Can it be considered cheap if the cluster is also small? The overhead of the
control plane+networking can be quite big, and can easily use more resources
than what a m1.small has to offer.

------
triplewipeass
How's this better than kops?
[https://github.com/kubernetes/kops](https://github.com/kubernetes/kops)

~~~
jordanbeiber
Kubeadm gives you a ”core” or bare bones setup which makes it much more
flexible in terms of addons and versions etc.

It’s a bit more work though, which is the trade-off, but since the last few
versions kubeadm makes it really easy to spin up clusters.

Personally I opt for complete flexibility.

~~~
raesene9
Also Kubeadm has a pretty sane set of default security settings, which some
other k8s distributions do not.

Specifically kops (by default) does not enable authentication on the kubelet,
meaning any attacker who gets access to one container in your cluster is very
likely to be able to compromise the whole thing.

------
joseph
My project keights[1] can build a cheap two node cluster in AWS, but is not
limited to small clusters. Though, it does spin up an ELB unlike this one.

1\.
[https://github.com/cloudboss/keights](https://github.com/cloudboss/keights)

------
moondev
You can use haproxy as a tcp load balancer for the apiserver instead of an elb

~~~
manishsharan
could you please comment on how you use Haproxy servers across multiple AWS
availability zones ? how do you configre and manage dns for your haproxy etc ?
My reason for asking is that I am super cheap and I want to avoid AWS ELB
charges.

~~~
yebyen
Put your Haproxy (or nginx-ingress, or whatever) into HostPort mode, or enable
hostNetwork.

I have a comment somewhere in my history that explains in a bit more detail:
[https://news.ycombinator.com/item?id=18660503](https://news.ycombinator.com/item?id=18660503)

that's it... hostPort mode, hostNetwork, and enable in DaemonSet mode. Now all
of your nodes are load balancers for ingress, and you don't need any ELBs.
This is not a recommended configuration because something has to point DNS at
your nodes, and the nodes are really not designed to be permanent.

If you autoscale, or scale your cluster manually, your DNS needs to be updated
to keep up with that. You may be able to find a way to automate that, but DNS
has limitations related to TTL, such that if you are doing this too
frequently, visitors to your cluster are likely to experience issues.

But if your nodes never come and go, this is a pretty good way to run a
cluster and keep it on the cheap. If the traffic you want to balance is not
HTTP then ingress won't help you (for now?), but the configuration for HAproxy
will be similar.

