
The AWS Spend of Our SaaS Side-Business - encoderer
https://blog.cronitor.io/the-aws-spend-of-a-saas-side-business-30bd5dbd91b
======
thesandlord
Disclaimer: I work for Google Cloud

> Cloud solutions like elastic load balancing are generally engineered for the
> average use case. Think about the ways you are not like the average.

This might be the case for some things, but many of Google Cloud's
"elastic/serverless" products (Load Balancers, App Engine, Pub/Sub, BigQuery,
Datastore, etc) can truly scale up from nothing to huge traffic spike without
the need for pre-warming and things like that.

~~~
nodesocket
Came here just to reiterate this. Google Cloud Load balancers don't need
warming, manual creation of support tickets, etc.

Also, networking on GCP is vastly superior to AWS. The VPC is setup with sane
defaults and VM instances can communicate privately to other VM instances in
any zone or region without needing complicated constructs such as VPN's or NAT
instances.

Lastly, Google Cloud is generally cheaper (sustained use-discounts), higher
performance, and takes all the lessons learned from AWS and applies them.

The one knock I have on GCP is billing. It is incredibly obtuse. Even the
monthly billing invoices I still have a hard time digesting (especially if you
have multiple projects under a single billing account).

~~~
tyingq
Any AWS users have a better experience with their newer "ALB" load balancer?
It doesn't specifically call out any performance benefits over the legacy
"ELB", but since it's much newer, I'm curious if it happens to do better.

~~~
aluminussoma
Yes. Although Administration / UI is slightly more confusing. We had unique
performance issues with the ELB offering that couldn't be fixed because it was
the way it was designed. With ALB, that performance issue no longer existed.

~~~
mwcampbell
ALB may be more complicated, but it's also more flexible, and I've especially
come to appreciate that in the past couple of days.

By the way, it looks like ALB might be based on nginx, based on the error
pages. I wonder if classic ELB is based on HAProxy.

------
cyberferret
Nice post, and interesting to see your learning pretty much matches my own
when building my SaaS business.

Curious as to why you didn't opt for RDS for the database side of things? I
actually have about 5 side projects as well as my main SaaS, and they all
share the same RDS instance for the database which saves me cost while keeping
performance high.

Also, do you use Lambda much at all for periodic scripts that need to run?
I've more recently been building more and more mini applets on Lambda for
things like health checks or replicating data across Amazon services etc. with
good results.

Also, and this might be a good complimentary service to your monitoring
service, but I've been using CloudWatch's timed tasks a lot recently to
trigger some of those Lambda instances - very quick and easy. I've been using
another service to monitor and report on missed triggers, but will look into
Cronitor a bit more as a viable option.

~~~
paulddraper
Failover time for RDS is high, like a minute or two.

That's probably fine for lots of stuff, but not everything.

~~~
mrep
How do you get higher availability than that?

~~~
paulddraper
Use Virtual IP rather than DNS for failover.

Minutes is a really long time to go down.

~~~
mrep
Have you tested that?

If you actually have an in-house solution to providing better availability
than amazon's RDS options, then I recommend you market your engineering
solution to the major cloud providers as they will pay you very well to help
them implement your solution to that problem.

~~~
paulddraper
> Have you tested that?

Only all the time.

> I recommend you market your engineering solution

I saw exactly that company at the last re:Invent. Can't remember what the name
was...

> to the major cloud providers as they will pay you very well to help them
> implement your solution to that problem.

My guess is that most reasonably sized organization manager their own
databases for uptime-critical systems (case in point). It lets them have more
control over backups, when to upgrade the database, how to handle failover,
etc.

------
encoderer
A few months ago my co-founder August did a great Indie Hackers interview
[https://www.indiehackers.com/businesses/cronitor](https://www.indiehackers.com/businesses/cronitor)
and we heard a lot of positive feedback on sharing revenue and growth numbers.
On that theme, we thought we'd write a blog post to give a little more color
to our largest expense, AWS.

~~~
aeorgnoieang
The post was great.

The only thing I didn't like was "spend" in the title – "costs" or "expenses"
would be less pretentious, to my ears and in my mind. This is a really trivial
quibble! [And apparently this is a _really_ old word in English anyways][1].
Now I feel like a grumpy old person.

[1]:
[https://english.stackexchange.com/a/79055/42347](https://english.stackexchange.com/a/79055/42347)

~~~
ci5er
Not to quibble with your quibble (but here I go anyway)... The term "Marketing
Spend of X" or "Operating Spend of Y" is fairly common business-analysis-
speak. At least in certain regions of the United States. IIRC, I also heard it
used by Telstra and ANZ execs during some projects I had to take care of in
Australia.

------
douglasfshearer
Great post. It's good to see an enterprise smaller than Segment [0] talk about
how complex AWS pricing is.

You mention you use SQS to queue incoming metrics. What
server/framework/language are you using to do this? IME using SQS requires
tools with good parallelism to deal with it's high-latency/high-throughput
performance characteristics.

[0] [https://segment.com/blog/spotting-a-million-dollars-in-
your-...](https://segment.com/blog/spotting-a-million-dollars-in-your-aws-
account/)

------
paultopia
How much of devops in AWS-land is really forensic accountancy?

Moneymaking idea for someone: a devops person, a lawyer, and an accountant get
together and charge companies to predict their cloud bill.

~~~
samstave
Use cloudability to monitor cloud spend and prediction is pretty easy..

~~~
paultopia
Naturally someone has already started that business :-)

------
gingerlime
I'm a bit curious about the cost/revenue ratio. Is 12.5% considered a
reasonable ratio? I just checked our spend, which includes not only hosting
but other IT related spends (helpdesk, email, slack, email marketing service,
etc). I think our spend is around the 3-4% mark compared to revenue.

Now I'm not trying to show off or say we're doing things better. If anything,
maybe this post can help me convince my cofounders that we should spend more
on infrastructure hosting (there's lots of things we could improve). I'm
mostly curious what's a typical ratio in other smallish startups.

~~~
augustflanagan
Cronitor co-founder here. This is a great question! I would love to see others
post some numbers here.

As Shane mentioned in the post 12.5% has been a consistent number for us as we
have scaled up over the past couple years. That said, at this point I suspect
we will see this percentage decrease a bit going forward. I'm basing this on
intuiting that we're a bit over provisioned at the moment and won't have to
scale our infrastructure linearly with user growth over the next year or two.
Of course, that remains to be seen...

To add some more numbers to the conversation my full time job is CTO at a
consumer facing tech company (Babylist.com) I just looked at our spending on
IT/infrastructure for the last couple of months and it's around 1.5-2% of
revenue.

~~~
gingerlime
Thanks. I guess there are economies of scale / barrier of entry costs involved
that should reduce this ratio over time.

On a side note. When people talk about x-figures revenue, are they typically
talking about x-figures monthly or yearly?

------
dasil003
I love AWS and for any startup that I expected to need to scale quickly I
would choose it in a heartbeat.

However when it comes to low-traffic side projects and experiments where costs
of a few hundred dollars matter, I prefer Linode or DigitalOcean with ansible
provisioning and B2 for block storage. It will cost you more time for sure,
but it will give you better performance at the low-end which means you can
break even sooner. If it takes off you can always migrate to AWS, GCP or Azure
later.

~~~
uji
GCloud recently updated their free-tier
([https://cloud.google.com/free/](https://cloud.google.com/free/)). They
provide $300 credit which can be used for any product in GCP, and some
services when used within quota are always free. So you might want to give GCP
a try for your low-traffic side projects.

~~~
shusson
Just to clarify, the $300 is linked to your personal account and once it
expires it's gone forever (at least for now).

~~~
nunez
can't you just create new accounts ad infintum?

~~~
icebraining
You need to add a CC or bank account.

------
desdiv
Dumb question: is there a way to show the total AWS costs you're incurring in
the past second/minute/hour?

The finest granularity I can find in the AWS web console is per day; many
times I've butterfingered an input and only caught it a few days after the
fact due to the unexpected bill increase.

~~~
cldellow
AWS supports publishing detailed billing information to an S3 bucket on an
hourly frequency with excellent granularity as to which resource and operation
is driving spend.

Unfortunately, to make meaningful use of this you pretty much have to roll
your own infra to download it, load it and analyze it. :(

~~~
samstave
Once again... contact cloudability.

~~~
thenaturalist
cloudability starts at $499 minimum per month. Not sure how this seems to be a
feasible option for side projects.

------
zitterbewegung
Reserved instances are never really good starting out. I think it's better
used when you understand your setup and your setup has been working for years.
Or if you are migrating a service that you understand the load that was on
premesis.

------
magacloud
Nice post indeed. Few cents from our experience at ActOnCloud, Often Cloud-
first and Cloud-only companies would face similar issues in future. Cloud is a
one way trip. Unless one takes care of vendor lock-in and prepare the team for
better governance and financial control, its going to be nightmare.

With cloud providers pushing for Serverless, it will be even more darker,
there is not a way to get control, Just hope that everything will be greener.

------
ecesena
Surprised to not see RDS and ElastiCache. What do you use as "db"?

~~~
encoderer
We run our own MySQL and Redis on an Ec2 instance. Originally it was due to
cost -- you essentially are charged by instance when using these services and
we could run them both from a single instance. Today, it's really due to not
wanting or needing to do the migration.

I will add, we've paid when we've had to do several manual db upgrades in that
time:

\- Original m3.medium on non-provisioned iops

\- Upgrade m4.large

\- Upgrade to an EBS with provisioned iops

\- Upgrade to m4.xlarge

These are easy enough, but not nearly as easy as an RDS upgrade is.

------
bluedino
What are the reasons for three months being 3X the rest?

~~~
encoderer
We purchased reserved instances. When I quote the 12.5% number it's amortizing
the ri cost.

Reserved instances have saved us a ton of money, even with my screw up last
October.

------
alberts00

      You should only use T2 instances within an auto-scaling group.
    

Could someone please elaborate? I always thought this is a horrible idea
because upon CPU credit exhaustion the AWS metrics show CPU utilisation
including the burst. For example, if you run out of CPU credits for t2.micro
it will show around 15% in AWS whilst on the instance itself you will see
100%.

~~~
philliphaydon
Using t2 rds in production for 1 year for some stuff at work. We used
cloudwatch to monitor the cpu credit balance. Moved from t2 small to t2
medium. Fixed some queries that used too much cpu. Still running strong. Never
had down time.

------
mi100hael
_> You pay by number of requests, not number of messages, so it reduces costs
and makes it significantly faster to gulp down messages._

Are there any other technical changes you've made to your application code
specifically in response to AWS costs? Any other recommendations when
designing a new application?

~~~
FigmentEngine
There is advice like this in the "Cost Optimization" whitepaper of Well-
Architected

aws.amazon.com/well-architected

disclaimer: work for AWS and author

~~~
FigmentEngine
[https://aws.amazon.com/well-architected](https://aws.amazon.com/well-
architected)

------
danerov
Really interesting to see the cost of other small companies.

