
MVPs and $100k AWS Bills: Reflections on our launch - paulstovell
https://octopus.com/blog/octopus-cloud-1.0-reflections
======
pjc50
> Octopus Cloud customers could start a free 30-day trial, which meant that
> those hundreds of trial signups per month, each of which cost us $100 to
> host, quickly added up.

> We also didn’t have our pricing quite right. We initially launched Octopus
> Cloud at $10/month, with a different pricing model from the one we currently
> use. Unfortunately, this was one of the most painful lessons we learned
> because the deficit between what we were charging and spending was magnified
> by the sheer number of people using Octopus Cloud; continued growth would
> further amplify the problem.

"Every person who fills in a web form gets to burn a $100 bill of ours" is the
real disaster here.

I know it's trendy to think that software isn't like regular business, and you
can lose money but make it up on volume, but going to a cloud vendor gives you
a real cost of goods sold, and that needs to be accounted for.

(also, the irony of a company whose product appears to be about cloud
deployment having a disaster by deploying to the cloud should not be lost on
us)

~~~
LeonM
Even without the free trial, how on earth can you burn $100 in cloud resources
in a month if you plan to resell it for $10?

~~~
matthewaveryusa
I think the basic idea is 'you need money to make money.' The hope is that
with scale and engineering they can burn less. It is dubious that anyone can
reduce from 100 to 10 -- 90% reduction would be an incredible feat, and this
is why:

AWS has something like profit 25% margin. Therefore assuming that AWS is
within 15% of doing a half-decent job at controlling costs, that's still max
40% profit margin if you were running bare-metal and beating AWS at their own
game to the tune of 15%. So if you're selling stuff for 10/month, your AWS
bill should be 17/month just to break even after all the awesome engineering
you've done with the VC money -- at least that's how I see it.

Now, if they are doing incredibly stupid stuff, to the tune of 90% wasted ec2
cpu cycles/db requests, then maybe they've priced it right -- who knows.

Would love to hear some insights on where I might be wrong/misjudging here.

~~~
jiveturkey
You're not including the scale factor in your reasoning.

AWS has (taking your number as correct) 25% margin _at immense scale_.
Hardware is cheap at this scale. unit pricing for power and space is cheap at
this scale. But labor cost is enormous at this scale. Even factoring in
leverage of automation, at this scale there is just enormous bureaucracy. One
guy or just one team just can't "do something". Anything worth doing will have
enormous repercussions across many areas and needs to go up and down the
chain. Stuff still gets done, because impact is large at this scale.

> 90% wasted ec2 cpu cycles/db requests,

So then, my point is, if you are just using EC2 and egress, AWS is very
expensive. You don't need all the big scale engineering they have, to launch
an MVP. Of course, you need to be able to hire the type of people that can
manage in-house compute, and that may be hard to find these days, at any
price. But taking labor shortage out of the calculation, it's vastly better to
DIY.

If you are using PaaS features, that's the sweet spot where you get tremendous
value out of AWS.

> profit 25% margin. Therefore assuming that AWS is within 15% of doing a
> half-decent job at controlling costs, that's still max 40% profit margin.

Getting back to this, even if you dispute all of what I've said, your basic
math is wrong. 15% short on 25% is 29.4%. (25/.85, not 25+15).

------
teddyuk
"To bring Octopus Cloud to market quickly, we did the simplest thing possible;
we took our self-hosted Octopus Server product and bundled it into an EC2
instance for each customer that signed up. We had to make changes to the
product, but mostly around permissions."

I have used octopus and I could have told them that they can't lift-and-shift.
The amount of requests the app makes is ridiculous, the way they wrote their
database is appalling (actually crazily dumb [https://octopus.com/blog/sql-as-
document-store](https://octopus.com/blog/sql-as-document-store)). Everything
about it would mean they can't scale in the cloud without wasting a ton of
money.

"Cloud stuff can be really expensive" \- yes if you write dog shit, putting
that in the cloud will be expensive.

~~~
teddyuk
From the "sql-as-document-store" post, I re-read it every now and then to
realise how actually terrible it is:

"For a many-to-many, we do something a bit naughty: we store the values with |
characters between each value:"

Not naughty, stupid.

~~~
swiley
Why would you bother with a relational database at that point? That sounds
like it’s not normalized.

I haven’t read the post... maybe I’m missing something.

~~~
teddyuk
because they didn't know what they were doing - I can say this as this is what
the blog shows, they don't understand databases and customers (including me)
have suffered this

~~~
jcadam
I've learned much too late in my career that competent software engineering is
completely orthogonal to creating a successful(from a market perspective)
software product.

~~~
swiley
Well yeah most of it’s just CRUD and that’s pretty straightforward even at a
large scale (ex stack overflow.)

------
echelon
I have a streaming multimedia app that I want to launch. It ingests from the
client, does some transformations, then sends it back to them (or to other
parties) in realtime. It's pretty novel though - Snapchat adjacent.

I can't do it clientside. Nature of the problem.

I'm writing it in Rust, so it's relatively fast. But throwing thousands of
people at it will still require extreme scale. I'm going to use K8S horizontal
autoscaling. This will be _obscenely expensive_.

I'm not sure I can afford this. But I think a month of successful
demonstration will get me funded (or bought out). So _in theory_ my costs
could wind up getting paid. Maybe.

Do I risk launching this and potentially having $100k multiples of AWS bills?
I can't afford that. I don't want to deplete my life savings or potentially go
bankrupt.

Could I put it on a corporate card as an LLC and do my best to make it work?
Then fold without harm or fowl if it doesn't? Is that unethical? Could I get
sued?

All that said, I'm almost certain that a successful demo at scale with the
Internet thrown at it will get me money. It's a really cool and novel product.
People will be talking about it.

What should I do? Advice will be very appreciated.

~~~
teddyuk
onboard customers slowly and charge them - as you scale, make sure your
charging as well and get paid.

you don't need to go from 0 - 100k a month in a single month, octopus could
because they have backing.

~~~
echelon
This is a social media app, so I can't charge money for it. My hope would be
for funding or a buyout. Otherwise it will be shut down.

~~~
teddyuk
Just stop what you are doing, you need to have a way to generate revenue if
you want to sell it.

------
blantonl
The architectural decision to deploy each customer on their own segmented
components to make sure that "no one user’s data to mingle with another" seems
to me like a solution in search of a problem.

First, the deployment was on AWS (EC2 and RDS among other services) which was
probably already co-mingled with other AWS customers.

Second, when you are selling a SAAS offering, the assumption is _already
there_ that you'll build security and multi-tenancy into the app and the
underlying infrastructure shouldn't matter.

And 100K/month for a MVP SAAS offering. I'm speechless. I just, I can't even
understand how this burn rate passed muster with the leadership.

But Kudos to the team for owning up to it and sharing. Lots of lessons learned
here.

~~~
Spooky23
It could make alot of sense for alot of government, healthcare and financial
use cases.

~~~
blantonl
I agree. In those cases, sell those customer separate products and
expectations and price it accordingly.

------
scarface74
The lesson to be learned here, and the one they admittedly already knew in
advance, is if you do a lift and shift to AWS without changing your processes
to be cloud native, you will always spend more.

Unfortunately, I’ve met way too many old school netops guys who got one AWS
certification and all they knew was how to log into the web console and
duplicate their colo infrastructure, charge their client a lot of money, and
call it a day.

That being said, I’ve used a lot of CI/CD tools and OctopusDeploy is by far my
favorite.

Even though my company is very much a “pay someone to do the undifferentiated
heavy lifting” type of company, OctopusDeploy is so easy to use and maintain,
I am not sure we would migrate.

~~~
teddyuk
Going to the cloud won't save you money.

Going to the cloud lets you scale beyond your own data centre

Going to the cloud can be cheaper if you use the elasticity and make good use
of PaaS services (servless,paas, etc)

~~~
scarface74
I agree, but it also lets you automate and probably lets you get rid of or not
hire as many infrastructure people between automation and outsourcing the
simple stuff to a managed service provider.

------
remon
I'm sorry but this was a bit of a painful read. Hindsight is 20/20 but this
was pretty poor engineering and business modelling from the start.

"To ensure there was no way for one user’s data to mingle with another, each
cloud instance had its own dedicated VM, database, and a large number of
security configurations to prevent any funny business.". AWS has a myriad of
security and isolation tools to do that in significantly better and cost
efficient ways.

"Octopus Cloud customers could start a free 30-day trial, which meant that
those hundreds of trial signups per month, each of which cost us $100 to host,
quickly added up." So you're essentially allowing anyone to press a "Submit"
button to spend $100 for you. This sort of below-cost methods to gain
marketshare works for Amazon, Google, Microsoft and son because they have the
deep pockets but it's a questionable strategy for smaller companies.

------
rubbingalcohol
This is insane. It's the engineering equivalent of smashing one of your hands
with a hammer and then being surprised it hurts. Well no shit.

At that level, at the very least, it would make sense to run some amount of
physical infrastructure.

------
notyourday
Pardon me for asking a really dumb question: but do you have a positive item
charge - CGS? Because I'm sure one can get a great demand by selling a $1.00
for $0.90 but that's not a business model.

~~~
paulstovell
The post didn't make this very clear. $10 was the starting price - at the time
it was for 5 users. Similar to Confluence or Jira with a very cheap (in this
case loss-making) starting tier. Once a customer got to ~10 users it became
profitable.

What we didn't know if whether there'd be enough customers for us to care
about the loss on those $10 tiers. E.g., if only 100 customers signed up a
month, the costs would have been around $10K/month - enough to write off as a
marketing expense. Once it got to $100K it became something we needed to fix.

~~~
notyourday
> The post didn't make this very clear. $10 was the starting price - at the
> time it was for 5 users. Similar to Confluence or Jira with a very cheap (in
> this case loss-making) starting tier. Once a customer got to ~10 users it
> became profitable.

Err.. That's not a correct way of looking at a long tail. CGS on a long tail
must still be lower than the unit charge, otherwise you are paying $1.00 for
$0.90 which gives you an incorrect signal about the demand.

Say you make apple juice. And each juice uses 4 apples and only apples. 4
apples cost you $3. Setting a price of juice to $1.00 does not actually tell
you if there's a demand for the juice because it is below the CGS, which you
cannot make up in volume.

The long tail test is to set the juice price at (CGS + x) where x is some
number and see if you get the demand even if x is less than the other costs
attributed to juice ( rent/electric/juicer/salaries). If you do, then you can
make up your loses in volume because the higher the volume you are going to
have the smaller process cost attributed to every unit will be so there's
going to be a number at which CGS + y < CGS + x where y is the process cost
attributed to a unit.

------
mothsonasloth
Can't wait for this Cloud meme to die out and we can go back to On Prem but
instead of VMs managed by VMWare, Cloudstack, Vagrant, LXC etc, we will have
nodes managed by Kubernetes.

~~~
bronco21016
Why would cloud go away? If you’re a team of 3-4 people and have a startup
idea what’s easier? Create an AWS account and smartly spin up required
servers?

Or find a location (rent$), get fiber($$) to that location($$), find
hardware($$), install said hardware. Now you’re back to where you were with
AWS.

For larger already established businesses on prem is probably a setup you
should explore but on day 0 of a new business, cloud will probably have a
large cost and flexibility advantage as long as you smartly set it up.

~~~
shiftpgdn
I can purchase and then colocate 2x Dell R640s (8c/16t, 64gb of ram, as much
NVMe storage as I can stomach) with tier 1 bandwidth billed at the 95th
percentile and 24/7 remote hands support for about $150/month. The purchase
price on the Dells is about $3500/ea.

Total first year cost on this setup: $8,800

Total three year cost on this setup: $12,400

2x m5d.4xlarge EC2 instances (this isn't counting ingress/egress rates, or
really anything else) First year cost: $15,621 .

Total three year cost for the EC2 instance: $46,863.36

For the cost of using EC2 you could hire an extra person to help grow your
company.

Yes, you can get discounts in AWS for agreeing to a three year term. Yes AWS
offers you flexibility to walk away at any time.

If you have a base line load (which most companies do) you should really look
at colocating your owned equipment. You also get the bonus of being out of the
watchful eye of Amazon's profitability team looking to see if they can copy
what you're doing.

~~~
tills13
> For the cost of using EC2 you could hire an extra person to help grow your
> company.

You _will_ be hiring an extra person to manage these machines. That's the
beauty of The Cloud (tm). Need more compute? You got it. Provisioned too much?
Get rid of it. If you're a startup, you don't have time nor money to be
managing your own infra.

Not to mention the ability to deploy to _any_ region within minutes. Now
you're talking $8,800 / region + someone to maintain other regions.

> ingress/egress rates

unless you're also using S3 or CloudFront, this doesn't really mean anything.
Put a free CDN in-front of S3 or your instances if you're worried about load
or cost of IO.

> You also get the bonus of being out of the watchful eye of Amazon's
> profitability team looking to see if they can copy what you're doing

Ok now you're reaching.

~~~
shiftpgdn
How much management does a bare metal box need? Nothing more than what an ec2
instance would need.

There are colocation providers all over the world. Psychz.net is a great
example. My assumption is a single box per region. There is no redundancy in
your ec2 systems.

You should google the Amazon profitability team if you don't believe me.

~~~
solidasparagus
I can't find anything related to AWS and "profitability team" on Google. Can
you share some info/links?

~~~
shiftpgdn
[https://news.ycombinator.com/item?id=19734261](https://news.ycombinator.com/item?id=19734261)

Tangentially related: [https://www.datacenterdynamics.com/news/ovh-ceo-unlike-
amazo...](https://www.datacenterdynamics.com/news/ovh-ceo-unlike-amazon-
google-we-will-never-be-in-competition-with-you/)

[https://sellercentral.amazon.com/forums/t/amazon-
literally-s...](https://sellercentral.amazon.com/forums/t/amazon-literally-
stole-my-product-idea-and-took-over-my-listing/323388)

[https://fortune.com/2016/04/20/amazon-copies-
merchants/](https://fortune.com/2016/04/20/amazon-copies-merchants/)

[https://www.theverge.com/tldr/2019/9/19/20874818/amazon-
allb...](https://www.theverge.com/tldr/2019/9/19/20874818/amazon-allbirds-
shoe-clone-copy-sneaker-206-collective-private-label)

------
DVassallo
> We quickly learned that everything in AWS has some kind of service limit

Except S3. Unlimited objects, and unlimited Put and Get requests.

~~~
johnnymangos
Your comment is not true in all cases. S3 is limited to 5,000 get/put requests
per prefix. So while it's possible to achieve what you've said, it's also
possible to stumble upon this limit in a fashion similar to the original
comment.

~~~
solidasparagus
It looks like the number is slightly higher now -
[https://aws.amazon.com/about-aws/whats-
new/2018/07/amazon-s3...](https://aws.amazon.com/about-aws/whats-
new/2018/07/amazon-s3-announces-increased-request-rate-performance/)

Do you know what exactly a 'prefix' means in this context? The first N
characters? What is N?

------
poxrud
_To ensure there was no way for one user’s data to mingle with another, each
cloud instance had its own dedicated VM, database_

No this will not protect you. Understanding the proper way to configure IAM
roles with principle of least privilege will.

The author mentions that _The issue here is Octopus is like a CI server -
customers can run arbitrary code in the instance (not a problem most SaaS apps
have). So a VM per customer was almost the bare minimum._

If you're using the same IAM role for your EC2 instances, and you're allowing
your customers to run arbitrary code, well now at the very least they have
access to everyone's databases.

------
TedLePoireau
Rent dedicated server from OVH, install ESXi, hire a sysadmin. You will save
huge amount of money. Cloud can be great, but sometime it looks like people
forget there are other solutions.

~~~
megaremote
Dedicated servers are often so much cheaper these days, but it is not the new
cloud, so a lot of startups look the other way.

------
Havoc
Well glad they had the capacity to recover.

Even accounting for hindsight being 20/20...every customer gets their own VM
is a rather ambitious approach to scaling.

Would love to know what made them shift to Azure. Especially while embracing
linux at same time.

~~~
paulstovell
Compute is one part of it but not much different across clouds. Databases are
the other big part (each customer needs their own DB). The SQL Server
databases in Azure for our usage scenario turns out a lot cheaper. We’ll
explain more in a future post.

------
plasma
Octopus is a fantastic deployment product for .NET (happy user here).

~~~
teddyuk
what's it like working for paul? :)

~~~
plasma
I’m a long term (5+ years) customer of Octopus, I know Paul is the CEO, I
don’t work for him, I’m a startup founder with my own company.

------
omouse
Ha, I'm re-reading The Lean Startup and this paragraph seems like a
contradiction in terms of what an MVP is:

 _We decided to build an MVP based on our best estimates and test the market
that way. The goal was to launch something in 6 months and test if the demand
was there; and if it wasn’t, we’d only wasted 6 months. We chose to optimize
for getting to market quickly rather than worrying about how much it would
cost._

Contrast it to what the Lean Startup principles say:

 _The Lean Startup methodology has as a premise that every startup is a grand
experiment that attempts to answer a question. The question is not "Can this
product be built?" Instead, the questions are "Should this product be built?"
and "Can we build a sustainable business around this set of products and
services?"_

So instead of doing what a lean startup would do, they did the _opposite_.
Let's see what they could have done instead...

"How many customers will actually use it?"

\- Lean startup: create a signup page with a mock up or landing page and a few
buttons that work and every potential behind a "coming soon" screen \- Their
approach: actually build the whole product and then launch it and see who's
interested

"What is it all going to cost?" \- Lean startup: As little as possible to
discover the insights we need to iterate and evolve the product \- Their
approach: well we already paid the salary for our engineers and we've given
them six months, so what's half the salary+benefits of X number of engineers?

"What should we charge for it? Is it going to cover the infrastructure costs?"
\- Lean startup: experiment on costing and pricing as you go along \- Their
approach: give away the product for a 30-day trial at a high cost to
ourselves! Unbelievably expensive customer acquisition costs.

Reference:
[http://theleanstartup.com/principles](http://theleanstartup.com/principles)

And now I guess I've truly learned why product managers exist. Someone let
this crazy experiment run for _six months_. You're saying in six months you
couldn't spend one month finding the simplest/cheapest ways to test these
various hypothesis?

$100k customer acquisition spend is insane.

The whole post is just a lesson in contradicting what an MVP _minimal viable_
product is.

 _Even when we were building v1, the team knew it wasn’t the ideal
architecture. Before v1 even launched, there were plenty of conversations in
our Slack about whether we should port Octopus to Linux and run it on
Kubernetes, or see if we could run it on Windows within Kubernetes or use
Nomad by Hashicorp?_

Lol, engineers always want something fun to work on and sometimes it turns out
to be a great idea. But they porting software for an MVP? Why not evolve the
new product so that it generates more revenue or reduce the costs in some
other way instead of talking about porting and rewriting?

If you're at an established company that has money to burn, go ahead, read the
whole blog series. If you're trying to create a new startup, whether it's a
product or service, avoid this article or read it as a warning. When you've
spent as much as a decent engineer costs in _one month_...something's gone off
the rails.

