Hacker News new | past | comments | ask | show | jobs | submit login
MVPs and $100k AWS Bills: Reflections on our launch (octopus.com)
92 points by paulstovell 5 days ago | hide | past | web | favorite | 97 comments





> Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up.

> We also didn’t have our pricing quite right. We initially launched Octopus Cloud at $10/month, with a different pricing model from the one we currently use. Unfortunately, this was one of the most painful lessons we learned because the deficit between what we were charging and spending was magnified by the sheer number of people using Octopus Cloud; continued growth would further amplify the problem.

"Every person who fills in a web form gets to burn a $100 bill of ours" is the real disaster here.

I know it's trendy to think that software isn't like regular business, and you can lose money but make it up on volume, but going to a cloud vendor gives you a real cost of goods sold, and that needs to be accounted for.

(also, the irony of a company whose product appears to be about cloud deployment having a disaster by deploying to the cloud should not be lost on us)


It seems like napkin math could have alerted them to the potential worst case scenario costs. EC2 instance + db instance per customer alone made my jaw drop. Multiply this by 1,000, 10,000, 50,000 and costs are really regrettable.

Edit: I know most database applications like Postgres can handle multiple databases. Does RDS let you have more than 1 database per instance?


EC2 instance + db instance per customer alone made my jaw drop

No, they mention in the article that they were using 30 databases per DB instance. Not sure why they used 30 when the RDS FAQ mentions[1] that for SQL Server you can have up to 100 databases per instance.

However if they would have used another database engine such as MySQL or PostgreSQL, they would not have had any limits to DB's/Schama's per instance and could have had an order of magnitude of cost savings. Using something like AWS Schema Conversion Tool[2] makes it easy to convert between different DB engines, and at a spend of $100K /month they could have easily afforded to pay a consultant to do the migration for them.

[1]https://aws.amazon.com/rds/faqs/ [2]https://docs.aws.amazon.com/SchemaConversionTool/latest/user...


> EC2 instance + db instance per customer alone made my jaw drop.

Especially since the DB is SQL Server; the license costs are death for that. An open source DB would save you a lot there if you have a real need for a DB instance per customer architecture.


Even without the free trial, how on earth can you burn $100 in cloud resources in a month if you plan to resell it for $10?

That's the lesson: The most obvious advice isn't obvious. If one goes through https://startupschool.org/library one realises how much of the advise ought to be common knowledge and common sense, but that's the thing, folks ignore it all the time.

Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't.


> Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't.

Yes, I have seen that happen a few times in SuS (I'm a 2018 and 2019 batch alumni). Founders are obsessive about their pricing and as a result they usually undercharge.

My usual advice to pre-market-fit stage founders is that you don't know yet what the right price is, so choose $random_price and move on, you can always adjust later. And for post-market-fit stage: raise your prices.

Of course, there is no single 'right' answer on pricing. It depends on many factors, but for B2B it usually holds true that pricing isn't that much of a differentiator.


What part of spending $100 to get $10 being a bad idea is not obvious?

The part where it:

* Lets you easily explore a new business offering

* Without investing a large amount of engineering time

As soon as they realized they had significant demand and horrible unit economics, they should have revised their cost structure/app architecture.

Which is exactly what this post is about.


If I start renting Porsches for $10/day I'm going to have massive demand. When I finally build out the rest of my infrastructure, get tired of burning cash, and decide to start charging a price that reflects the true costs, the initial measure of demand is totally irrelevant.

Amazon was built doing just that.

Apples and oranges.

I think the basic idea is 'you need money to make money.' The hope is that with scale and engineering they can burn less. It is dubious that anyone can reduce from 100 to 10 -- 90% reduction would be an incredible feat, and this is why:

AWS has something like profit 25% margin. Therefore assuming that AWS is within 15% of doing a half-decent job at controlling costs, that's still max 40% profit margin if you were running bare-metal and beating AWS at their own game to the tune of 15%. So if you're selling stuff for 10/month, your AWS bill should be 17/month just to break even after all the awesome engineering you've done with the VC money -- at least that's how I see it.

Now, if they are doing incredibly stupid stuff, to the tune of 90% wasted ec2 cpu cycles/db requests, then maybe they've priced it right -- who knows.

Would love to hear some insights on where I might be wrong/misjudging here.


You're not including the scale factor in your reasoning.

AWS has (taking your number as correct) 25% margin at immense scale. Hardware is cheap at this scale. unit pricing for power and space is cheap at this scale. But labor cost is enormous at this scale. Even factoring in leverage of automation, at this scale there is just enormous bureaucracy. One guy or just one team just can't "do something". Anything worth doing will have enormous repercussions across many areas and needs to go up and down the chain. Stuff still gets done, because impact is large at this scale.

> 90% wasted ec2 cpu cycles/db requests,

So then, my point is, if you are just using EC2 and egress, AWS is very expensive. You don't need all the big scale engineering they have, to launch an MVP. Of course, you need to be able to hire the type of people that can manage in-house compute, and that may be hard to find these days, at any price. But taking labor shortage out of the calculation, it's vastly better to DIY.

If you are using PaaS features, that's the sweet spot where you get tremendous value out of AWS.

> profit 25% margin. Therefore assuming that AWS is within 15% of doing a half-decent job at controlling costs, that's still max 40% profit margin.

Getting back to this, even if you dispute all of what I've said, your basic math is wrong. 15% short on 25% is 29.4%. (25/.85, not 25+15).


We’re confident our v2 architecture can. Current costs are between $10-15 per trial. Future posts will go into detail around how it works.

The theory is that if you burn through a few (many) kilodollars doing it the inefficient way then you can at least prove the existence of a market for doing it properly ... and have a head start in actually having subscribers.

I see the theory but $100k/mo in AWS fees is madness. They could probably knock a zero off just by deploying in containers.


Why is that? (Sincere question)

The $10 was the starting price. It’s not uncommon for companies to have a cheaper “loss leader” tier.

A company should figure out what the "real" price should be, and account for the difference in the marketing budget. ie, if the service costs $100/mo to deliver, and if you would otherwise charge $120/mo for the service, but you're only charging $10/mo, you should consider that $110/mo a marketing expense if it needs to be a "loss leader".

It's got to show up in the books somewhere.


The deployment itself worked flawlessly :-)

> the irony of a company whose product appears to be about cloud deployment having a disaster by deploying to the cloud should not be lost on us

Is that really irony though?

I would think irony here would be them writing a blogpost about their troubles with their cloud deployments.


Writing software is hard. These guys have done it a bit meh but there's ten thousands of steps between hello world and what they've done. Their mistakes have got a lot of company - Dropbox once allowed any password to sign in and Steam shipped an "rm -rf /"!

https://hardware.slashdot.org/story/11/06/21/045228/dropbox-...

https://www.theregister.co.uk/2015/01/17/scary_code_of_the_w...


"To bring Octopus Cloud to market quickly, we did the simplest thing possible; we took our self-hosted Octopus Server product and bundled it into an EC2 instance for each customer that signed up. We had to make changes to the product, but mostly around permissions."

I have used octopus and I could have told them that they can't lift-and-shift. The amount of requests the app makes is ridiculous, the way they wrote their database is appalling (actually crazily dumb https://octopus.com/blog/sql-as-document-store). Everything about it would mean they can't scale in the cloud without wasting a ton of money.

"Cloud stuff can be really expensive" - yes if you write dog shit, putting that in the cloud will be expensive.


From the "sql-as-document-store" post, I re-read it every now and then to realise how actually terrible it is:

"For a many-to-many, we do something a bit naughty: we store the values with | characters between each value:"

Not naughty, stupid.


> Not naughty, stupid.

Why didn't they just make 1 table with 1 column and put a JSON object in it with all customer data? This would fit their design objective "Avoid the need for joins as much as possible"! It's brilliant! We'll call it: noSQL, because we don't even have to write that pesky SQL!

Can't wait for the next blog post where they discover another naughty trick called primary keys to increase their NoSQL performance even more!


Why would you bother with a relational database at that point? That sounds like it’s not normalized.

I haven’t read the post... maybe I’m missing something.


because they didn't know what they were doing - I can say this as this is what the blog shows, they don't understand databases and customers (including me) have suffered this

I've learned much too late in my career that competent software engineering is completely orthogonal to creating a successful(from a market perspective) software product.

Well yeah most of it’s just CRUD and that’s pretty straightforward even at a large scale (ex stack overflow.)

> Why would you bother with a relational database at that point? That sounds like it’s not normalized.

It's non-1NF, though they sometimes (based on expected cardinality) normalize that aspect via a trigger, using the application-accessed table as, in effect, something of a write-through denormalized materialized view. It's not an entirely unreasonablev way to use an RDBMS in and of itself.


except they used sql server and sql server performance with this is awful, oracle would have been able to handle it much better

It’s much better now since SQL Server 2017 added a new string split function. But I actually had to do basically the same thing in an earlier version of SQL Server (in my case, it was because I was interacting with a legacy application that spits out pipe-delimited strings) and the string parsing increased the execution time by several orders of magnitude.

For ACID guarantees?

ACID guarantees have nothing to do with relational vs non-relational. FoundationDB is one example of a non-relational key value store that none the less provides ACID guarantees.

Agreed.

Lifting and shifting is hard. It's harder when you do it in a careless way.


The approach seemed reasonable enough until I hit that point.

I was like, ya, I do that sometimes too... Wait wtf?

The best part is the "simplest thing possible". They actually didn't do that at all. I hate to dog-pile on this team, but damn, 100k spend in a month isn't simple at all!

While I agree with you that it seems like they make some serious engineering mistakes, it was a successful MVP in that it told there there was sufficient demand to justify a rewrite for the cloud.

None of this had anything to do with how the database is used.

I have a streaming multimedia app that I want to launch. It ingests from the client, does some transformations, then sends it back to them (or to other parties) in realtime. It's pretty novel though - Snapchat adjacent.

I can't do it clientside. Nature of the problem.

I'm writing it in Rust, so it's relatively fast. But throwing thousands of people at it will still require extreme scale. I'm going to use K8S horizontal autoscaling. This will be obscenely expensive.

I'm not sure I can afford this. But I think a month of successful demonstration will get me funded (or bought out). So in theory my costs could wind up getting paid. Maybe.

Do I risk launching this and potentially having $100k multiples of AWS bills? I can't afford that. I don't want to deplete my life savings or potentially go bankrupt.

Could I put it on a corporate card as an LLC and do my best to make it work? Then fold without harm or fowl if it doesn't? Is that unethical? Could I get sued?

All that said, I'm almost certain that a successful demo at scale with the Internet thrown at it will get me money. It's a really cool and novel product. People will be talking about it.

What should I do? Advice will be very appreciated.


> I'm writing it in Rust, so it's relatively fast. But throwing thousands of people at it will still require extreme scale.

Do some maths.

Get one (1) small EC2 instance on the free tier. Deploy the test version there. Keep adding test clients until the thing is maxed out (presumably on CPU for your application?). That gives you a ceiling for "peak usage". You may find this is surprisingly high.

Come up with a ballpark figure for how many client requests a normal user will make in a day.

You can now start producing cost numbers for using various solutions, from AWS to dedicated hosting. If you don't have deep pockets, do not choose an option which can incur unlimited bills.

Before you can even think of selling it to someone, you need to have a compelling revenue story. Not actual revenue, but a convincing path to revenue. These days that's usually advertising. Can you bung some ads in there from Google to cover startup costs?

Which would you rather have, the system page you in the middle of the night to add more instances manually, or the system do it on your behalf without regard to cost?


> Could I put it on a corporate card as an LLC and do my best to make it work?

Nobody is going to give a line of credit to a brand new company (LLC or not) without a personal guarantee from the owner.

> All that said, I'm almost certain that a successful demo at scale with the Internet thrown at it will get me money.

I would not rack up $100k of personal debt to get to a demo.

But maybe don't listen to me, I'm a 40 year old programmer with a few failed attempts at bootstrapping behind me. Quite possibly at least one of those died due to being capital starved :)


I think you are over confident on the "public face" of your app, and not the "private side" of it. You are saying it would be great and people would talk about it because it is really cool.

I am not saying that's not true - but that isn't enough to make it a product (or being bought out). Some of the most popular products out there aren't hard to do. The complicated part is how to scale them to be as fast with that many users and don't lose money. Twitter? It could be done in a few months (I am not saying it is a weekend project), but that doesn't mean you have Twitter. The hard part of Twitter is not tweet, publish, have followers, is doing in almost realm time with that amount of data. You could say almost the same for YouTube (Upload and play video? Easy. Do it for millions of video at the same time... oops!).

Try to optimize your backend, so it doesn't cost much money. If not, you have just something that does something nice but it doens't do it well. Many people can do that. The secret of success is doing great things without consume many resources.


I would adjust my AWS service limits to prevent scaling past a certain number of instances. Set the limit to serve enough hypothetical users to demonstrate the demo is successful but prevents you from going bankrupt (better to load shed extra users than to end up with a bill you can't pay). If there is no number of instances that fits that criteria, I would consider rearchitecting to be cost conscious (I've heard positive things about Lambda-based services for keeping costs down).

Go simpler and go cheaper. Why not do real-time processing for all of 2 minutes. That's about the length of an elevator pitch and shouldn't cost so much.

Or limit the number of beta users there can be.

Or work some freelance jobs for a few months and save up the cash and during that time pitch to investors. As you progressively have more cash you can hire an engineer to part-time hack on cool demos or to give you the opportunity to increase the number of beta users.


I don't think your problem is unique. I have a few ideas that if I could only get substantial funding for just might work. That said:

>Could I put it on a corporate card as an LLC and do my best to make it work? Then fold without harm or fowl if it doesn't? Is that unethical?

Any bank worth their salt won't simply hand over a huge line of credit for an idea. Banks are risk averse, that's why VC exists. But hey, if you are honest with your plan and they give you money, this is the route.


Approaching VC with a small scale demo is a fantastic idea!

I'm hoping for a quick viral demo, press coverage, and then either more funding or a quick exit. Enough funding to handle the intital traffic will absolutely accomplish what I want.

I've already written the app, so I can show it off.

This is a great idea! Thanks so much! :)


You need plumbing, not a pile of clever rust code.

TL;DR Pay someone who knows how to work with baremetal servers, convert your $100k/mo AWS bill to $20k/mo infrastructure geek bill + < $10k/mo infrastructure bill.

It will give you a longer runway.


onboard customers slowly and charge them - as you scale, make sure your charging as well and get paid.

you don't need to go from 0 - 100k a month in a single month, octopus could because they have backing.


octopus is a bootstrapped company

This is a social media app, so I can't charge money for it. My hope would be for funding or a buyout. Otherwise it will be shut down.

Just stop what you are doing, you need to have a way to generate revenue if you want to sell it.

Trying to wrap my head around this: someone needs to make money from it, or what would they be buying out? So why not you?

If you need funding, then you have to build a compelling case sans demo. It's not unheard of.


> This is a social media app, so I can't charge money for it. My hope would be for funding or a buyout.

If you have no clue how to monetize it, why would anyone fund it or buy it? People fund or buy businesses, but without a plan for monetization, you don't have a business.


I'm sorry but this was a bit of a painful read. Hindsight is 20/20 but this was pretty poor engineering and business modelling from the start.

"To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database, and a large number of security configurations to prevent any funny business.". AWS has a myriad of security and isolation tools to do that in significantly better and cost efficient ways.

"Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up." So you're essentially allowing anyone to press a "Submit" button to spend $100 for you. This sort of below-cost methods to gain marketshare works for Amazon, Google, Microsoft and son because they have the deep pockets but it's a questionable strategy for smaller companies.


The architectural decision to deploy each customer on their own segmented components to make sure that "no one user’s data to mingle with another" seems to me like a solution in search of a problem.

First, the deployment was on AWS (EC2 and RDS among other services) which was probably already co-mingled with other AWS customers.

Second, when you are selling a SAAS offering, the assumption is already there that you'll build security and multi-tenancy into the app and the underlying infrastructure shouldn't matter.

And 100K/month for a MVP SAAS offering. I'm speechless. I just, I can't even understand how this burn rate passed muster with the leadership.

But Kudos to the team for owning up to it and sharing. Lots of lessons learned here.


The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum.

Cloud CI platforms typically solve this by timesharing build agents, but that takes engineering effort, which is a waste of effort if there's not a lot of demand in the first place. That's why we started with the MVP.


It could make alot of sense for alot of government, healthcare and financial use cases.

I agree. In those cases, sell those customer separate products and expectations and price it accordingly.

The lesson to be learned here, and the one they admittedly already knew in advance, is if you do a lift and shift to AWS without changing your processes to be cloud native, you will always spend more.

Unfortunately, I’ve met way too many old school netops guys who got one AWS certification and all they knew was how to log into the web console and duplicate their colo infrastructure, charge their client a lot of money, and call it a day.

That being said, I’ve used a lot of CI/CD tools and OctopusDeploy is by far my favorite.

Even though my company is very much a “pay someone to do the undifferentiated heavy lifting” type of company, OctopusDeploy is so easy to use and maintain, I am not sure we would migrate.


Going to the cloud won't save you money.

Going to the cloud lets you scale beyond your own data centre

Going to the cloud can be cheaper if you use the elasticity and make good use of PaaS services (servless,paas, etc)


I agree, but it also lets you automate and probably lets you get rid of or not hire as many infrastructure people between automation and outsourcing the simple stuff to a managed service provider.

This is insane. It's the engineering equivalent of smashing one of your hands with a hammer and then being surprised it hurts. Well no shit.

At that level, at the very least, it would make sense to run some amount of physical infrastructure.


Pardon me for asking a really dumb question: but do you have a positive item charge - CGS? Because I'm sure one can get a great demand by selling a $1.00 for $0.90 but that's not a business model.

The post didn't make this very clear. $10 was the starting price - at the time it was for 5 users. Similar to Confluence or Jira with a very cheap (in this case loss-making) starting tier. Once a customer got to ~10 users it became profitable.

What we didn't know if whether there'd be enough customers for us to care about the loss on those $10 tiers. E.g., if only 100 customers signed up a month, the costs would have been around $10K/month - enough to write off as a marketing expense. Once it got to $100K it became something we needed to fix.


> The post didn't make this very clear. $10 was the starting price - at the time it was for 5 users. Similar to Confluence or Jira with a very cheap (in this case loss-making) starting tier. Once a customer got to ~10 users it became profitable.

Err.. That's not a correct way of looking at a long tail. CGS on a long tail must still be lower than the unit charge, otherwise you are paying $1.00 for $0.90 which gives you an incorrect signal about the demand.

Say you make apple juice. And each juice uses 4 apples and only apples. 4 apples cost you $3. Setting a price of juice to $1.00 does not actually tell you if there's a demand for the juice because it is below the CGS, which you cannot make up in volume.

The long tail test is to set the juice price at (CGS + x) where x is some number and see if you get the demand even if x is less than the other costs attributed to juice ( rent/electric/juicer/salaries). If you do, then you can make up your loses in volume because the higher the volume you are going to have the smaller process cost attributed to every unit will be so there's going to be a number at which CGS + y < CGS + x where y is the process cost attributed to a unit.


That is Amazon's business model.

Amazon business model:

$cost_of_a_thing < $charges_of_customer.

$cost_of_a_thing + $additional_expenses_allocated_to_ thing could be > $charges_of_customer

Described business model:

$cost_of_a_thing > $charges_of_customer.

hence $cost_of_a_thing + $other_expenses_allocated_to_thing is > $charges_of_customer

First one can be profitable via scale. Second cannot.


Can't wait for this Cloud meme to die out and we can go back to On Prem but instead of VMs managed by VMWare, Cloudstack, Vagrant, LXC etc, we will have nodes managed by Kubernetes.

Why would cloud go away? If you’re a team of 3-4 people and have a startup idea what’s easier? Create an AWS account and smartly spin up required servers?

Or find a location (rent$), get fiber($$) to that location($$), find hardware($$), install said hardware. Now you’re back to where you were with AWS.

For larger already established businesses on prem is probably a setup you should explore but on day 0 of a new business, cloud will probably have a large cost and flexibility advantage as long as you smartly set it up.


I can purchase and then colocate 2x Dell R640s (8c/16t, 64gb of ram, as much NVMe storage as I can stomach) with tier 1 bandwidth billed at the 95th percentile and 24/7 remote hands support for about $150/month. The purchase price on the Dells is about $3500/ea.

Total first year cost on this setup: $8,800

Total three year cost on this setup: $12,400

2x m5d.4xlarge EC2 instances (this isn't counting ingress/egress rates, or really anything else) First year cost: $15,621 .

Total three year cost for the EC2 instance: $46,863.36

For the cost of using EC2 you could hire an extra person to help grow your company.

Yes, you can get discounts in AWS for agreeing to a three year term. Yes AWS offers you flexibility to walk away at any time.

If you have a base line load (which most companies do) you should really look at colocating your owned equipment. You also get the bonus of being out of the watchful eye of Amazon's profitability team looking to see if they can copy what you're doing.


> For the cost of using EC2 you could hire an extra person to help grow your company.

You _will_ be hiring an extra person to manage these machines. That's the beauty of The Cloud (tm). Need more compute? You got it. Provisioned too much? Get rid of it. If you're a startup, you don't have time nor money to be managing your own infra.

Not to mention the ability to deploy to _any_ region within minutes. Now you're talking $8,800 / region + someone to maintain other regions.

> ingress/egress rates

unless you're also using S3 or CloudFront, this doesn't really mean anything. Put a free CDN in-front of S3 or your instances if you're worried about load or cost of IO.

> You also get the bonus of being out of the watchful eye of Amazon's profitability team looking to see if they can copy what you're doing

Ok now you're reaching.


Once you get beyond a few instances you'll need a fulltime person to manage your cloud assets anyway. VPC & associated networking + managing different services is not a trivial task. You'll still have an ops team. Whether it's old school sys admins or devops coders is down to your choice of where you run your system.

How much management does a bare metal box need? Nothing more than what an ec2 instance would need.

There are colocation providers all over the world. Psychz.net is a great example. My assumption is a single box per region. There is no redundancy in your ec2 systems.

You should google the Amazon profitability team if you don't believe me.


I can't find anything related to AWS and "profitability team" on Google. Can you share some info/links?


2x m5d.4xlarge EC2 instances for 3 year no-upfront reserve pricing is $20,813.76 (You should really do that).

But what are you getting 2x m5d.4xlarge instances from the get go? Start with the smallest instance size your app will run on and then add them on one at a time? Need to run 10 servers in the day but only 2 at night? Then configure autoscaling to do that. What happens if you need to expand out into another part of the world? Okay, now you just copy your setup to another Region instead of trying to find another provider in another country.

What if your service becomes super popular (You got on the front page of HackerNews), with autoscaling, it just grows automatically. No scrambling to get more servers, it just scales.

Don't listen to the "Woe is me!" stories about using the Cloud and feel sorry for these companies. They designed poorly, wrote crappy quick code and paid a price for it.


also, spot instance is dead cheap. a reduction of 60-70% in cost is possible with the use of mixed reserved instances and spot instances.

I'm pretty sure colocation is cheaper if you know what you're doing (what to buy, where to colocate, what kind of bandwidth required, etc). I'm also pretty sure a good portion of cloud customers don't know or don't want to risk it.

It probably depends what you measure. How do you scale up to a meet a relatively quick increase in demand? Will you lose customers during that time and how does that factor into your costs?

I think, at least for some business types, colocation will usually be more expensive because you have to overprovision or risk losing customers.


Aren't we moving to a world where K8s will be spread across multiple on prem and different cloud providers with K8s running on VMs. The hope is that devs will have less to do since it would be plug-n-play

> We quickly learned that everything in AWS has some kind of service limit

Except S3. Unlimited objects, and unlimited Put and Get requests.


Your comment is not true in all cases. S3 is limited to 5,000 get/put requests per prefix. So while it's possible to achieve what you've said, it's also possible to stumble upon this limit in a fashion similar to the original comment.

It looks like the number is slightly higher now - https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...

Do you know what exactly a 'prefix' means in this context? The first N characters? What is N?


Just keep in mind if you're using SSE KMS to encrypt you are subject to limits of that service when using S3 (there are a few ways to mitigate this).

To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database

No this will not protect you. Understanding the proper way to configure IAM roles with principle of least privilege will.

The author mentions that The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum.

If you're using the same IAM role for your EC2 instances, and you're allowing your customers to run arbitrary code, well now at the very least they have access to everyone's databases.


Rent dedicated server from OVH, install ESXi, hire a sysadmin. You will save huge amount of money. Cloud can be great, but sometime it looks like people forget there are other solutions.

Dedicated servers are often so much cheaper these days, but it is not the new cloud, so a lot of startups look the other way.

Well glad they had the capacity to recover.

Even accounting for hindsight being 20/20...every customer gets their own VM is a rather ambitious approach to scaling.

Would love to know what made them shift to Azure. Especially while embracing linux at same time.


Compute is one part of it but not much different across clouds. Databases are the other big part (each customer needs their own DB). The SQL Server databases in Azure for our usage scenario turns out a lot cheaper. We’ll explain more in a future post.

Octopus is a fantastic deployment product for .NET (happy user here).

I agree. I've used Octopus to automate deployments of internal LOB crud apps in the insurance space for a few years now.

My questions is, is it still aimed at .NET? I know it leans on Nuget as the package repo. But have they broadened to embrace deployment of other tech stacks?

Is Octopus considered best in class for what it does? Who are the primary competitors? Does Azure DevOps spell the end for Octopus?


Should I reply here or to sales@octopus.com?

what's it like working for paul? :)

I’m a long term (5+ years) customer of Octopus, I know Paul is the CEO, I don’t work for him, I’m a startup founder with my own company.

Ha, I'm re-reading The Lean Startup and this paragraph seems like a contradiction in terms of what an MVP is:

We decided to build an MVP based on our best estimates and test the market that way. The goal was to launch something in 6 months and test if the demand was there; and if it wasn’t, we’d only wasted 6 months. We chose to optimize for getting to market quickly rather than worrying about how much it would cost.

Contrast it to what the Lean Startup principles say:

The Lean Startup methodology has as a premise that every startup is a grand experiment that attempts to answer a question. The question is not "Can this product be built?" Instead, the questions are "Should this product be built?" and "Can we build a sustainable business around this set of products and services?"

So instead of doing what a lean startup would do, they did the opposite. Let's see what they could have done instead...

"How many customers will actually use it?"

- Lean startup: create a signup page with a mock up or landing page and a few buttons that work and every potential behind a "coming soon" screen - Their approach: actually build the whole product and then launch it and see who's interested

"What is it all going to cost?" - Lean startup: As little as possible to discover the insights we need to iterate and evolve the product - Their approach: well we already paid the salary for our engineers and we've given them six months, so what's half the salary+benefits of X number of engineers?

"What should we charge for it? Is it going to cover the infrastructure costs?" - Lean startup: experiment on costing and pricing as you go along - Their approach: give away the product for a 30-day trial at a high cost to ourselves! Unbelievably expensive customer acquisition costs.

Reference: http://theleanstartup.com/principles

And now I guess I've truly learned why product managers exist. Someone let this crazy experiment run for six months. You're saying in six months you couldn't spend one month finding the simplest/cheapest ways to test these various hypothesis?

$100k customer acquisition spend is insane.

The whole post is just a lesson in contradicting what an MVP minimal viable product is.

Even when we were building v1, the team knew it wasn’t the ideal architecture. Before v1 even launched, there were plenty of conversations in our Slack about whether we should port Octopus to Linux and run it on Kubernetes, or see if we could run it on Windows within Kubernetes or use Nomad by Hashicorp?

Lol, engineers always want something fun to work on and sometimes it turns out to be a great idea. But they porting software for an MVP? Why not evolve the new product so that it generates more revenue or reduce the costs in some other way instead of talking about porting and rewriting?

If you're at an established company that has money to burn, go ahead, read the whole blog series. If you're trying to create a new startup, whether it's a product or service, avoid this article or read it as a warning. When you've spent as much as a decent engineer costs in one month...something's gone off the rails.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: