Of course we started spending all the money on new people and AWS, and soon there was no money.
At one point we were dumping like $15K a month on AWS for a dozen unnecessary over-engineered toys that nobody was using. This is the real cost of AWS.
I'd love to see Amazon's data on money invested vs actual user traffic for small startups, that's got to be some of the most interesting and valuable data on earth. Forget companies, I'll bet Jeff is sitting around predicting when entire industries rise and fall weeks before anyone else based just on this data.
Using AWS as an example, at one of the businesses I worked at, we used Kinesis as an event bus. One shard handles 2MB/sec of output. This worked pretty well for thousands of messages a second, we even got up to the 100's of thousands by compressing the payload of the event message. After that, you can employ any number of strategies that work easily, such as shading and adding additional streams, and use a lambda to pipe the output of one stream to another. It scaled up to millions of messages by essentially pushing buttons in the AWS Console.
Take a look at your architecture. More than likely, outside of a FB/Google/Netflix traffic scenario there is probably an easier and more straightforward architecture you can use that scales to your realistic use cases. Worry about the billions when you get there, which you yourself probably wont at that point because you would have exited, or moved into a higher role most likely by now.
I can now show exactly which departments and dev teams are driving all the costs, and on what (CPU, storage, network). In a way that I never could for on-prem stuff.
The closest I got to an org that did this well was a big company that ran Cloud Custodian in all their AWS accounts and if you launched an EC2 instance, it would terminate it immediately with extreme prejudice if it didn't have values for three required tags, one to identify the "owner" individually and two for accounting purposes.
The only problem with that is there's no mechanism to make sure that the values of the cost centers values were correct. There was a bit of a scandal when one group (who presumably just copied and pasted a bunch of CloudFormation from another group's repo) was running 5 figures a months of infrastructure under the other group's billing codes.
ALSO, as many have said, bandwidth is a big part of the cost, and at this time it's nearly impossible to do showback/chargeback on bandwidth. There may be a way to do it using Flow Logs by correlating IP addresses to instances and using those tags, but I've never heard of someone doing this successfully.
In this case, a service tag, set in some cases, not in others.
For some apps, perhaps a "component" or "service" tag would also be important.
Startup I was at burned 400k/month average sometimes when buying RI we spent 800k.
Had a couple engineers route the database connections over the public load balancers for a month, that cost 20k in network alone.
AWS is not cheap at scale, period.
I have many stories about this startup. The idea is great. If the startup succeeds it will be in spite of how its being managed technically
Problems come from many areas... like the fact it was in the data management space and the 1500 or so i3's running Cassandra plus the hybrid cloud approach of front end in AWS talking to APIs in GKE which talked to backends in AWS because cool technologies you know.
No architecting was done. Build this and put it in the cloud. The team from RU didn't even know about autoscaling groups and when I tried to bring them in I ran into resistance.
I have many examples of "opportunities to optimize" from this place, needles to say my stress level is much lower not being there.
- Let's not hire for this idea because it eats into our burn
- Hmm, let's hold off on launching this feature until we have more data
If the company you worked for was profitable, then they could've structured a leveraged debt payoff to the investors to get them off the cap-table. Unless the company took so much money that the investors owned 60%+ and they unanimously do not agree about being profitable, then this is something that can be passed as a board resolution.
It sounds like the founders at your company were just inexperienced.
Somebody mentioned 1:3.
Would there be any "conflict of interest" issues is some of the same people requiring AWS spending were also Amazon shareholders?
If you think anyone cares about any conflict of interest among the investing class you're beyond naive, you're just delusional.