We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down.
So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually launching instances.
The increased scale of instances pummeled the control plane
with thousands of instances all trying to come online and pull down their needed data to get operational -- which them killed our DBs, pipeline etc...
We had to reboot our entire production environment at peak service time...
The poster is aware of this, which is why they talked specifically what they did wrong.
Think long and hard why you felt it necessary to make your comment and what value it actually provided.
Yeah, that's what I was telling the other poster in a nicer way. Same goes to you. You can provide advice without repeating criticism for no reason when the person specifically said they did something wrong.
And save me the lecturing on "long and hard." My comment has 15 upvotes, so it's pretty unlikely I'm the one off the mark in this conversation.
That statement is a joke, here it is reworded:
> A person should be invulnerable to criticism as long as they make a humbling remark.
Doesn't sound so great now does it.
Also, I'm not surprised you got 15 upvotes. This place has ceased to be a hacker forum for many years now. Too many eternal politically correct Septembers.
You were the one using upvotes to validate your argument when it's a fact that posting a political opinion in either a left and right forum will net highly different responses. Of course the environment plays a part.
Won't even bother dissecting the first shot. Your arguments have been weak at best till now this final one was the final straw, man.
You know what I think its a fault of:
Lack of a canonical DevOps "university" stemmed from SV startups.
DevOps at this point should not just be a degree -- it should be a freaking field of study and classes offered by YC.... Look at their pedigree of companies at scale. We should all make an effort to give back in this regard...
Couldn't query, but could initiate.
Would love to hear what others are also tracking.
Goes to show what their business numbers are....
We also added smart loading across AZs due to spot instances getting whacked when our fleet was outbid and AWS took them back.
As well as other monitoring methods to be sure we werent caught with a smart system doing dumb things.
Limits are malleable based on your use case. Speak with your rep.
You might even not know how limits came to be... I am.
I have had a time when git suffered a flaw, and a junior dev also suffered a flaw in checking in secrets.... thousands of instances across the globe were launched... for bitcoin mining... $700,000 in a few hours...
We all learned a bit that day.