
Cloud migrating production first in 12 hours or less - wtf_did_we_do
Cloud migrating production  without a plan, and how it made me a better programmer ;)<p>We have a b2b webapp product in cloud_provider_a which started having trouble serving requests, timing out.<p>My team (supporting another product in cloud_provider_b) was tasked with setting up a duplicate of prod in cloud_provider_b. I can&#x27;t even remember why mgmt thought that would help.<p>By the end of the day, by hook or by crook the call was made to switch PRODUCTION traffic into a newly made stack in a different cloud than where that application was currently hosted. Despite protest from the engineering and QA teams.<p>Its all a blur now. Things went horribly wrong for two days. On the third day my team (with no prior knowledge of how this product works) found that the docker containers running the app had an nginx config with a low worker_connections setting.<p>We changed that one line and the app is back up. But the cost has been high, our customers had a far higher level of disruption these 3 days. Our engineers feel burnt out and disrespected, and our QA teams are appalled.<p>AMA I guess
======
wtf_did_we_do
I read here a while back that as Engineers we have an ethical obligation to
only present good options to management. That when pressed for more options we
need to be careful on what we present.

I think I have gained a lot of understanding around this now, and have pushed
back time and time again to prevent management demanding dangerous and risky
changes without even quantifying the impact of those, positive or negative.

If only I had that before this mess began.

------
imhoguy
I see same too often recently. There is too much printed money on tables.
Quality assurance degrades, business in scoping histeria, management
micromanaging everything ad-hoc aka flat but same old structures, engineering
forced to deliver over weekends and cut every corner.

This bubble has to crash or people will go mental en masse.

------
K0SM0S
Why aren't you interviewing for another job right now instead of sharing on
HN? ;-) haha

Joke aside. "No pain no gain" as they say. So what's one thing that was
painful today but eventually is learning / growth / wisdom for you?

~~~
wtf_did_we_do
I am. I think others in the company are now too. To be told that our knowledge
and expertise isn't respected was a real blow.

I did also learn a lot about cloud architecture though.

~~~
K0SM0S
Ah yeah nothing like the most barebones PoC to learn a lot about something.
Bird's eye view and all that.

Good that you're already thinking 'next'!

The blow you speak of is common unfortunately between different types, and
there is always a sorta 'dominant' type in every business. Sometimes it's the
engineers, sometimes it's the marketing, sometimes it's sales, sometimes it's
c-execs. That's one piece of info worth having to make more informed career
choices, depending on your goal — you usually thrive as part of the dominant
type, i.e. "ethos" of the company. Look at the CEO and background of the
managing backbone. It usually says a lot about a company's culture, who
_matters_ and who's recycled perpetually.

~~~
wtf_did_we_do
It's definitely engineering that get recycled, and our C level is made of
people focused on sales.

Now I know what to look for in a company, this event paid out in wisdom coins
for me.

------
tuldia
That is great! You are doing system administration now, enjoy!

I believe people should be doing that more often, you know, just moving stuff
around.

~~~
wtf_did_we_do
I wonder for how much longer?

If my team was asked to spend a day investigating instead of migrating we
would have found it and solved it without any overtime and midnight call outs.

At least I've grown a pair over the last few days.

