Nothing like this story of getting bullied to do something, but I once talked a director out of rebooting the cell phone switch that handled all cellular phone calls for I think it was 4 provinces in Canada. This is something I don't think we had ever done before, and was expected to be a 100% outage for all cell phones calls + 911 for like 20 - 25 minutes or something.
We were having elevated error rates due to an earlier outage, and had just run the next closest hail mary to recovery the switch by telling it to resync all temporary state. The problem was that resync didn't work the way anyone on the team thought it did and the director didn't give enough time for it to be effective. We later found out the resync for a particular record didn't happen until the cell phone itself checked in with the network for a periodic update, so we had to wait for however long that timer was to expire on all cell phones. Luckily there was an active 911 call on the switch which caused the team to delay long enough for me to get the right data (the real world failure rate was far below 1% and starting to trend towards normal after the resync command) and talk the director out of the reboot.
The funny thing is that equipment wasn't even mine, I worked on the internet side of cell phones, and I don't even remember why I was pulled onto that outage call since it didn't have anything to do with my part of the network.
> The funny thing is that equipment wasn't even mine,
Did you have a reputation for being good at debugging things? Expertise and experience in one area often has crossover in connected systems.
I got pulled in to advise on an automated options market making system written in Java when the closest I'd done was equity execution algorithms in Slang and C++ (and integration with the options automated delta hedging facility).
Yea I did alot of debugging of the networks hardest problems, and the crossover is especially true with telecom systems since all the inter-connectivity comes from the standards bodies, which share a lot of the same base and history. But some stuff differed as well, it was difficult for a bunch of the hardened circuit switched folks to move to an all IP world, and I only had a cursory view into the circuit switched world which was on it's way out when I was in industry.
A long time ago a customer called in and asked me to install some hardware they sent (an Ethernet card for a router) right away. I declined - the router it needed to be installed in was a shared device, and we'd had problems with hot swapping cards[0] in this model of router before. I told them I'd do it after hours that evening, but not during the day. This would have been maybe a 3 or 4 hour delay for them.
About 30 minutes later, my boss comes into my office and says the owner of the other company called to complain that they felt like "marcus0x62 never wants to help us". (I had - just the week before - gone into the data center on Friday night on about 1 hour notice to do something for them without complaint of any kind.) My boss said "I don't need [the other company] thinking we don't want to help them." He seemed pretty angry and wasn't interested in any explanations. Ok, I said. No problem. I'll take care of it right away. I figured there was, based on past history, about a 25% chance this router would reboot on the spot when I put that card it.
So, I went into the data center and inserted their card into an open slot. And all the blinkenlights lit up. I went back into my office, and the boss was still there. I told him about 25% of our WAN customers would be down for 5 minutes while this router rebooted, and the tech support staff should expect calls.
0 - The router, a Cisco 7200 series device, supported "Online Insertion and Removal", but it didn't always work. And when it didn't work, the router usually rebooted spontaneously.
I once fought a bad boss for a year. I finally recognized my only remaining power was my ability to quit.
The system continued to run for about 9 months after I quit before it started having weekly outages, and my new job pays me more with better coworkers.
I wonder how it turned out for Norman, I hope his manager did not blame him. These days seems you need provable evidence to be safe in some jobs.
I had a different but similar situation, but for me, my immediate manager supported me. So I did not have to do what I was being told (I was going to quit). It was a VP who wanted that activity done. All turned out fine for us and I stayed there for many more years.
The more often you walk out on an abusive boss, the easier it gets.
I remember feeling physically sick, the first time.
By the time I walked out on my last tech* position, it wasn't even a conscious decision; I was gathering my tools and heading to the door without even thinking about it.
* I was 2 hours late on site - after rescuing 2 girls from a wreck on the M5, and being made to hang around and give a statement by the Police; even though I stayed onsite to finish the job beyond normal end of day - I got docked half a day's pay and told I should have left the girls to die.
Fun Fact: EVERY firm I walked out on, went bust within 24 months of my leaving; am I a jinx, or have a 6th sense?
my google discover feed used to fill up with stories like this one: "my boss told me X so I Y". It took a while for the "I'm not interested" feedback loop to kick in.
Recent example:
"my boss told me I'm not allowed to work from home so I uninstalled teams. now he's mad he can't reach me after hours"
This entry even the title has a typo in it. These mostly read the same. I think the prompt is something like "this guy was told by his manager to yada yada. he was not having it so he yada yada. make an article of about 5 to 8 paragraphs"
We were having elevated error rates due to an earlier outage, and had just run the next closest hail mary to recovery the switch by telling it to resync all temporary state. The problem was that resync didn't work the way anyone on the team thought it did and the director didn't give enough time for it to be effective. We later found out the resync for a particular record didn't happen until the cell phone itself checked in with the network for a periodic update, so we had to wait for however long that timer was to expire on all cell phones. Luckily there was an active 911 call on the switch which caused the team to delay long enough for me to get the right data (the real world failure rate was far below 1% and starting to trend towards normal after the resync command) and talk the director out of the reboot.
The funny thing is that equipment wasn't even mine, I worked on the internet side of cell phones, and I don't even remember why I was pulled onto that outage call since it didn't have anything to do with my part of the network.
reply