Hacker News new | comments | show | ask | jobs | submit login

Definitely looks like several "teach-able moments" here: They learned the hard way about:

1. Developing a fix without understanding root cause (try-something development)

2. Sufficient testing, including load testing, prior to initial deployment

3. Better change control after initial deployment

4. Sufficient testing for changes after initial deployment

5. Rollback ability (Why wasn't that an option?)

6. Crisis management (What was the plan if they didn't miraculously find the bad line of code? When would they pull the plug on the site? Was there a contingency plan?)

7. Perfect being the enemy of good enough

Looks like they were bailed out of the cost but what if that didn't happen?




2-7 are sort of understandable for a quick, hacky startup just trying to ship something fast, with minimal experience. But 1 is the really crazy one. Load times spiking to 30 seconds once they start getting significant traffic, and instead of doing a solid investigation, including instrumenting their FE and-or server so they can see where the slowness is (maybe Firebase even includes decent observability by default?), they go all in on upgrading Angular for seemingly no reason? That’s just ... completely illogical.


That raised an eyebrow for me, too. I recently moved a project from Angular 1 to 6, so this is fresh in my mind. There aren't enough changes between 4 to 6 (or even 1 to 6) to cause mysterious 30 second lag times by themselves. And upgrading the front end's base framework under a time crunch is almost always a bad decision, regardless of which framework you're using.

In the companies I've worked for, these guys would be written up and likely put on a performance improvement plan, if not flatly fired.


0. Choose a platform that doesn't bankrupt you if you're less than perfect at 1 to 7.


Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.


Yes, it's a good investigation tool, but definitely not a fix. That is, if "it started to work now, I don't know why", you're still in trouble, and should continue digging.


updating the framework version is like.. literally the last try-something thing to do though.


I think "try-anything" would be a better characterization in this case. They had smoke pouring out of the engine and tried changing the tire.


No disagreement there.

I don't understand how you can build a complex application like that without doing basic performance checks like, are we hitting the file system or database too often, our the image assets correctly sized, etc.

I'm not a software engineer however.


0. shut the app down before it costs even more money.

A money clock on the table isn't fun and if you replace the the app with a landing page and a newsletter form it's completely acceptable for the visitors.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: