>Over the last two and a half years, we’ve picked up a few tips and tools about scaling Postgres that we wanted to share—things we wish we knew when we first launched Instagram.
A common failure mode for myself and, I suspect, others, is thinking that we have to know every single thing before we start. Good old geek perfectionism of wanting an ideal, elegant setup. A sort of Platonic Ideal, if you will.
These guys went on to build one of the hottest web properties on earth, and they didn't get it all right up front.
If you're postponing something because you think you need to master all the intricacies of EC2, Postgres, Rails or $Technology_Name, pay close attention to this example. While they were launching growing, and being acquired for a cool billion, were you agonizing over the perfect hba.conf?
I think Instagram shows pretty much the opposite example of the "throw anything at the wall with any techno you want and fix later" mentality that is trendy todays.
They were seasoned enough to choose Python and Postgres, two technologies that are both relatively easy to start with and to scale later.
And, oddly enough, those two technologies are on the "do things right" side, and do not (usually) sacrifice their correctness to some convenience or fashion.
So, sure, one cannot expect to know every corner of the techno one chooses, but still choosing carefully the best options and shielding oneself against current hotness is still not a waste of time or energy.
I can't remember where I read/heard it, but they received advice about what to use, and change to use in order to scale better, and were willing to listen to that advice.
So the lesson is not just be good at what you do, but be willing to listen to other people who knows better than you do.
I think you are correct in that you don't need to know everything to get started. But it also shows the benefits of maturity of the technology stack. As much as I would love to develop my geospatial analytics app in Clojure...I'm doing it in Python/GeoDjango. And unlike my attempt in Clojure, it is going incredibly smoothly. The language bugs me, but it is better than tracking down bugs due to the immaturity of the ecosystem.
Yes, chasing a platonic ideal is a good way of putting it. It's right there in the name: best practices. It's important to realize there aren't really any _best_ practices. To actually start a project, we have to give ourselves permission (or straight up force ourselves) to use good, or maybe even just acceptable, practices and then polish from there.
I like how you phrase it: We need to give ourselves permission. I think that sometimes, we're afraid that someday "they" will discover the imperfections behind our code, infrastructure, etc. that lies beneath what we build. I suspect that ofttimes, the 'they' is really ourselves.
A shame, really: The very thing that drives us to be the best ends up holding us back.
Maybe you are giving instagram too much credit? Isnt it just a photo uploading website. Sure the "scalability" challenges might be non-trivial but definitely not rocket science
>Over the last two and a half years, we’ve picked up a few tips and tools about scaling Postgres that we wanted to share—things we wish we knew when we first launched Instagram.
A common failure mode for myself and, I suspect, others, is thinking that we have to know every single thing before we start. Good old geek perfectionism of wanting an ideal, elegant setup. A sort of Platonic Ideal, if you will.
These guys went on to build one of the hottest web properties on earth, and they didn't get it all right up front.
If you're postponing something because you think you need to master all the intricacies of EC2, Postgres, Rails or $Technology_Name, pay close attention to this example. While they were launching growing, and being acquired for a cool billion, were you agonizing over the perfect hba.conf?
More a note to myself than anything else :)