> You develop your first little toy app using Framework X.
> ...
> Then one day you become Twitter, and all hell breaks loose.
Really? Web apps usually become Twitter? I agree that knowing how to scale a web app is important, and that it's not always exactly straightforward, and that it could be made easier with some better tools. Heck, I even gave a talk at PyCon about it (http://www.polimetrix.com/pycon/slides/).
However, statistically speaking, almost zero percent of web apps end up like Twitter. And if they do, they have VCs knocking down their door so hard that they can afford to hire enough engineers and buy enough hardware that scaling won't be an issue for long. That's a problem we'd all love to have.
Not that we shouldn't continue to improve our server-side technology, but 99.9% of my pain developing web apps these days is due to client-side issues. The stuff that runs on the server usually Just Works and is an absolute pleasure to code.
hire enough engineers and buy enough hardware that scaling won't be an issue for long
I don't know about that. Twitter is still having problems with downtime due to scaling. The way I see it: don't completely ignore scaling, but don't focus on it (at all) at the beginning. Keep it in your mind though, and try not to close yourself in, and create something which can't be scaled without significant modifications to services. Do not confuse that with modifications to your code. If you're going to scale large, expect at least one complete re-write, but don't offer services which are incapable of scaling.
Within the context of describing scaling pain, I think it's appropriate, if a bit hyperbolic. Not everybody becomes as big as Twitter, but lots of people get big enough to outgrow their original set of tools.
Agreed. My experience is that optimizing and scaling your app from day one in preparation for Twitter-like traffic is a self-defeating prophecy. Web app scaling is hard, and if you spend so much effort on it from the beginning it will leave little time for actual development, and as a result your web app will most likely fail to become popular.
I'm convinced that scaling is something you need to worry about after you become popular. There's no point to worry about it any earlier.
Does anyone have thoughts about the utility of cappuccino for web app development where you load a GUI and then do all your state control for each [user not model perspective] entity via rpc to the server? I haven't tried it yet but expect this will be easier because you can bypass much of the grunt work of templating, and you don't have to deal with the back button issues either. I think the back button maps OK onto entities (i.e. get the user to link if they're going to a new booking) but not state transitions within an entity (i.e. if they add a person to the booking then there's no need to provide back functionality).
"The big problem, though - the elephant in the room - is that gateways suck, too. Gateways aren’t APIs." Too true.
On python - the python critics could fork, stick in the features they want, and benefit from the libraries. I'd have thought this would be a better starting point for a new language in that direction than a blank canvas. I admire clojure for similar reasons - find a VM and leverage it.
Very helpful! A web app recently jumped into my hobby project hopper, and the last thing I can remember of web work is CGI--and it's all new and shiny the way I remember it, too. I knew that was a bad sign, I knew I was so out of touch that I didn't even know what questions to ask . . . but I didn't realize a whole generation had come and gone without me.
the house analogy breaks down when you consider that you can build a house to look like anything and it doesn't have to interface with anything else, like your neighbors' houses.
a web application has to interface with standard browsers and protocols, so maybe it's like a car that has to operate on standard roads and can't take up 2 lanes or fly.
Actually, the things houses have to 'interface' with are so standardized you don't even think about it. Appliances, plumbing fixtures, computers, furniture. Everything is standard sizes, so they fit through doors, plug in and hook up properly.
Lumber dimensions, electrical connections, even door sizes are standardized.
Yes. It's so practical compared to old-school GUI development, and so it pulls you in and mushrooms into a scaling/integration/maintenance fiasco unless you can say "no no no!" to every non-essential feature. Things like Django, JQuery, HTML5 are app-level improvements but we're still asking too much of our feeble software infrastructure.
Step back, ask what we're trying to do with the web (decentralized computing?), and come up with a system that solves the basic UI/OS/protocol/language needs "once and for all". Implement it as a kludge on top of the web; use and refine; discard the web; discard the old OSes. It's a pretty tall order :-)
[To the person I replied to, who shall remain nameless] No reason to delete your comments, I agreed with much of what you said and it was valuable to the discussion. Odd.
Let's add a dose of perspective here. Do a job search on craigslist or simplyhired for "php," "rails" and "django" and you'll see that there are vastly more PHP jobs than either (10,726 on simply hired right now) and many more Rails jobs than Django jobs (2,553 vs 328). Django is popular with some circles, such as people reading HN on Saturday nights (which is when that poll was posted), but this does not reflect actual usage.
"Current PHP openings are more than likely for legacy web development."
That's quite a massive assumption. If we do a job search on startup-specific job boards, (like startuply.com and startuphire.com), you'll see the same sort of thing. On Statuply, "php" returns 280 hits, "rails" 121 and "django" 35. On StartUpHire, "php" 458, "rails" 119, "django" 9.
Long story short, don't believe every poll you read on the internet.
Also, don't take this as me talking bad about Django; it's a great framework and programming is not a popularity contest.
> You develop your first little toy app using Framework X.
> ...
> Then one day you become Twitter, and all hell breaks loose.
Really? Web apps usually become Twitter? I agree that knowing how to scale a web app is important, and that it's not always exactly straightforward, and that it could be made easier with some better tools. Heck, I even gave a talk at PyCon about it (http://www.polimetrix.com/pycon/slides/).
However, statistically speaking, almost zero percent of web apps end up like Twitter. And if they do, they have VCs knocking down their door so hard that they can afford to hire enough engineers and buy enough hardware that scaling won't be an issue for long. That's a problem we'd all love to have.
Not that we shouldn't continue to improve our server-side technology, but 99.9% of my pain developing web apps these days is due to client-side issues. The stuff that runs on the server usually Just Works and is an absolute pleasure to code.