Ok, I'll tell 37signals that it's time to shut down Basecamp. You get on the phone with Shopify and break the bad news.
I think the problem here is that they choose a technology that is difficult - but not impossible - to scale, and they didn't have either the in-house know-how to solve their problems neither the money to hire someone with the knowledge to do it.
I think RoR could have been a good decision a few years ago, when the web framework landscape what atrocious, but nowadays there're a lot of good options based on other programming languages that have already proved could be scaled without finding untraceable bugs. Heck, just a few days ago an acquaintance showed me a webapp running over a perl - Amazon, IMDB, Slashdot... use perl heavily - async framework 20% faster than node.js.
"A kernel issue" was not "the problem." There was no "the problem". There were a slew of problems of which that was merely the icing on the nasty problem cake.
And considering how many times I lauded our sys admin as an expert and top talent, not sure why you would think he was "run of the mill." Nope, we hired somebody who had one of the best possible track records in the RoR world. We had a redundant architecture from day 1. My demand was that we have a system ready to work at scale so we wouldn't be growing and 8 or 9 months in, have to migrate servers, complete with customers. Not my cup of tea. Unfortunately, that caution didn't help (and maybe it hurt).
EDIT: As of 4pm eastern I'm off to meet my interior designer and then attend a joinery demo at a wood shop so it may be a bit but I will absolutely read & respond to your questions later or tomorrow. My goal is to share. And maybe mock people a little, if they can't think up an original insult when they attack me.
Thanks for sharing your experiences in such an open manner. It certainly takes much courage and bravery to shutting down your service and even more so, writing honestly about it.
Could I ask you a few questions:
- Is it true to say that if this product required less maintenance (in terms of uptime, server administration, future migration), you would not have abandoned it?
- Secondly, assuming that you are not against raising external funding (which you are not in favour of, as mentioned in your blog post) and hiring the right team for it, would it be fair to say that you would be willing to continue with Charm?
- Lastly, and hypothetically, if Freckle and your other opportunities were not around, would you have continued with Charm? Would it be fair to propose that because of the high maintenance required by Charm upon you and your team, the opportunity cost is simply too high, relative to the other options that you have?
Thank you, and hope to hear from you. :)
Would we not have carefully shut it down if it wasn't for the running-of-it issues… hard to say. Aside from the server problems, I described our staffing / resource problems. We didn't have enough staff to keep building Charm, adding features our users deserved. This is a probelm.
2. Not sure what the funding part has to do with willing to work on Charm. I will never, ever, ever take funding, unless it's to pull a 37signals or a Github — on completely my terms for a tiny sliver of the company — to get a personal advisor or a big "in." But that I doubt (doubt we'd ever get big enough for that to be attractive to an investor). So, assuming I wasn't against funding is too big an assumption to make.
3. Good question. If Charm were our only baby, it would have been much easier to keep going… though frankly I don't think we ever would have attempted Charm as a first product (that bit of sense we lost after the success of Freckle). The opportunity cost, as you say, was huge.
As for the architecture, it was a Rails app with fully-live Backbone/CoffeeScript UI, a tuned Ruby cron job for scraping mail (POP, IMAP), a Postgres database and I forget which full-text search engine because we must have tried all of them before we settled on one. Nothing terribly fancy.
If it's any small consolation, I'll bet you beat yourselves up about the downtime issues way more than your customers ever would.
It was upsetting, though, on an intellectual level. I wanted to do it better.
I don't mind at all answering technical questions, ask away. What I object to is people who spot a jargon term or two they understand and fail to read all the surrounding paragraphs, and then lambast me for something they know nothing about. Which you clearly didn't do :)