This is a great response. I'm curious why something like this wasn't posted within 24 hours of RapGenius going public? I'd bet that a more thorough, technical reply would have mitigated a lot of the PR issues.
Agreed, I wish we could have done it much sooner. It took a shocking amount of time to sort through all the entangled issues, emotion, and speculation to try to get to the heart of the matter, which ultimately was about web performance and visibility.
Also, we wanted to respond to our customers first and foremost, and general community discussion second. So we spent close to a month on skype/hangout/phone with hundreds of customers understanding how and at what magnitude this affected their apps.
That was hugely time-consuming, but it gave us the confidence to speak about this in a manner that would be focused on customer needs instead of purely answering community speculation and theoretical discussions.
Thanks for replying. As a paying Heroku customer (who's not affected by the routing issue), while seeing a blog post earlier would have been nice, it's great to hear that you spent so much time with affected customers.
It's pretty much what they have been saying all along; Heroku switched to random routing to support other stacks than only Rails and they kept the routing infrastructure between Cedar and Bamboo the same, resulting in an undocumented performance degradation for large customers still on the old stack or running non-concurrent web frameworks on the new one. The last part was an oversight on Heroku's part, but the rest makes sense, especially considering Cedar supports other stacks than just Rails, including massively concurrent ones(Nodejs, Play, etc.).
The rest of the whole thing was people who were unsatisfied by the change and wanted the old product offering to return and/or felt shortchanged by Heroku. Luckily, Heroku doesn't lock you into their platform like GAE does, for example; it's essentially a bunch of shell scripts to deploy binaries on EC2 workers, a hosted instance of Postgres on EC2, some routing and ops as a service. Anyone who wasn't happy about the change could've contacted Heroku to say "you said we would be sold non-random routing, but you've sold us random routing" or moved their app away to another provider or even their own servers.
Does this suck if you wanted the old service? Sure. Witness the flak that the canning of Google Reader got and it's obvious that discontinuing old services isn't exactly a popular decision. On the other hand, you can self-host (it's massively cheaper than Heroku) or switch to another provider. And if there are no other providers that specialize in Rails hosting, isn't that a business opportunity?
As Adam mentioned in so many words: most of the people pissed off about all of this are most likely right here on HN, and not even customers of Heroku. He mentioned most customers were not that mad about the issue overall.