I agree on most of this. My biggest disappointment is Heroku not implementing a reasonable form of autoscaling to mitigate the worst aspects of this if it occurs. Though I’d still contend that for a site like Rap Genius at the time, 3000ms server response times were still wholly unsatisfactory and they needed to get it under 100ms. Because the likelihood of this occurring compounds significantly in proportion to request time _and_ request volume. They knew what the required solution was.
The original “intelligent” routing was possible because the router effectively had a single shared state to track all these things which was also a single point of failure. And because the state of the art of web servers for Ruby back when it was implemented was Mongrel and it was single threaded.
Add in support for nodejs (and other multi-threaded languages to soon follow) and suddenly each dyno can process multiple request concurrently and the request accounting isn’t as valuable. Make the router more fault tolerant and the overhead in trying to manage that accounting at scale becomes almost impossible without adding significant overhead to every customer request and degrading performance for everyone.
I bristle when people seem to think there was some nefarious revenue incentive from Heroku here. They were well reasoned platform changes to improve availability and increase performance for more efficient languages. In the process the marketing got disconnected from the implementation.
The original “intelligent” routing was possible because the router effectively had a single shared state to track all these things which was also a single point of failure. And because the state of the art of web servers for Ruby back when it was implemented was Mongrel and it was single threaded.
Add in support for nodejs (and other multi-threaded languages to soon follow) and suddenly each dyno can process multiple request concurrently and the request accounting isn’t as valuable. Make the router more fault tolerant and the overhead in trying to manage that accounting at scale becomes almost impossible without adding significant overhead to every customer request and degrading performance for everyone.
I bristle when people seem to think there was some nefarious revenue incentive from Heroku here. They were well reasoned platform changes to improve availability and increase performance for more efficient languages. In the process the marketing got disconnected from the implementation.