
Coding Cloud Heroku May Face Lawsuit Over Bait-and-Switch Claims - wwarnerandrew
http://www.wired.com/wiredenterprise/2013/03/hieroku/
======
redthrowaway
>Heroku declined our request for further comment. “Unfortunately, we can’t
publicly comment on legal matters,” said spokeswoman Dana Oshiro in an email.
“Heroku is committed to our customers’ success and focused on delivering the
best possible product and experience. We’ll continue to update our customers
via our blog.”

I don't run a company, and I'm not responsible for maximizing return on my
shareholders' investment. So maybe I lack the mental schemas that make these
sorts of statements seem reasonable, but they always strike me as short-
sighted. Yes, admitting wrongdoing may open you up to legal liability. But it
seems to me that, in the long run, it's far better to take your licks and own
up to your mistakes than to hide behind the shield of pending litigation. My
gut instinct is to offer a sincere mea culpa and provide a transparent roadmap
to improvement, and if you lose a court case then so be it. I'd like to think
that the good will and respect you build from accountability and honesty will
ultimately outweigh any short-term losses from court settlements or
judgements.

As I said, I don't live in that world. I would be saddened, though not
necessarily surprised, to learn that I was wrong.

~~~
kyrra
Seeing that there is a website/lawfirm[1] threatening legal action, I can bet
Heroku/Salesforce lawyers jumped on that ASAP and let all their employees that
are allowed to talk publicly not to comment on the issue anymore.

The US legal system is setup that if a company admits fault in something like
this, it makes defending yourself much more difficult. Meaning if a lawsuit is
brought about, it's not about proving innocence anymore, it's about mitigating
the damages you will have to pay. Your lawyer is going to want to have as many
options available to him/her to deal with your case during it's lifespan, and
admitting fault before the lawsuit even begins will limit the lawyers options
when it comes to negotiations and what tactics they can use in the courtroom.

[1] <http://herokuclassaction.com/>

~~~
incongruity
In the grand scheme of things, this is why it's always better to defect in the
simplified idea game of the prisoner's dilemma. Collaboration is rarely
rewarded once lawyers get anywhere near an issue and opting to collaborate
(i.e.: admit fault and try to fix it) more or less wins you no points in
court, in the U.S.

Given that each possible liability is its own case and stands on its own
merits, it pretty much matches the simplest case of the prisoner's dilemma
perfectly, IMHO... and so here we are. Sadly.

~~~
OGinparadise
_Collaboration is rarely rewarded once lawyers get anywhere near an issue and
opting to collaborate (i.e.: admit fault and try to fix it) more or less wins
you no points in court, in the U.S._

Might, in part, have to do with how lawyers get compensated. A short case or a
quick settlement will result in very little compensation for them. Over the
course of years and decades that matters so we see everything analyzed to
death. At $500+ an hour. Per lawyer.

~~~
jacques_chester
The careful decomposition of problem domain into solution domain followed by
recursive search of the solution domain is the lawyer's job. If they don't
analyse to death, they are breaching their fiduciary duty.

Don't like the legal method? Don't hire a lawyer. Try doing it yourself. See
how far you get when the other guy's lawyers have done the exhaustive analysis
and found the deciding element buried in a logical node on a distant subgraph
you didn't even know was there to be explored.

~~~
OGinparadise
Yeah, other developed countries are all going downhill for not being
overlawyered up like in US.

It's a system set by lawyers for lawyers so of course you need a lawyer more
than in many other countries. "Everyone" sues over everything

~~~
jacques_chester
The overlawyered US is the richest and most powerful country on Earth by a
country mile.

~~~
OGinparadise
_"The overlawyered US is the richest and most powerful country on Earth by a
country mile."_

Thanks to the lawyers, no doubt. I am sure you didn't meant to say that if we
pass some tort reform and limit some nonsense lawsuits (IP, malpractice etc)
the country will lose it's wealth /power rank.

~~~
jacques_chester
The rule of law is a key part of the mix of institutions, geography, culture
and luck that have made the USA the richest country that has ever been.

And with the rule of law comes lawyers. I know it's fashionable to hate
lawyers, but they're a necessary profession.

Law is a field in which small details can have very large consequences once
the full line of reasoning is unfolded. Because of the importance of
exhaustively covering all lines of argument to the maximal possible depth,
lawyers are _required_ to do so.

They don't do it to bill you more. It is their _legal duty_.

If you don't like that lawyers are required on pain of loss of income, loss of
profession and potentially loss of personal liberty to give you the fullest
and most complete service that they are able to give you, then you are
entitled to represent yourself.

~~~
notahacker
Let's be honest, it isn't this firm's _legal duty_ to set up a marketing
website to try to find a plaintiff to launch a lucrative lawsuit. Law firms
seek to offer the level of service that maximises their returns, just like any
other corporation.

~~~
jacques_chester
No, but once they have a client for a lawsuit, they're still bound by
fiduciary duty.

------
bridgeyman
In MIT's Mathematics for Computer Scientists course (6.042), I remember
learning from Tom Leighton that in certain cases randomly distributing jobs
actually IS the optimal solution to load balancing. This technique was somehow
associated with Akamai, Professor Leighton's company.

Maybe I am misremembering...

~~~
w-ll
I dont know, but I would assume it has to deal with the fact you can't predict
how long a job will take to complete, and a random selection would have more
resources-more free-more often, verses FIFO where a group of really large
tasks could halt everything.

~~~
manmal
And what if they turned the whole thing into a request queue and workers get
the "messages" as they become available? To ensure there are no timeouts, they
could define a max time a request may stay in the queue, and then take it out
and handle it with emergency workers or kill it?

------
eli
IANAL, but my understanding is that _threatening_ a class action lawsuit is
easy, but actually bringing an action to court is extremely difficult. And (at
least to me) this seems like an awfully weak case -- I don't see any obvious
or intentional fraud.

The whole thing seems strange to me. I don't really understand what Rap Genius
hopes to gain here. If Heroku wasn't providing enough performance for their
money (regardless of technical cause), then why did they stick around so long?

Would Rap Genius still have sued if the documentation had been 100% correct
and instead the problem was just plain old slow I/O on Heroku's side?

~~~
tomlemon
> If Heroku wasn't providing enough performance for their money (regardless of
> technical cause), then why did they stick around so long?

We didn't know how bad the performance was because Heroku's tools (logs and
New Relic) reported incorrect performance data (i.e., they said requests
weren't queuing when they were actually spending a ton of time queuing)

EDIT:

> Would Rap Genius still have sued if the documentation had been 100% correct
> and instead the problem was just plain old slow I/O on Heroku's side?

To be clear, we have no official association with
<http://herokuclassaction.com/> or the lawyer behind it – I actually found out
about the site from the article's author when he interviewed me for the story.

(But I do think Heroku owes its customers a refund)

~~~
jaggederest
New Relic isn't Heroku's tool, man. It's like a car dealership selling you a
Pioneer stereo.

~~~
jacalata
It's certainly not a disinterested third party that Heroku knows nothing
about: [http://news.heroku.com/news_releases/heroku-and-new-relic-
pa...](http://news.heroku.com/news_releases/heroku-and-new-relic-partner-to-
provide-ruby-developers-with-performance-management-for-the-cloud)

~~~
jaggederest
Agreed, but it's not the one true monitoring system. I happen to be biased and
think it's pretty good, but it's not a first-party monitoring system that they
swore on a bible was correct.

~~~
elbear
Maybe they didn't swear on the bible, but when you use a tool from a company
with which Heroku has a partnership, you expect that tool to give accurate
readings. I would, at least.

------
lancefisher
From the article, Heroku says, "The problem affected an older version of
Heroku’s service, he said, and those who use a new version of the service
would not have these issues."

As far as I understand that is not really the case. The newer version is the
Cedar stack which supports concurrent apps where Bamboo only supports non-
concurrent apps. This would reduce the problem with random routing, but not
eliminate it.

~~~
randall
The problem is not that they provide random routing, the problem is they said
in their documentation that they provided intelligent routing, a design choice
they backed off of when they switched stacks.

Customers still have a problem, but Heroku doesn't from a legal / class
action-y perspective.

------
DigitalSea
At the end of the day Heroku are in the wrong here. Performance issues which
Heroku basically wouldn't admit until they were outed about them meant that
companies would have invested tonnes of time and effort which in turn equals
money on trying to fix issues that were caused by Heroku the whole time.

~~~
bascule
I'm sorry, did I miss something since aphyr's analysis?
[http://aphyr.com/posts/278-timelike-2-everything-fails-
all-t...](http://aphyr.com/posts/278-timelike-2-everything-fails-all-the-time)

Why is Heroku "in the wrong" exactly?

------
rdl
Heh, nothing will make people go back to taking Heroku's side faster than a
class action lawsuit on the other side. This would be an awesome form of
submarine marketing if they were actually behind it :)

~~~
chubot
Um, why? I was one of many plaintiffs in a class action lawsuit against EA
(which was settled in our favor). That did not help EA's image at all :)

It looks pretty bad when your customers complain about problems that are
hidden by your own monitoring tools, and then you tell them to buy more crap
to fix it! It's not a small amount either -- as mentioned the queuing time was
larger than the request time. And on top of it you're paying out the nose for
this crappy monitoring tool.

~~~
rdl
Even people who have been somewhat screwed by this issue with Heroku tend to
love Heroku; Salesforce could make this right with generous payouts to
affected people and 2-6mo of engineering to offer a better load balancer as a
service (Actually, I think I could build this as a third party provider, even)

EA is basically a lost cause; their business model and culture is the problem.
Having better sacks at Dachau wouldn't have helped the Nazi image, either (to
auto-Godwin)

~~~
chubot
Well that's what they should have done BEFORE the threat of class action
lawsuit. It makes them look at a lot worse if it happens after.

I'm pretty surprised they didn't offer refunds already. It was pretty clear
they were asked for, and with no update by now, it's safe to say they were
refused. That seems unreasonable to me. If you were in the position RapGenius,
would you honestly not have expected a refund already?

~~~
rdl
I wonder what happened to Heroku. They used to be awesome. They still have
elements of being awesome, but seem either resource constrained or self
destructive. I find it hard to believe Salesforce wouldn't give them what they
need to do things correctly -- ranging from "go multi-AWS Region or move off
AWS" to building a decent load balancing system to handling this fiasco.

------
LAMike
On one hand the Rap Genius expose seemed like they got duped and mislead, but
on the other hand something about this lawyer mentioned in the article comes
off as ambulance-chase-y to me.

~~~
rayiner
Class action lawyering usually sound ambulance chasey, because any given
person doesn't lose that much in a given instance and so the actual people
harmed have little incentive to sue.

At the same time, it's an edge case in the economy. If you harm a large number
of people a small amount, they won't sue you to recover for their injury.

------
jacques_chester
I guess I saw this coming:

<http://news.ycombinator.com/item?id=5237920>

<http://news.ycombinator.com/item?id=5241686>

Really, though, given how much money Heroku charge and how much money
Salesforce.com would be able to stump up in a settlement or judgement ... it
was inevitable that a fast-moving law firm would jump on this.

------
citricsquid
If the decisions made by Heroku to misrepresent the product were made prior to
the Salesforce acquisition (December 2010) do the founders of Heroku have any
liability? Would this sort of thing be a consideration with an acquisition?

~~~
chc
I don't believe any decision was made to misrepresent the product. The product
just evolved and the product documentation didn't. The end result is the same
— the product was advertised with inaccurate claims — but this appears to be
more a case of neglect than a deliberate misrepresentation.

~~~
jacques_chester
It's not a criminal matter. _Mens rea_ \-- "guilty conscience", the intention
to act unlawfully -- is a component of crime. But it's not necessary for lots
of civil legal matters.

For example, if we both enter into a contract, you can hold me to my end. If I
fail to hold up my end without intending to fail, then you can still insist
that I make right or seek remedies for my failure.

(IANAL, TINLA).

~~~
chc
That all sounds right to me, but I think you've mistsken my meaning. The point
wasn't that they lack mens rea, but that the decision to misrepresent the
product can't have been made before the acquisition because that decision was
never made at all. It was misrepresented by reality gradually changing out
from under initially truthful statements.

~~~
jacques_chester
You don't have to decide to misrepresent in order to be misrepresentative. If
your material is wrong, it's wrong, and depending on the jurisdiction you may
be on the hook.

Ridiculous? Well consider that it gives a strong incentive to ensure that
claims made to customers are somewhat accurate. Reducing the cost of _caveat
emptor_ makes society richer by reducing transaction costs and so encouraging
trade. Sellers have a strong informational advantage over buyers; in general
the cost to them of making a correct statement is much lower than having each
customer having to find out the truth independently.

Not that this is the _purpose_ of torts law, which evolved mostly to fill
glaring, unjust gaps in the remedies of the older forms of common law. But
that is the social and economic _function_ that it has come to serve.

(IANAL, TINLA).

------
scott_meade
"[The attorney who set up this site] has spoken with one customer, who has
since backed off. “That being said, I am in the process of drafting a
complaint, in case their feet warm up,” he told us."

------
piggity
Only lawyers think it's a good idea to deploy lawyers into the routing mesh.

~~~
raverbashing
But you can bet money will be intelligently routed to their pockets

------
dubcanada
<http://www.wired.com/wiredenterprise/2013/03/hieroku/>

HIEROKU!?

Ignoring the funny url. I doubt this will go anywhere besides in the domain
expiry list.

------
vertis
It really disgusts me how quick people are to bite the hand that feeds them.
Heroku is not perfect, but they give a whole lot more value than they charge
for.

As unlikely as it is, I hope that Heroku terminates the account of anyone that
joins the class action against them.

~~~
Karunamon
>It really disgusts me how quick people are to bite the hand that feeds them.

Really? Because it really disgusts me when a company:

    
    
        * Misrepresents their product (saying it's A when it's actually B)
        * Privately acknowledges this misrepresentation under the radar
        * Makes large and ridiculous amounts of bank on this misrepresentation (since you have to throw more dynos at the app to mitigate the lag caused by the misrouting.. and Heroku may be many things, but economical it isn't)
        * And on that note, provides very expensive monitoring and performance analysis tools that further perpetuate the misrepresentation!
        * And then force you to resort to litigation to get some portion of this huge mis-spent investment refunded. (Looking at the suit, this is all that's being asked for)
    

This wasn't a mistake on Heroku's part, this was calculated. And even if you
accept that it was a mistake, the fact of the matter is that their negligence
in this matter made them a hell of a lot of money.

This is definitely grounds for a suit.

~~~
res0nat0r
By misrepresents you mean that the end users didn't exactly understand how
underlying architecture worked, didn't like the performance, but kept using
the service anyway. If every customer I dealt with while working on the EC2
platform could sue because they didn't understand how every technical detail
of AWS worked and they couldn't optimize for it...well there would be more
lawyers employed by Jeff Bezos than engineers.

This lawsuit will go nowhere and it is sad that such a supposedly tech savvy
crowd wants to bring out the pitchforks so fast. But I guess since Heroku has
a lot of cash they are now the enemy since they've made it and they should be
taken down.

~~~
jacques_chester
The problem is that Heroku made claims about their architecture and subsequent
performance characteristics that were flatly untrue.

The further problem it looks as if they _knew_ the claims were untrue.

These are not matters of internal consideration. They cut directly to the
heart of the Heroku marketing material: "Write for our platform, our expensive
secret sauce will make your stuff linearly scalable."

In which case people expect that it will be linearly scalable. Not
exponentially anti-scalable.

~~~
res0nat0r
The same can be said for EC2. "Amazon EC2 reduces the time required to obtain
and boot new server instances to minutes, allowing you to quickly scale
capacity, both up and down, as your computing requirements change."

If my instances don't launch "quickly" or I've designed my app in such a way
that it isn't 100% linery scalable I can now sue Amazon? You can't sue heroku
for their marketing speak.

~~~
Karunamon
>You can't sue heroku for their marketing speak.

You can sue anyone for anything, success is not assured.

That bit of pedantry aside, Heroku led people on to believe that their
queueing backend was intelligently distributing requests to nodes. Rather than
randomly.

You are not allowed to argue this, as Heroku themselves acknowledged it!

Furthermore, _they blatantly lied about it_. From the rap genius article:

 _Heroku claims that though they received reports of unexplained latency over
the past couple of years, they weren’t able to figure out the request queuing
issue until they read the Rap Genius article. But there is evidence that
Heroku knew about the problem for more than 2 years._

And the conclusion, also from the same article:

Based on Heroku's response so far, it appears their approach to fixing the
problem is, “We promised you intelligent routing, we delivered random routing
(which is worse than intelligent routing), so we're going to change our
documentation to make it clear that we're only promising you random routing.”

This works for future customers, since once Heroku makes these documentation
changes, everyone who signs up will understand exactly how routing works. But
it does nothing to address the time and money that existing customers have
spent over the past few years. What does Heroku owe them?

~~~
res0nat0r
Who's to say random routing doesn't qualify as "intelligent"? What if random
routing is the most efficient routing schema for 99% of their users? Trying to
sue because "intelligent" doesn't mean what you think it means isn't going to
be very successful.

~~~
Karunamon
>Who's to say random routing doesn't qualify as "intelligent"?

 _Heroku themselves!_

Again, they changed their documentation.

~~~
res0nat0r
You mean..they changed their marketing speak. Suing because buzzwords don't
work the way you think they should isn't a very good basis for a lawsuit.

~~~
Karunamon
>You mean..they changed their marketing speak.

Marketing speak is not allowed to be misleading or flatly untrue (as it was in
Heroku's case). This can and has been the basis of successful lawsuits.

~~~
res0nat0r
I don't know how "intelligent routing" can be defined to mean anything. It can
be interpreted a million different ways. This is probably one of the reasons
why they used it in their marketing because it sounds cool and is a generic
term that could be applied to anything.

I don't see much of anything in a lawsuit unless specific metrics related to
SLA's that were agreed to were broken.

~~~
jacques_chester
The documentation used to say that requests were routed to the next idle dyno,
but the behaviour was silently changed to be random distribution, which is
substantially worse for the customer. There seemed to be previous cases where
Heroku were aware of the discrepancy but didn't act to fix it.

That's generally considered uncool.

------
raverbashing
I think the money for this class action lawsuit would be better used if it
would go for a sysadmin/devops person to actually deploy the systems involved.

"Bait and switch"? If a couple of Dynos is not enough for your load, you
should be moving to self-managed hosts

------
jrockway
TLDR: RapGenius (YCS11), in a painful attempt to remain relevant, decides to
sue their web host for running their slow code slowly.

~~~
mikeryan
If you're going to tldr; an article you should really read it. Rap genius
isn't suing. No one, in fact, is actually yet suing.

~~~
jrockway
The first line in the article "Rap Genius founder Tom Lehman says Heroku
misled him and cost his company money."

I have to imagine they're involved based on the number of articles I've read
about them over the last week. Their own article, tech blogs interviewing them
about the article, and now the founder's picture at the top of this article.

~~~
dandelany
Granted, the article is poorly written and seems to imply their involvement.
But it explicitly states, near the end, that the law firm involved has yet to
find any plaintiffs:

 _He’s spoken with one customer, who has since backed off. “That being said, I
am in the process of drafting a complaint, in case their feet warm up,” he
told us._

Furthermore, Tom (the dude pictured) commented in a thread above saying he
didn't even hear about the lawsuit until Wired called him to interview him for
the article. Consider putting away your "Jump to Conclusions" mat and joining
us in the world of facts.

~~~
jrockway
You're right. I wish I could delete my comments.

