
Why Law Firm Rankings Are Useless - wikiwawa
http://blog.litimetrics.com/2016/09/15/quantifying-the-value-of-better-legal-representation/
======
sandworm101
>> "the job of a lawyer is to persuade, which is part legal reasoning and part
hard work. Since the ability to persuade directly affects litigation outcomes,
participants are willing to pay highly regarded barristers upwards of $28,000
a day, and some law firm partners an hourly rate of over $1,000. In this post,
we analyse data from 22,000 cases to measure the differential impact law firms
and barristers have on litigation outcomes."

No. That's what TV lawyers are paid for. That's the equivalent of learning
about doctors from watching MASH. Most doctors are not surgeons. Most lawyers
are not litigators. Many great and valuable lawyers will never win any case.
Imho the best lawyer is the one who's client never faces the uncertainty and
protracted negativity that is modern litigation, just as a great and valuable
doctor is one who's patients avoid the knife. (Apples and oranges but the
point stands.)

Lawyering is, in my world, mostly about listening to clients and understanding
how their realities fit into external standards such as laws and industry
norms. It's about hedging, documentation and interpretations. And sometimes
it's about holding the client's feet to a fire. Litigators only appear once
all of that has failed. Litigators may have the highest hourly rates, but do
not have the highest salaries overall. Those are reserved for corporate
counsels and board members.

~~~
iplaw
Depends on the subject matter and the willingness of a plaintiff and defendant
to negotiate a fair settlement.

In some areas, big money comes with big litigation.

------
rayiner
As they acknowledge, it is essential to account for case difficulty in
comparing success rates. Having been on both sides of the "v." I can say it is
easier on the defense side (at least on the civil side). Defense counsel
representing Big Cos. will have a better record not only because those
companies are often the target of weaker lawsuits, but because various rules
intended to filter out those weaker lawsuits stack the deck against
plaintiffs. Thus, there are fantastic lawyers working at Sierra Club or NRDC,
but they're not going to have the same record of wins as similarly-good
lawyers at a big New York firm.

According to the article, their analysis does account for that. They allegedly
"calculate the probability of a successful outcome for the applicant" using
machine learning. But there is no description of how they do that. If I had an
algorithm that produced halfway decent results in an automated fashion, I
wouldn't use it to get into the legal technology market. I'd set myself up as
a litigation funder and make boatloads of money with accurate valuations of
potential investments.

~~~
andrewfong
If I were interested in litigation funding, I wouldn't evaluate the ability of
law firms to litigate but rather the ability of lawyers to accurate predict
the outcome of cases (regardless of whether they're actually representing the
client) -- e.g. associates at law firms frequently write memos to assess the
probability of success before a case makes it way to trial. Seems like there's
value in actually correlating memos with actual outcomes.

~~~
rayiner
The article says they have a machine-learning algorithm that can "calculate
the probability of a successful outcome" of a case. Then they calculate firms'
ability to improve (or not) that outcome. But if I had such an algorithm I'd
just use it directly to decide what cases to fund.

My understanding from talking to litigation funders is that lawyers are
terrible at valuing cases. There is money to be made for anyone that can use
technology to improve those predictions.

------
pitt1980
"To sidestep the randomness of observed outcomes, we focus on modeling the
probability of a successful outcome for a given case. Secondly, to account for
the context or difficulty of the case, we measure the change in the
probability of a successful litigation outcome, from substituting a legal
service provider in. The difficulty of the case is not important, but rather
the impact a legal service provider has on that original probability of
success. From now on, we will refer to this measure as the impact on expected
performance. This substitution exercise is known as a counterfactual, as we
apply the substitutions to historical data."

\-----------------

how exactly did they model the probability of successful outcomes for given
cases?

because those depend on highly detailed fact patterns, many of which are
largely qualitative in nature

if they were actually able to put all those in a data base that you could then
model

that'd be a more interesting accomplishment to read about

~~~
Digory
Right. The post links to slideshow "model" for counterfactuals given at Hulu.
And I won't pretend to analyze it scientifically, but it's hard to shake the
feeling that it's circular in this application.

We want to rank lawyers based on effect on outcome, not win-loss record. But
it looks like the model is one that derives "counterfactual" outcomes by
making comparisons based on win-loss records.

Can someone give a better lay-level discussion of the expected outcome model?

------
spitfire
Why not use a modified Elo rating? That way you can not only easily figure out
if you're going to win, but simulate the outcome of different scenarios?

~~~
robrenaud
What is a modified elo rating?

~~~
lordnacho
Elo rating is a system used to gauge chess players. You win/lose point
depending on your points difference before a match. If the favourite loses,
more points are lost to the underdog than if the underdog loses.

You can apply this to a wide variety of sports where most teams don't get to
play most other teams, eg you can guess that Germany are better than India at
soccer despite the rarity of the tie.

And of course you can modify the system to take account of various issues such
as the retirement of better players, which tends to remove points from the
system.

------
charlesdm
As someone who works with a tax law firm that is ranked pretty high on the
legal500 (and obviously, I know a rank can be gamed relatively easily), I must
say the experiences I've had with them have been excellent. For all intents
and purposes, they are exactly what I would expect from a good law firm.

I worked with several "independent" lawyers before, and honestly, they are
nowhere near close to the level of skill of my current law firm. In addition,
they're quite responsive, much more than the average independent lawyer. And
yes, they're more expensive, but not outrageously so. They seem to reach a
favourable outcome faster, and seem to be more effective.

Tl;dr: Love the lawyer I work with in this firm, he gets the job done when I
need him to + has a great team with broad experience to back him up. He's
managed to resolve any tax issue before we even went to court.

------
soufron
I worked on applying machine learning algorithms to data mined from court
judgments during my PhD. You want to know what we learned? That it's better to
be white and rich than black and poor. Point. Everything else was bullshit.

~~~
steveeq1
What about black and rich vs white and poor?

~~~
sjdchid
Sample size too small I'd guess.

------
soufron
Also, being a Lawyer, if someone comes to me, it's because of me, not because
of the firm I am working in. And trying to rank lawyers? Why not. But what's
the point. I wouldn't accept being ranked anyway.

~~~
hackuser
> being a Lawyer, if someone comes to me, it's because of me, not because of
> the firm I am working in

Are you speaking hypothetically, i.e., 'if I was a lawyer'? I'm not an
attorney but for a variety of reasons I have relatively deep insight into the
legal industry, and I think this statement is generally not true.

As in any field, most business comes from referrals and networking, and much
of that happens within a firm. If your career is at a top firm then you are
going to build a much different network than a similar attorney who opens
their own office in a strip mall. When Goldman Sachs needs an outside
attorney, where do they go and who do they ask? Even ignoring referrals, just
imagine who is there when you are invited to dinner by your colleague: Is it
the mayor and local Fortune 500 CEO, one of whom takes a liking to you and
asks you to join the board of a local charity, or is it the lovely neighbors
who drive Uber or run a small medical practice, and ask for some legal tips?

Personally, I don't enjoy the hunt for status and I think it impedes socio-
economic mobility, but I'm afraid that's the way it works.

~~~
charlesdm
I have a tax lawyer I work with, and I couldn't care less which firm he works
at. However, he does work at a good firm, and a good name helps in
negotiations (in this case, with the tax authority). A while back, I had a tax
audit, and surprisingly the person doing the audit told me my firm can turn
"black" into "white". That says quite a bit about the name and reputation of
the firm.

------
libertymcateer
> By applying machine learning algorithms to data mined from court
> judgments...

And, with that sentence, this post becomes bullshit.

The overwhelming majority of disputes do not reach litigation. The
overwhelming majority of litigation does not reach judgment. And then a
gigantic proportion of those judgments are appealed, and a non-trivial
percentage are overturned.

Even at that, only on the order of a single digit-percentage of disputes ever
reaches a verdict.

This is like judging a hospital-system based on its success-rate for
transplant surgery. It's just deeply ignorant of what constitutes the vast,
vast majority of the practice of law - _even_ in disputes.

This article has no value, whatsoever, aside from a witty aside at a cocktail
party about how deeply misunderstood the legal profession is. This is
particularly shocking given that litimetrics is a legal service provider.

> Although A has not faced B, if both have faced C, then we have some
> information about the hypothetical A and B matchup. This is the same
> reasoning that allows one to compare tennis players across different eras.

The implication is that the lawyer efficacy, based on _judgments_ , is subject
to the transitive property? Sweet mercy that is ridiculous.

This post is a catastrophe.

~~~
wikiwawa
A large part of statistics is devoted to making valid inference from noisy
samples, largely by making sensible assumptions. Even in your example, it is
common practice to assess a surgeon on outcomes of surgery, even though a lot
of a surgeon's work does not lead to surgery.

We aren't claiming to assess the wide practice of law, just litigation. We do
Litigation Analytics.

~~~
umanwizard
do you understand the difference between "noisy" and "biased"?

~~~
wikiwawa
I understand sarcasm, that's for sure.

As a matter develops, parties tend to drop out if they think that their
chances have declined beyond some threshold. But if they continue, or more
likely, their legal representation suggests they continue, it implies that
they think they have a good chance of success.

Of course, you can get irrational litigants but most of the time, if it goes
to judgement, both parties think they have a high chance of success. Otherwise
they would have bailed.

