
New York City Moves to Create Accountability for Algorithms - dthal
https://www.propublica.org/article/new-york-city-moves-to-create-accountability-for-algorithms
======
mc32
This is a good thing --an algorithm with up to a 30 error rate is unreliable.
One hopes this scrutiny will lead to development of more dependable and
trustworthy algorithms. Also hopeful politicians don't have the ability to
tweak things in order to fit their agendas.

For example an algorithm is too good at detecting which gov't employee is
stealing from the public. Or it tells them they should decrease fines in some
area to get better results --but that would affect their budget adversely,
etc.

~~~
candiodari
This is an unrealistic law that doesn't matter. It sounds good, but it's
futile. If this taskforce takes the decision that an algorithm that is working
is unacceptable (which we all know they'll do on the worst possible basis -
publicity), then what happens ?

They'll pause the algorithm ? Replace it ? They can't do that ...

The issue is that once you automate something you can't unautomate it without
providing the workforce necessary. Because in nearly all cases a bad automated
algorithm will far outperform 1% of the required workforce in humans, that's
how algorithms win. Not because they beat humans, they don't. They win because
they actually take action in 100% of the cases, whatever that number is. A
dumb algorithm taking action 100.000 times can easily beat a very hardworking
human that takes 100 smart decisions in a lot of cases. So that will be tens
to hundreds of people in easy cases, and thousands to tens of thousands in bad
cases, and this is New York, easily the size of a decent country. So it'll
always be "we need to pause this NOW !", "OK, no problem at all, that'll be
$15 million per hour. Under what budget item do we fit this ?" "Erm .... How
about you just make it look like it's paused ?"

Keep in mind stopping automation doesn't even just cause damage directly, it
will also cause overloading costs onto other departments and even onto private
companies. Often in surprising ways. Algorithms respond quickly, under nearly
all circumstances, at any time. You wouldn't believe how efficient this makes
interacting with organizations. Pausing automation has an enormous and
accumulating cost, making the decision impossible.

Also algorithms don't solve corruption (they may make it easier to track
though, although there are ways around that).

~~~
matt4077
> which we all know they'll do on the worst possible basis - publicity

That's such a cynical, nihilistic way of looking at government. I can only
guess people arrive at it after being exposed to a rather superficial look at
governments' work over a long time.

In reality, the vast bureaucracy that is government takes thousands of actions
every single day, almost all of which are uncontroversial. They work hard to
establish procedures minimising uncertainty. The work is far more transparent
than any private organisations'. And all decisions are subject to judicial
review–with the judiciary having its own, long tradition of thoughtful
deliberation and even-handedness.

As one example, the list at
[https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=...](https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=postedDate&po=0&np=90&dct=N%2BFR%2BPR)
shows some recent (federal) actions. Note that this list is only the tip of
the iceberg, with the most controversial administration in modern history. Yet
it is dominated by "Class E Airspace; Revocations: Eaton Rapids, MI" and other
items of rather low publicity value.

~~~
candiodari
Yeah - I've consulted for years for about 4 governments, going from department
to department. I still think I'm far too hopeful. The whole thing is utterly
corrupt, the politicians running things have utter disdain for actually
running things (to the point that they have their security agents shove them
aside sometimes - seen that happen). To say that the vast majority of the
government - both politicians and bureaucrats - from high to low, everyone,
doesn't care about running the place is a complete understatement.

And the government is so full of abuse, it's just disgusting. The thing is
even "unintentionally corrupt" as I call it. Regulations that get important
things done (especially where it pertains to hiring, consulting, real estate,
...) by just asking it in the right (and "published") location and person.
Then, they tell this to 2-3 companies and the rest have to figure it out on
their own. Then, of course, they switch to actually corrupt, and change where
they need to ask a few months later.

------
patkai
One of those paradoxical situations when something _really_ needs to be
regulated but can't be regulated. E.g. how do you know what data set is the
result of an algorithm (a dubious consolidation step?), or how do you regulate
algorithmic learning when we don't fully understand how learning works?

~~~
mannykannot
If you look at the specific issue that provoked this law, you will see issues
that can - and should - be regulated. the city was using DNA matching software
that turns out to have demonstrable errors, yet the office of the medical
examiner stonewalled all attempts to have it audited until, thankfully, the
courts forced the issue.

This sort of unjustifiable secrecy (the accused absolutely have a right to
examine the premises of the accusation) can be regulated. Unfortunately, this
law substitutes nebulous criteria which, no matter how worthy, are likely to
turn a clear-cut situation into a tar-pit of legal wrangling that the victims
cannot afford to enter.

The chief medical examiner is still holding fast on the very dubious claim
that these flaws raise no doubts about the convictions in other cases where it
was used, another area where I think specific legislation is needed.

> How do you regulate algorithmic learning when we don't fully understand how
> learning works?

This was not one of those cases. There are, however, cases - and this would be
one if it applied - where it is reasonable to say that you can't use it until
you can explain how it works.

~~~
tomjen3
In this particular context, maybe, but in general - who cares if you can't
explain all the values in the neural network, when it is demonstrately safer
than a human driver.

~~~
Daishiman
You can't say "demonstrably" until you can specify the full breadth of the
algorithm's specs. Something being "demonstrable" is something incredibly hard
to achieve in certain classes of NNs.

If it breaks on edge cases, that's important.

------
skate22
I wonder how an algorithm with low explainability will be treated, like a
large nueral network. Seems like it would be hard to pin it on the algorithm
rather than the data.

~~~
vnchr
If there's a way to objectively evaluate quality of output, that seems
sufficient. It's not like a building inspection where it's a component-based
evaluation. The results need to be reasonably and verifiably reliable to an
appropriate degree, but the courts don't need to care how that is
accomplished.

~~~
AnthonyMouse
> If there's a way to objectively evaluate quality of output, that seems
> sufficient.

But that's the whole problem. If you had something that calculates the the
thing algorithm is supposed to do in a better way than the algorithm actually
does it, you could just use that as the algorithm.

------
bob_theslob646
I find this incredibly off putting since the city cannot even account for its
own wasteful budget.

There is nothing wrong with accountability and am glad they are starting
somewhere but until they start holding others like the ones who purposesly
destroy on time metrics of transit in order to keep thier salary gravy train
choo-chooing, I am not holding my breath.

[https://mobile.nytimes.com/2017/11/18/nyregion/new-york-
subw...](https://mobile.nytimes.com/2017/11/18/nyregion/new-york-subway-
system-failure-delays.html?referer=https://www.google.com/)

~~~
sologoub
> Signal problems and car equipment failures occur twice as frequently as a
> decade ago, but hundreds of mechanic positions have been cut because there
> is not enough money to pay them — even though the average total compensation
> for subway managers has grown to nearly $300,000 a year.

That's a pretty incredible quote from the article above, since it's an
average...

------
SauciestGNU
I think it's a travesty that the source code won't be shared with the public,
but I hope it would at least be provided for review to the agencies adopting
the algorithms.

In my field we are highly regulated and must provide the regulatory agencies
with an audit trail for all data. I would like to see an audit trail for these
algorithms that would allow someone to follow the decision tree and review the
outcome of the algorithms.

------
lelandgaunt
This is a good move. I hope other states follow suit. I highly recommend
reading Weapons of Math Destruction: How Big Data Increases Inequality and
Threatens Democracy

~~~
an_cap
I recommend reading this review before deciding to read the book.
[https://thezvi.wordpress.com/2017/06/04/book-review-
weapons-...](https://thezvi.wordpress.com/2017/06/04/book-review-weapons-of-
math-destruction/)

~~~
afsina
This was a good read, thanks.

