This is a good thing --an algorithm with up to a 30 error rate is unreliable. One hopes this scrutiny will lead to development of more dependable and trustworthy algorithms. Also hopeful politicians don't have the ability to tweak things in order to fit their agendas.
For example an algorithm is too good at detecting which gov't employee is stealing from the public. Or it tells them they should decrease fines in some area to get better results --but that would affect their budget adversely, etc.
This is an unrealistic law that doesn't matter. It sounds good, but it's futile. If this taskforce takes the decision that an algorithm that is working is unacceptable (which we all know they'll do on the worst possible basis - publicity), then what happens ?
They'll pause the algorithm ? Replace it ? They can't do that ...
The issue is that once you automate something you can't unautomate it without providing the workforce necessary. Because in nearly all cases a bad automated algorithm will far outperform 1% of the required workforce in humans, that's how algorithms win. Not because they beat humans, they don't. They win because they actually take action in 100% of the cases, whatever that number is. A dumb algorithm taking action 100.000 times can easily beat a very hardworking human that takes 100 smart decisions in a lot of cases. So that will be tens to hundreds of people in easy cases, and thousands to tens of thousands in bad cases, and this is New York, easily the size of a decent country. So it'll always be "we need to pause this NOW !", "OK, no problem at all, that'll be $15 million per hour. Under what budget item do we fit this ?" "Erm .... How about you just make it look like it's paused ?"
Keep in mind stopping automation doesn't even just cause damage directly, it will also cause overloading costs onto other departments and even onto private companies. Often in surprising ways. Algorithms respond quickly, under nearly all circumstances, at any time. You wouldn't believe how efficient this makes interacting with organizations. Pausing automation has an enormous and accumulating cost, making the decision impossible.
Also algorithms don't solve corruption (they may make it easier to track though, although there are ways around that).
> which we all know they'll do on the worst possible basis - publicity
That's such a cynical, nihilistic way of looking at government. I can only guess people arrive at it after being exposed to a rather superficial look at governments' work over a long time.
In reality, the vast bureaucracy that is government takes thousands of actions every single day, almost all of which are uncontroversial. They work hard to establish procedures minimising uncertainty. The work is far more transparent than any private organisations'. And all decisions are subject to judicial review–with the judiciary having its own, long tradition of thoughtful deliberation and even-handedness.
As one example, the list at https://www.regulations.gov/searchResults?rpp=25&so=DESC&sb=... shows some recent (federal) actions. Note that this list is only the tip of the iceberg, with the most controversial administration in modern history. Yet it is dominated by "Class E Airspace; Revocations: Eaton Rapids, MI" and other items of rather low publicity value.
Yeah - I've consulted for years for about 4 governments, going from department to department. I still think I'm far too hopeful. The whole thing is utterly corrupt, the politicians running things have utter disdain for actually running things (to the point that they have their security agents shove them aside sometimes - seen that happen). To say that the vast majority of the government - both politicians and bureaucrats - from high to low, everyone, doesn't care about running the place is a complete understatement.
And the government is so full of abuse, it's just disgusting. The thing is even "unintentionally corrupt" as I call it. Regulations that get important things done (especially where it pertains to hiring, consulting, real estate, ...) by just asking it in the right (and "published") location and person. Then, they tell this to 2-3 companies and the rest have to figure it out on their own. Then, of course, they switch to actually corrupt, and change where they need to ask a few months later.
This is patent nonsense. There is absolutely no fundamental rule that requires that there be unaccountable opaque algorithms making consequential decisions about the lives of citizens, and the burden of proof is on you to show there is.
What specific use cases match your description? What is an example of a "case" in your third paragraph above?
One of those paradoxical situations when something _really_ needs to be regulated but can't be regulated. E.g. how do you know what data set is the result of an algorithm (a dubious consolidation step?), or how do you regulate algorithmic learning when we don't fully understand how learning works?
If you look at the specific issue that provoked this law, you will see issues that can - and should - be regulated. the city was using DNA matching software that turns out to have demonstrable errors, yet the office of the medical examiner stonewalled all attempts to have it audited until, thankfully, the courts forced the issue.
This sort of unjustifiable secrecy (the accused absolutely have a right to examine the premises of the accusation) can be regulated. Unfortunately, this law substitutes nebulous criteria which, no matter how worthy, are likely to turn a clear-cut situation into a tar-pit of legal wrangling that the victims cannot afford to enter.
The chief medical examiner is still holding fast on the very dubious claim that these flaws raise no doubts about the convictions in other cases where it was used, another area where I think specific legislation is needed.
> How do you regulate algorithmic learning when we don't fully understand how learning works?
This was not one of those cases. There are, however, cases - and this would be one if it applied - where it is reasonable to say that you can't use it until you can explain how it works.
In this particular context, maybe, but in general - who cares if you can't explain all the values in the neural network, when it is demonstrately safer than a human driver.
You can't say "demonstrably" until you can specify the full breadth of the algorithm's specs. Something being "demonstrable" is something incredibly hard to achieve in certain classes of NNs.
There is no binary choice between "not regulated at all" and "completely regulated".
For example, just implementing a process for public scrutiny of algorithms and datasets may result in those building the algorithms and collecting the data becoming more aware of their biases. (That's "may" with a capital M, though.)
What you mean to say is that politicians currently take decisions on things they aren't even remotely qualified to even talk about, then force it through with heavy hand, and therefore this won't change a thing ?
That's true, and despite that it will change things.
I wonder how an algorithm with low explainability will be treated, like a large nueral network. Seems like it would be hard to pin it on the algorithm rather than the data.
If there's a way to objectively evaluate quality of output, that seems sufficient. It's not like a building inspection where it's a component-based evaluation. The results need to be reasonably and verifiably reliable to an appropriate degree, but the courts don't need to care how that is accomplished.
> If there's a way to objectively evaluate quality of output, that seems sufficient.
But that's the whole problem. If you had something that calculates the the thing algorithm is supposed to do in a better way than the algorithm actually does it, you could just use that as the algorithm.
I find this incredibly off putting since the city cannot even account for its own wasteful budget.
There is nothing wrong with accountability and am glad they are starting somewhere but until they start holding others like the ones who purposesly destroy on time metrics of transit in order to keep thier salary gravy train choo-chooing, I am not holding my breath.
> Signal problems and car equipment failures occur twice as frequently as a decade ago, but hundreds of mechanic positions have been cut because there is not enough money to pay them — even though the average total compensation for subway managers has grown to nearly $300,000 a year.
That's a pretty incredible quote from the article above, since it's an average...
I think it's a travesty that the source code won't be shared with the public, but I hope it would at least be provided for review to the agencies adopting the algorithms.
In my field we are highly regulated and must provide the regulatory agencies with an audit trail for all data. I would like to see an audit trail for these algorithms that would allow someone to follow the decision tree and review the outcome of the algorithms.
This is a good move. I hope other states follow suit. I highly recommend reading Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
For example an algorithm is too good at detecting which gov't employee is stealing from the public. Or it tells them they should decrease fines in some area to get better results --but that would affect their budget adversely, etc.