
Help EFF Track the Progress of AI and Machine Learning - sinak
https://www.eff.org/deeplinks/2017/06/help-eff-track-progress-ai-and-machine-learning
======
comboy
The project is very interesting, but I'm not sure why they are doing it. How
is that protecting user rights? This doesn't measure AI/ML progress that's
available to state actors.

~~~
mlinksva
> EFF’s interest in AI progress is primarily from a policy perspective. We
> want to know what types of AI we need to start engaging with on legal,
> political, and technical safety fronts. Beyond that, we’re also just excited
> to see how many things computers are learning to do over time.

> Given that machine learning tools and AI techniques are increasingly part of
> our everyday lives, it is critical that journalists, policy makers, and
> technology users understand the state of the field. When improperly designed
> or deployed, machine learning methods can violate privacy, threaten safety,
> and perpetuate inequality and injustice. Stakeholders must be able to
> anticipate such risks and policy questions before they arise, rather than
> playing catch-up with the technology. To this end, it’s part of the
> responsibility of researchers, engineers, and developers in the field to
> help make information about their life-changing research widely available
> and understandable. We hope you’ll join us.

~~~
comboy
Yes, it's not that I didn't read the page. But even in case of corporations,
it's not really possible to gauge what t's available to them, because they
don't share everything.

Well on the other hand whatever is available publicly, they have at least
that. So maybe the whole research does make some sense in this context.

~~~
mlinksva
Perhaps forcing governments and corporations to share more will be explored if
it seems that this publicly documented baseline is far from the state of the
art. Impossible to gauge, perhaps, but even realizing that highlights the risk
of public ignorance.

~~~
ethbro
A question mark is more valuable than simply reporting nothing.

And I think as the Samsung / Intel / semiconductor fab article (about birth
defects and cancer caused by chemical exposure) the other day noted,
traditional lawmaking is ill-equiped to deal with fast-moving technical
fields.

The first step is public awareness.

------
startupdiscuss
Part of what we are learning, in AI/ML, is the number of problems where the ML
is relevant and where it is not. (Or, to put it another way, where we can find
statistical relevance between features and targets).

So I think you need some kind of "meta-metric" that measures the growth of the
taxonomy itself. And perhaps some kind of weighting for the impact of the
solution.

There is also an interaction effect (for instance, Natural Language Processing
is powerful, and "common sense reasoning" is powerful, but put them together
and you have a knockout), but I don't know how to go about measuring that.

~~~
daveguy
I think you'd have to measure "common sense reasoning" first. With NLP you can
have accuracy/precision scores. With common sense it's almost equivalent to
"what you need to beat the turning test" and not easily measured. I think if
you get that part you essentially have the NLP. In that those last few percent
error of NLP are generally "common sense" issues.

------
samstave
Who is, or is seeking to be, the canonical source of truth for AI, ML etc
policy/position/etc and why should/how do we trust them?

In twenty years, what body will be directing the policies and laws regulations
etc WRT to how humanity deals with essentially what is another "sentient"
species?

Edit.. just read this and apparently this is exactly what the EFF is
attempting to do...

But the question still remains: how do we trust these policies, how do we
request/reject them?

I don't want to deal with this the same way the legal system is currently set
up, lawyers and the law is flawed in many respects and I don't think it's a
good idea to map the old to the new and uncharted directly.

(Apologies for the clunky language/terms.. please educate me on how to speak
of this if you know)

~~~
pde3
You're asking great questions :)

EFF isn't necessarily a fan of the way that current institutions of governance
or the law operate, but when those institutions attempt to interfere with the
development of technology, we step in to try to mitigate the damage and make
the case for sensible outcomes.

In the case of general-purpose human level AI, which to be clear is an
extremely speculative kind of technology that might not happen in our
lifetimes, I don't think anybody knows how humanity would deal with it. If it
does happen, I think the biggest responsibility of participants in that
process would be to minimize the risk of instability and conflict while humans
and the new species (possibly species, plural; possibly not a species at all),
figured out how to relate to each other.

How best to accomplish that is largely a very difficult and mostly unanswered
research question, though you can find some pointers to some interesting early
work in the safety section of the Notebook.

~~~
samstave
If we cant determine when/if we will get on parity with humans in AI
capability... we should be focused on AI as a tool that will be used by those
with the know-how-resources against those who do not have ability/access;

Rich vs poor:

* HFT will never be a common man's tool

* Government surveillance

* Corporate surveillance

* Behaviorally informed/adjusted pricing etc...

Basically ML and AI will be used, in large part, by those who can to exploit
those who cannot (or at least those who wont) defend themselves.

There will be no opting out.

So, with that said, if we go from the bottom up - and there is no opting out,
then what is the best that one can hope for? I'd say complete ownership and
control of one's own "meta-cloud" \-- Any data that is a resultant trail of
any action I take as an individual should be owned by me, and I should be able
to see it all, and delete or block it.

Or, in the extreme case, shouldn't I be able to require that any information
presented to me (such as an ad or a price) be required to inform me as to how
that information was formulated:

"You are seeing this price because the following factors were analysed..."

The real question is, in the future, is there even such a concept as "off
grid"

------
arikr
For ease of reading/UI, I think they should use a different color for the bars
that represent "human score" and those that represent "excellent performance"
\- so that someone doesn't skim and assume they are the same thing, given that
currently both are represented as red dotted lines but mean two different
things.

------
DanielBMarkham
I believe the time has come for an independent institute to track AI and Big
Data technology applications and begin creating guidelines for both industry
self-certification and regulation. The other paths available, ignoring it,
fearmongering about it, and trying to fit it into other political movements,
do not seem to me to be heading towards an acceptable outcome.

It has to be a technology-heavy group, otherwise it won't create much value.
It also has to be grounded in history, philosophy, and political science,
otherwise it'll just be reactionary. And we have enough reactionary groups
already.

------
tectonic
Nice use of a shared Jupyter Notebook for data gathering.
[https://www.eff.org/ai/metrics](https://www.eff.org/ai/metrics)

~~~
shmageggy
Nice that it's open and shared, but kinda weird that they are storing what is
essentially tabular data in code. Looks to me like it would be much more
readable if the data were in csv files, and just the graph generating code
were in, well, code.

~~~
pde3
The measurements for each metric are tabular, but the taxonomy of problems
above them is currently tree (or forest) structured.

CSV export or import for specific metrics would be very easy to add if you'd
like it. We already have a rough JSON export of the data:

[https://raw.githubusercontent.com/AI-metrics/AI-
metrics/mast...](https://raw.githubusercontent.com/AI-metrics/AI-
metrics/master/export-api/v01/progress.json)

------
DiabloD3
Dupe:
[https://news.ycombinator.com/item?id=14603250](https://news.ycombinator.com/item?id=14603250)

