
Human in A.I. Loop - sndean
https://sanealytics.com/2016/09/09/human-in-a-i-loop/
======
noonespecial
"What we need to build is a gamified quick truther thingy that all our 200
employees in main office can log in during their lunch break and rate a few
items."

Lunch break? Or you could just, I don't know... go with me here, I know it
sounds crazy at first..., _pay_ them for providing this value to your company.

~~~
nxzero
Or, strange concept, the employees benefit from their efforts to improve the
company, and idea of working during lunch on a trival task makes sense if they
don't have anything else worth doing.

~~~
y4mi
its no longer a break if you're still working.

~~~
MaulingMonkey
And the break is worth doing. Do you want your highly paid managers,
engineers, salespeople, etc. properly recuperating? To better tackle the
nontrivial tasks you pay the big bucks to tackle?

Or do you want really really expensive mechanical turks?

------
nxzero
"In the loop" is a reference to the OODA loop, which is the decision cycle of
observe, orient, decide, and act, developed by military strategist and United
States Air Force Colonel John Boyd. "In the loop" has a specific tactical
meaning, which is that you're able to do a full OODA loop cycle faster than a
comparable AI/man system which gives a strategic advantage and maybe used
against oppositional forces in a variety of ways; most interesting is applying
it to PsychOps.

"Man in the loop" reference can be found in use by the US AirForce going back
to the 80s for weapon systems design; specifically between fighter jet systems
and pilots. I've heard that US AirForce used brain implants as early as the
90s in fighter pilots.

Here's an example of the US Army reads soldier's brain waves to speed up image
analysis:

[http://newatlas.com/us-army-eeg-brainwave-image-
analysis/403...](http://newatlas.com/us-army-eeg-brainwave-image-
analysis/40309/)

~~~
mmastrac
I'm pretty sure it's just a reference to the phrase "in the loop", which dates
back beyond the 80s:

[http://idioms.thefreedictionary.com/in+the+loop](http://idioms.thefreedictionary.com/in+the+loop)

~~~
nxzero
Yes, "in the loop" does date back further, but as used to reference
human/systems integration earliest reference I was able to find was the 80s
which was the same time that Boydian like systems started popping up.

------
datstanfordphd
Looks like what CrowdFlower calls "human in the loop computing" which is a
nicer name for it.

[http://www.computerworld.com/article/3004013/robotics/why-
hu...](http://www.computerworld.com/article/3004013/robotics/why-human-in-the-
loop-computing-is-the-future-of-machine-learning.html)

~~~
imh
Also called human computation. The past paper awards from this conference [0]
gives a good flavor for the field.

[0]
[http://www.humancomputation.com/2016/past%20meetings.html#aw...](http://www.humancomputation.com/2016/past%20meetings.html#awards)

------
amelius
It seems that the name of the concept this guy is looking for is: Mechanical
Turk service.

~~~
rawnlq
The concept is actually more commonly called "human computation" (which
includes stuff like digitizing books using CAPTCHAs).

It has academic journals/conferences under that name:
[http://hcjournal.org/ojs/index.php?journal=jhc](http://hcjournal.org/ojs/index.php?journal=jhc)

------
ayw
Applications of human-in-the-loop systems like this are exactly why we built
Scale API (www.scaleapi.com, YC S16).

Unfortunately, Mechanical Turk is often a bad choice these days for these
types of applications—it's extremely difficult to guarantee quality, not to
mention hard to use. Using employees also works well to a point (we did some
of this at companies I've worked at too), but it doesn't scale, and should be
done sparingly.

Great application of humans in the loop—this is the type of thing that really
excites me. We're trending towards more and more human-augmented software, and
it opens up so many possibilities for what software can do!

------
Eridrus
Does anyone know any good tools to make that sort of page quickly in Python? I
mean, I could cobble together a web app, but I feel like dataset annotation is
so common that there should be something out there already.

~~~
nxzero
Why Python? Why not just focus on finding the best open source data annotation
framework?

(Love Python, but normally don't bias my selection of a solution by the
language it's written in.)

~~~
Eridrus
You're right, though I do need a way to interact with it programatically from
python, since that's what I prefer to write my scraping & munging code in.

It would probably be fine if it was a solution that had an API that was easily
callable from Python without a silly amount of overhead, specifically being
able to dump data into a named dataset is what I want.

Python would also be nice since it would be easier for me to customize; though
I could deal with Java or Node. I've avoided Ruby for this many years and
would prefer to keep it that way.

------
blueeyes44
we wanted flying cars - instead we got tagged couture.

