
Stealing Machine Learning Models via Prediction APIs - 0x0
http://arxiv.org/abs/1609.02943
======
ChuckMcM
This is interesting but the real money is in extracting HFT models from other
firms by messing with the order book and evaluating the response :-).

~~~
lpage
Thankfully "messing" isn't the word I would choose for most market
participants but you're describing the algorithmic game theory that goes into
designing a good execution algorithm.

------
danvoell
Very interesting. It makes sense that you could learn the model for something
like classification which has a cut and dry answer. Just brute force queries
at the API, log the results, and start working on your own model based on
theirs.

~~~
dharma1
API rate limits? Suppose you could get around that with multiple IPs/accounts.
How many queries do you need to steal a (reasonable size) model?

~~~
yid
Depends on too many factors for even a ballpark. Take Google's Machine Vision
API for instance. The limiting factor here is that the larger your model (and
deep networks are very large models in terms of free parameters), the more
training data you need to make a good approximation. To come close to
"stealing" their entire trained model, my guess is that your API use would
probably multiply Google's annual revenue by a small positive integer.

Alternatively, you could restrict your "stolen" model to a smaller domain and
use fewer, more targeted examples for training. But at this point, you might
as well start blending in predictions from other APIs, perhaps even training
one off the errors of another. This is basically a technique that has been
around for a long time, and in one incarnation is called "boosting" (see
Adaboost).

------
aab0
This is not necessarily that surprising. Hinton's 'dark knowledge' (not cited
in the paper) already showed that a remarkable amount of information is hidden
in the classification probabilities emitted by a model, and that one neural
net can learn a lot from and reverse-engineer another neural net given just
its precise predictions.

------
glup
Isn't this analogous to trying to understand human cognition by using human
responses to inputs, e.g. the endeavor of cognitive science? Just in the case
of inferring the ML architecture, there's a smaller hypothesis space than for
what we think people are doing.

------
jdonaldson
You wouldn't steal the model per se, but you could uses this technique to
generate some nice training data.

Of course, model providers could just as easily have some sort of protection
against this, similar to what's done with "trap streets" on maps.

------
gm-conspiracy
So, is this just brute-forcing API calls to create a training set?

Is this only for supervised learning?

Also, couldn't this be done offline, pseudo-legitimately, using your API call
log data later on? I don't see how that can be mitigated.

------
zellyn
This also seems useful for extracting a model, creating an interpretable
version of parts of it, and proving whether it is prejudiced against certain
races, genders, etc.

~~~
Houshalter
Why would a model be prejudiced against a certain race? Very rarely do people
give race as a feature to statistical models to begin with. And even if they
did, they do not have human prejudices. They train on actual data and care
only about making the most accurate predictions possible.

~~~
sp332
[https://motherboard.vice.com/read/why-an-ai-judged-beauty-
co...](https://motherboard.vice.com/read/why-an-ai-judged-beauty-contest-
picked-nearly-all-white-winners) “It happens to be that color does matter in
machine vision,” Alex Zhavoronkov, chief science officer of Beauty.ai, wrote
me in an email. “and for some population groups the data sets are lacking an
adequate number of samples to be able to train the deep neural networks.”

~~~
Houshalter
Well of course a beauty contest judged by AIs would go horribly wrong.
Appearance is highly subjective and arbitrary.

But even so it's not clear their algorithm was the cause of the bias, or that
the bias was significant. For instance, it's possible that black people have
slightly worse "facial symmetry" on average, or whatever made up metric they
were using. And even if black people only scored 1% worse on average, that
means the extremes will be dominated by whites, because of the way gaussian
distributions work. So it may appear to be way more biased than it actually
is.

~~~
vintermann
Even with no bias, uncertainty can cause problems. Say an ML system is tasked
with finding the top 10 candidates in terms of "confidence that they will be
able to do the job". Then if it has little training data on candidates of a
particular class, and those in that class are actually quite different on many
different variables (so it can't generalize very well), it may not be able to
reach the required levels of confidence for them.

I think this is actually the reason for a lot of accidental discrimination,
because human judges would have exactly the same problem.

I remember in school, playing chess a couple of times against a guy fresh from
Sudan. He had the most unsettling smile, and played very unorthodox openings.
I won some, he won some - but I always suspected he was stronger than me, and
just being polite/testing me. It's just impossible to read someone from a
culture so different. I'm glad we didn't play poker, to put it like that.

------
rememberlenny
How would you go about aggregating an image library to properly gauge the
classification results? Would you just use ImageNet content or try using
something entirely new?

------
omginternets
Is this even an issue? This strikes me as being economically infeasible given
the size of training datasets.

~~~
swordswinger12
"On Google’s platform for example, an extraction attack would cost less than
$0.10, and subvert any further model monetization"

------
macawfish
woah... this is the stuff of science fiction.

when the supreme ai gains the ability to thirst for knowledge, it will steal
all the machine learning models via prediction APIs...

------
gcb0
havent read yet, but am not expecting more than what we had in the 90s of
trying to figure out search engines prioritization algos to use on our
optimization ones.

