
Show HN: I am a high school student and I did research on 3D adversarial attacks - c0deb0t
https://arxiv.org/abs/1908.06062
======
c0deb0t
If you have any questions, I (the first author of the paper) will be more than
happy to answer them. This is my second work in a series on adversarial
attacks/defenses in 3D space (first paper [1]).

A high-level overview of the research:

Basically neural networks are weak against adversarial attacks that change the
input by a little bit to cause the prediction to be wrong. We look at these
adversarial attacks in 3D space, specifically on 3D point clouds (think LiDAR
and RGB-D data). In the paper, four attacks in two different categories
(distributional and shape attacks) are proposed. The main benefit of
distributional attacks is their imperceptibility. On the other hand, shape
attacks are more easily crafted in real-life (though more perceptible) and
also robust against point removal defenses that were proposed in previous
work. If you want a more comprehensive (but less dense than the paper)
overview, take a look at my blog post [2].

[1] [https://arxiv.org/abs/1901.03006](https://arxiv.org/abs/1901.03006)

[2] [https://blog.liudaniel.com/birth-of-a-new-sub-sub-
field](https://blog.liudaniel.com/birth-of-a-new-sub-sub-field)

~~~
dchichkov
The field is in a really sad state. Children pocking holes in the state-of-
the-art. ;) .

Joking aside, fantastic work. The differentiable reformulation in the
distributional attack is a tour de force.

A question. Why did you do it?

~~~
c0deb0t
I don't really think its fair to say that the "field is in a sad state".
Plenty of insightful and well-written papers are put out everyday by
hardworking and intelligent people. I still have a long way to go.

I do research because I like solving hard problems that people have never
considered. I like to ensure that what I have learned will be put into
practical use.

~~~
dchichkov
And for example, solving 1-1 Starcraft problems wouldn't be a problem that
people have never considered. And can't be put to practical use. So you
wouldn't do it? I'm genuinely curious, what makes a difference and motivates
to engage into research, rather than playing challenging games.

~~~
c0deb0t
Well, I used to play a lot of competitive FPS games because I found it fun. I
have also done competitive programming problems for fun/accolades. But after
doing more practical research, I realized it felt better to do impactful stuff
(especially getting recognized). Also, research is nice because I perform
terrible at short events (games, contests) under pressure. I think that if I
tried something else before research that met the same criteria I probably
wouldn't have done research.

------
gchamonlive
Any proposed methods of attack mitigation comes to mind after this research? I
would suspect, and take this with a grain of salt, I haven't been working with
machine learning for quite some time, that smaller subsets of the same data
each representing a different kind of attack mapped to another dimension in
the network output (that would be then recurrently fed to the same network)
could prevent the issue. Kind of teaching the network to defend itself,
recognising patterns of attacks and discarding or compensating them as it
would noise or other kind of interventions.

Are the attacks statistically identifiable? Can they be translated to a
training subset? What would you propose?

~~~
c0deb0t
Feeding examples generated by an attack back into the network is a very
classical defense mechanism. This works ok, but it is not attack-agnostic, and
removing adversarial points works better in 3D. There are also works (mostly
in 2D) on detecting adversarial examples with neural networks.

I am not sure about statistical identification, but we show that it is
difficult to identify and remove adversarial points by looking for statistical
outliers points.

I am not sure about truly robust 3D-specific defenses---if anyone has some
idea, I am open to collaboration. I would imagine some sort of provably robust
method built specifically to handle the varying density and distribution of
points.

------
totoys
This is not at all related to the work you did, but nevertheless I have a
question:

As a fellow high school student who has a good amount of experience and
knowledge in deep learning, how would you recommend me to move forward? I am
struggling to find opportunities to show off my work and knowledge, and would
like some advice.

~~~
c0deb0t
If you want research experiences, then try to email professors directly.
Summarize your experiences and how it aligns with the professors' interests.

Otherwise, try reimplementing algorithms and blogging about them. Do fun
projects like deploying a model online or to phones. If you are a fan of
competitions, then you can try some Kaggle competitions. With some projects
(if you say you have experience then you probably already done some) it should
not be hard to get research experiences because you have something to show
off. Remember to post your projects on Reddit and HackerNews to get internet
points and encouragement! It is quite motivating.

------
schoen
How did you get connected with the university researchers whom you
collaborated with on this work?

~~~
c0deb0t
Cold email plus a bit of luck, and then an informal interview in person. My
best advice for anyone wanting to get connected is to not be shy. To get
opportunities and connections, you have to put yourself out there. This comes
with the risk of criticism.

~~~
Jenz
> My best advice for anyone wanting to get connected is to not be shy.

That’s where I (fellow High School student) struggle. I’m not really at a
point where I can contribute much either way.

~~~
c0deb0t
There is one thing I forgot in my advice: you need to have a good foundation
first. For example, when I cold emailed, I had many programming projects and I
wrote about them in the email. But after you have stuff done, I think
aggressively (but politely) "marketing" your work is important to get those
opportunities.

------
darkmighty
I did not read the paper fully, just some general commentary, hopefully
useful.

First, fantastic that you're doing academic work so early. I believe most
students wait far too long to be exposed to this aspect of academia (and their
lives), which is far more about asking good questions, achieving deep
understanding, and getting good results, than memorizing some procedure that
isn't necessarily useful (but happens often in school). Keep creative and keep
working on creating a good toolset (find math tools you find
interesting/useful and own them!).

Now on to the paper. I'd reiterate the importance of asking good questions
almost above results. For example, the result of adversarial sticks and sinks
looks good -- but is it asking the right question? If you think realistically,
adversarial attacks can occur in a number of ways.

One of them is that a human classifies a dataset one way while a machine
another. In this case you would also want the human not to be able to tell you
data is weird or there's something funky going on -- that is clearly the case
with sticks (and sinks to a lesser extent). A human could be easily trained to
spot them, and generally tell something weird is going on.

Another attack scenario is where you can modify some object, like a picture,
but have some restriction on how much you can modify it. For example, you can
manipulate only some bits of an image, or only perturb a small part of a real-
world object that is under classification (say by putting a sticker on a car
and fooling a system into thinking it is a dog, or something). _If there is no
restriction on your perturbation, this problem would be trivial_ (just replace
the data with intended object data). The justification behind sticks and sinks
does not look very well fundamented.

So sticks/sinks do not fare too well in either case, despite looking very good
in terms of success vs defenses (although there's a chance they could inspire
more practical attacks).

The commentary on Haussdorf distance is relevant here, but only on the first
case (fooling human judgement), and it is of course an imperfect proxy (the
true metric is human perception) -- another hint that fundamentals (and
applications) are important to keep in mind.

Overall the paper seems well written and I specially like the numerous
illustrations.

Keep the good work and don't forget to always look for the inspiring,
beautiful and impactful, and seeking understanding. With a little of this in
mind I have no doubt you can achieve very much. Good luck!

~~~
c0deb0t
Thanks for your comments and encouragements!

The success rates of the attacks are not really emphasized---we only show that
they work. They provide other benefits like robustness against point removal
defenses.

The attacks are optimized for different criterion. The sticks attack is
supposed to be easy to construct, at the cost of perceptibility. The
distributional attack is more geared towards imperceptibility. Indeed, we do
bound the perceptibility of the sticks and sinks attacks, just with different
metrics. You can even argue that the number of sticks we generate is a measure
of perceptibility. Compared to other papers in terms of visual perceptibility,
our attacks are not that crazy. Of course, human perception is the true
metric, and I think more work must be done on quantifying perceptibility in
3D. This paper is the first step, and I mainly wanted to show that there are
factors other than perceptibility that we care about.

------
____smurf____
Super! Keep the great work UP

------
stirner
Off-topic, but this is the first time I’ve seen a colleague’s name show up in
a random academic setting. I worked on a group project with Ronald Yu as an
undergrad at USC. A smart and friendly guy.

~~~
c0deb0t
Yep, Ronald is extremely nice and helpful!

