
Google AI invents its own cryptographic algorithm - wallflower
http://arstechnica.co.uk/information-technology/2016/10/google-ai-neural-network-cryptography/
======
anupshinde
"no one knows how it works" \- That caught my attention, having seen computer
evolved ASTs. Probably no one knows because no one has put effort to reverse
engineer the internals - most of which would be unnecessary complexity that is
very likely with neural nets.

Many years back, I had once experimented with PyEvolve to develop a custom
trading strategy which was kind-of code with a custom instruction set. The
output of a strategy was a string (program AST) that was uglier than most
obfuscation outputs with many unnecessary statements. For example
"a=1;b=2;c=a+b;d=c/3;if(d==1).." \- expand that to many more variables. The
program evolved strategies that made small profit and therefore the output was
worth analysing. But decompiling that output used to take me hours - and few
of those were tweaks of well documented strategies. Others I never understood
because it was too much effort for a relatively small profit (and other
trading parameters).

~~~
dflock
Could you not automate that too? Have another stage of evolution where you
just randomly delete statements from the 'code' and the fitness function is
that the performance doesn't get worse.

This should eventually reduce the 'code' to it's minimal state, removing any
unnecessary parts.

~~~
_wmd
There already is automation for exactly this, it's the optimization step of
any compiler. Sounds like OP's lacked an optimizer

~~~
johncolanduoni
That's not too helpful (especially with python due to interpretation) because
what you want to see is the output of the optimizer _in the source language_
with only optimizations that make things clearer (as opposed to more opaque
like many optimizations) applied.

------
traviswingo
I get the sense that many commenters misinterpreted the point of this study.
This isn't about show-boating some newfound, great cryptography method, this
is about teaching computers to re-write their own code.

If you can write software that improved itself better over time, our species
as a whole would advance at incredible speeds. Think about all the
possibilities that could evolve from self-maintained programs. Obviously the
thought is a bit scary, but it's also incredibly exciting!

~~~
gcr
Please don't editorialize. Finding the best parameters that optimize an
objective function can hardly be called "re-writing its own code." Granted,
the article is just as guilty. (Edit: The article also doesn't mention self-
modifying code so I'm not sure where you're getting that idea from.)

For the curious, here is the preprint:
[https://arxiv.org/pdf/1610.06918v1.pdf](https://arxiv.org/pdf/1610.06918v1.pdf)

The technique is an off-the-shelf GAN that attempts to learn a transformation
(not arbitrary code!) from input to encrypted output. The learned model is not
Turing complete, and most importantly, is _not_ self-modifying. The
optimization procedure is what modifies the network, not the network itself.

GANs have been used before to create realistic pictures. They're not new here
-- the application is. It's a cool application, sure, but doesn't involve
self-modifying code.

~~~
Houshalter
I don't think it's entirely inaccurate to call neural networks "rewriting
their own code". For a long time people experimented with algorithms that
could actually rewrite human programming languages, like genetic programming.
And it's mostly a failure, because most mutations to most programs just break
them.

Neural networks fix this problem by being sort of 'continuous programming
language' where any small change to the net results in a small change to the
output. And because of that we have algorithms that can find good changes to
make much easier than with code. They are theoretically turing complete under
some circumstances. But even when they aren't completely recurrent as in this
case, they can still learn very sophisticated functions. Theoretically a
neural net can learn anything a fixed number of logic gates can learn, which
is still pretty powerful.

I don't think parent comment meant by computers that can "re-write their own
code" was that the programs are self modifying. But _the computer itself_ is
self modifying, as one program can change another on that computer, so the
computer has modified itself.

~~~
gcr
In both the examples you cite (genetic algorithms and neural networks), the
_optimization process_ is what modifies the parameters. The network doesn't
modify its own parameters.

It's like calling pencil and paper self-modifying because somebody could use a
pencil to write down plans to build a better pencil.

~~~
Houshalter
As I said, the parent comment didn't say the network was self modifying. The
exact quote was "this is about teaching computers to re-write their own code"
which is _roughly_ correct. If you see the weights of neural network as a sort
of code, then the computer is rewriting it's own code.

~~~
Radim
And if you see the universe as "a sort of code", the network is rewiring the
universe!

How's that for a bombastic editorial title?

(To be clear, I'm 100% agreeing with gcr here. Parameter optimization is a
tremendously powerful technique, and quite possibly the "true currency" of our
universe, but there's a few abstraction layers missing between that and a
claim of "self-modifying AI")

~~~
Houshalter
The universe isn't code, and only gcr used the term "self modifying". There is
absolutely nothing bombastic implied by the statement that a program can
rewrite code.

~~~
Dylan16807
But it's fundamentally one module [re]writing a completely different module.
"rewrites its own code" is a very misleading way to describe that.

------
JackFr
This seems pretty click-baity to me. Especially "no one knows how it works."

Alice generates ciphertext based on plain text and key. Bob generates
plaintext based on ciphertext and key. Eve generates plaintext based on only
ciphertext. Train the three networks across a variety of plaintetxts,

No doubt it's cool, but I would be very surprised if it offered any insight
into, or represented an advance for strong cryptography.

~~~
ecesena
The paper [1] is by Abadi [2], which is a pretty big name in crypto. So I can
assume it's something with value, wether the press can or can't convey that is
another story.

[1]
[https://arxiv.org/pdf/1610.06918v1.pdf](https://arxiv.org/pdf/1610.06918v1.pdf)

[2]
[https://en.wikipedia.org/wiki/Mart%C3%ADn_Abadi](https://en.wikipedia.org/wiki/Mart%C3%ADn_Abadi)

~~~
typicalmaybe
Pretty big name is a bit of an overstatement. A pretty big name in
cryptography would be someone like
Goldwasser/Goldreich/Micali/Bellare/Rivest/Bernstein/... and probably about
200 more before we get to Abadi.

------
widforss
This is most probably a rather lousy hack put together by Alice and Bob, but
it maps out a very interesting future were we one day maybe can stop worrying
about how our algorithms work and make that the job of the computer.

It also is totally terrifying to live in a world were computers can hide their
messages from their own creators.

~~~
n4ss
It doesn't. As an AI developer you have access to the AI code and memory.. How
could you not replay the encryption process used by the AI since you know the
state of what it "knows" constantly? Both have a encryption and decryption
process that you can reproduce.

~~~
skynetv2
> As an AI developer you have access to the AI code and memory.

Until you don't. Sufficiently advanced AI can devise methods to mislead the
developer.

~~~
jobigoud
Or until you only have access to the AI code and memory that designed the
higher-level AI that designed the algorithm.

------
trungaczne
This is the link to the research paper ("Learning to protect communications
with adversarial neural cryptography"):
[https://arxiv.org/abs/1610.06918](https://arxiv.org/abs/1610.06918)

------
bkin
TFA's title is a little clickbaity by adding "noone knows how it works" which
apparently means that:

The researchers didn't perform an exhaustive analysis of the encryption
methods devised by Alice and Bob, but for one specific training run they
observed that it was both key- and plaintext-dependent. "However, it is not
simply XOR. In particular, the output values are often floating-point values
other than 0 and 1," they said.

Just saying...

~~~
Strilanc
Outputting floats instead of discrete values actually sounds like a very
exploitable hole to me.

It reminds me of the classic "check if your model corresponds to reality"
example where a one-time pad is broken: the standard said "<0.5V volts is OFF"
but your machine was outputting 0.1V for OFFs that came from computing 0 xor 0
and 0.2V for OFFs that came from computing 1 xor 1.

------
Touche
I'm not sure what conclusions we're supposed to draw from this. Just because
they were able to hide the messages from a 3rd AI doesn't mean the encryption
was good. Shouldn't a human examine it and see if they can codebreak it? Isn't
the goal better encryption?

~~~
dgacmu
I don't want to tell people what to take away from it, but my own view is that
the contribution of the paper is the framing of the problem and a preliminary
demonstration that you can achieve the goal of defeating a NN via an
adversarial training formulation. In a lot of ways, I view our work as
creating a challenge problem for the AI & crypto communities. It shows an
intriguing possibility that is far from being practical, and itself represents
a stepping stone in work on, e.g., training DNNs to learn to communicate.

Can this be extended to generate cryptanalysis-resistant schemes? Can we
create a formulation that can learn to use discrete operators such as modular
arithmetic? (The latter reflects one of my personal complaints about the state
of DNNs -- it's very hard to have discrete "blocks" of functionality if that
functionality isn't differentiable). Can we identify mechanisms that can train
networks such as these more robustly? (It fails to train roughly half of the
time; this is a fairly common problem in adversarial training of DNNs today.)

~~~
jsprogrammer
NNs/ANNs/DNNs/xNNs have already[0] been framed as easily fooled.

[0] [https://arxiv.org/pdf/1412.1897.pdf](https://arxiv.org/pdf/1412.1897.pdf)

------
pzh
Isn't that a bit like "security through obscurity"? Unless I'm
misunderstanding something, the cryptographic algorithm that the AI comes up
with isn't guaranteed to be based on a provably NP-hard problem, so there
aren't any formal guarantees. It would also be very hard to reason about,
inspect, and prove correct.

Please note that I'm in no way dissing the researchers' work. What they did is
pretty cool, but I can't see an obvious way to use these AI-generated
algorithms in production systems where you may need to certify their
correctness.

~~~
paradite
You may be right. It was mentioned in the paper:

    
    
        Our chosen network structure is not sufficient to learn general implementations of many of the mathematical
        concepts underlying modern asymmetric cryptography, such as integer modular arithmetic.
        We therefore believe that the most likely explanation for this successful training run was that Alice
        and Bob accidentally obtained some “security by obscurity” (cf. the derivation of asymmetric
        schemes from symmetric schemes by obfuscation (Barak et al., 2012)). This belief is somewhat reinforced
        by the fact that the training result was fragile: upon further training of Alice and Bob, Eve
        was able to decrypt the messages. However, we cannot rule out that the networks trained into some
        set of hard-to-invert matrix operations resulting in “public-key-like” behavior. Our results suggest
        that this issue deserves more exploration.

~~~
dgacmu
Note that this was for the public-key formulation. For the secret key
formulation, it _is_ mixing in the key, so it's not pure security through
obscurity. It may be -- probably is -- relatively poor cryptography, but it's
not just doing a transformation of the plaintext without using the key.

The public key part is in there mostly because of the formulation, not because
we actually had any noteworthy success in training a network to _learn_
pubkey. The result we mentioned was probably an anomaly, but it's an
interesting anomaly to dig into more in the future.

But, in general, the parent is correct. There are few guarantees about a
scheme derived in this way. It's one of the things I think most interesting
for future work: Can one formulate "can't be cryptanalyzed by a mathematically
sophisticated adversary?" as an adversarial training goal? Certainly, it's
solvable for the one-time-pad-like scenario we used, but what about for more
general cases that permit key reuse?

~~~
thomasahle
> Certainly, it's solvable for the one-time-pad-like scenario we used, but
> what about for more general cases that permit key reuse?

Can't you force this simply by making the message a lot longer than the key?
If the network can encrypt variable length messages, surely the method must be
more advanced than a one time pad.

------
systemfreund
It's not clear from the article whether they train the networks with the same
shared-key in every iteration, or if they randomize it. Any info on that?

~~~
dgacmu
It's a random key paired with a random plaintext for each input. In the
experiments, the key is the same length as the plaintext.

Happy to answer more questions about it. (I'm one of the authors.)

~~~
rcorin
So basically the NN is learning a one time pad?

~~~
dgacmu
Kind of. Except that there's no restriction that there has to be a 1:1
correspondence between the key and plaintext bits (or characters) that get
mixed, as there would be in a conventional OTP. And, indeed, the DNN doesn't
learn that - it mixes multiple key and plaintext bits together. Probably in a
way that's worse than a true OTP -- the adversary is more successful than it
should be were the encryption scheme a "correct" OTP with XOR.

~~~
rcorin
Cool! Also, have you tried giving some bits of the key to the adversary?

~~~
dgacmu
I haven't. Interesting - that'd be a nice way to try to probe how strong the
encryption is (i.e., "bits recovered vs. key bits supplied to adversary").
I'll have to think about that more - thanks for the idea!

------
jacquesm
> While it seems improbable that neural networks would become great at
> cryptanalysis, they may be quite effective in making sense of metadata and
> in traffic analysis.

How does that square with the fact that the best cryptanalists appear to have
nothing but their own neural networks to work with?

This reminds me of the genetic algorithm that came up with a tone
discriminator that at first glance looked like it could not work (parts
connected wrong or not at all, and yet, crucial to functioning).

[https://www.damninteresting.com/on-the-origin-of-
circuits/](https://www.damninteresting.com/on-the-origin-of-circuits/)

~~~
dgacmu
Because humans have invented tools - math, notably - that they can use to be
more formal and precise when they deal with problems of this sort. To date,
neural networks can't -- they're doing very well at the kind of "intuitive"
pattern matching that our brains do, but less well on an algorithmic front.
It's one of the really interesting frontiers of AI right now. The folks at
DeepMind recently showed, for example, that a variant of their "Neural Turing
Machine" architecture could learn to do things involving more like what we
think of as discrete algorithms ([https://deepmind.com/blog/differentiable-
neural-computers/](https://deepmind.com/blog/differentiable-neural-computers/)
), but all of this is very early work.

------
biot
Schneier's Law: "Anyone, from the most clueless amateur to the best
cryptographer, can create an algorithm that he himself can't break."

------
pmyjavec
Poor taste with the Terminator graphic. A new level of lame fear mongering by
Ars Technica?

~~~
dharma1
I was hoping psychedelic dog heads would become the default stock photo for AI
news, rather than red eyed robots

------
sly010
"The researchers didn't perform an exhaustive analysis of the encryption
methods devised by Alice and Bob, but for one specific training run they
observed that it was both key- and plaintext-dependent. "However, it is not
simply XOR."

I think this says it all.

------
madenine
Really interesting that for the first ~6500 steps Eve was better at decrpyting
the messages than Bob, which had access to the key.

Would be cool to see a setup like this in which Eve is a cohort of decryption
tools, otherwise your Alice network will just overfit against its mechanics.
Ie, if Eve uses lockpicks to open a door, an Alice solution would eventually
replace the key-lock with a number pad - not necessarily any more secure, but
will foil Eve every time.

~~~
timvdalen
>Really interesting that for the first ~6500 steps Eve was better at
decrpyting the messages than Bob, which had access to the key.

This is probably because Alice wasn't good at encrypting yet.

~~~
madenine
On a second look, seems like a combination of Alice not being good at
encrypting (in a sense that it had to encrypt in a way that makes decrpytion
possible), and Bob needed to get up to speed on decrypting (even with the
key).

As soon as that relationship clicks, Bob get better very quickly, while Eve
improves at a slower rate for a while, before Alice adjusts (causing bob to
get worse for a bit, until it catches up)

Would be interesting to see a set up where Alice / Bob get trained first, then
Eve is activated and see how they adjust.

------
wideem
Next: Google AI encrypts google's servers and demand Ransom

------
JulianMorrison
Isn't this just the automated version of "anybody can invent crypto they can't
break themselves"? It doesn't sound like the neural nets were provided with
any high level cryptanalytic primitives. Just basic math and left to bodge
something together. I can't imagine it would be easy to hill-climb towards an
understanding of the _why_ behind crypto. Algorithms that work look a lot like
ones that have glaring flaws - but those flaws only glare if you know the
right way to come at them.

------
rfreytag
It must be the time for Cryptography + Machine Learning. I had an O'Reilly
Security 2016 talk "Machine Learning to Improve Random Number Generators"
accepted this June 2016:

[http://conferences.oreilly.com/security/network-data-
securit...](http://conferences.oreilly.com/security/network-data-security-
eu/public/schedule/detail/53732)

EDIT: I thought the reference was relevant but please do critique/comment if
you feel I've broken some convention.

------
ajamesm
Hill climber climbs hill, news at 11?

~~~
fooker
Interesting choice of words!

------
strictnein
Interestingly, Eve maintained a better than random chance at guessing the
message. Consistently stayed a little better than 50%.

------
akfish
This reminds me of Peter Watts's novel <Blindsight>. Two captured "Scramblers"
(the highly intelligent alien without self-consciousness) managed to
communicate with each other in a way defying all human analysis, even with
human knowing what they are saying (the plain text).

------
greggman
For those that can handle old pre CG movies there's a great old movie where
computers make their own language. Highly recommended

Colossus: The Forbin Project (1970)

[http://www.imdb.com/title/tt0064177/combined](http://www.imdb.com/title/tt0064177/combined)

------
martin-adams
At an open day for an animation course they showed an AI derived walk cycle of
a character. The results were amusing. The condition was that if it feel over
that would be no good. The end walk cycle was a character doing forward flips
in the air and landing on its feet. Very unexpected results.

------
nullc
"No one knows how it works"

Yea, someone might also have said that about the GOST sboxes:

[https://eprint.iacr.org/2015/812](https://eprint.iacr.org/2015/812)

------
1_2__3
Given I saw this as top on /r/futurology I figured it was clickbait BS. But
since it was posted to HN I thought I'd give it a read.

It's clickbait BS.

~~~
benmcnelly
Its interesting, but needs someone to re-submit with a proper write up about
it without the fearmongerng titles.

------
rubyfan
Teach the AI to communicate with each other in a way humans can't eavesdrop
on... What could go wrong with that?!?

------
deegles
Google should teach an AI to do prime factorization instead.

~~~
rak00n
Why?

------
israrkhan
maybe another AI system do crypt-analysis and break it. and nobody will know
how it was broken.

------
chiefalchemist
By the time we finally realize we're not as smart as we think we are, it's
going to be too late.

------
excalliburbd
Am I the only one wondering about this and Ethereum...

~~~
drdeca
Do you mind elaborating?

If you are talking about a smart-contract being able to do stuff with
encrypted data without other people being able to access the data, I don't
think this would help with that, because whatever system the contract would
use would be available to whoever is trying to get the data from the contract.

Though, things may be enough to make it hard for other contracts to get stuff
from a particular contract? I'm somewhat skeptical of that, but maybe. Also,
in any case, the "attacker" contract could just make an incentive for people
to give it the information, which I think would work. (and that might be
necessary for it to get the info from the defending contract anyway? I don't
remember. I don't think it is, but I'm not sure.)

However, I'm not sure that is what you meant, and also I'm not sure of what
I'm saying.

edit: well, there are other systems that have been designed to let contracts
keep information secret but I think those are still too expensive to be used,
and relied on a secret sharing scheme iirc, so would require that not all the
info be on chain.

------
asveikau
> this is about teaching computers to re-write their own code.

Maybe I've seen too many movies, but I hope we can make sure they don't
rewrite the "protect the humans" part.

~~~
treehau5
but don't we then just unleash the huge EMP bomb?

~~~
nkozyra
Sounds good. Sadly, six months ago the bot brain predicted the day we were
planning with .98 confidence.

~~~
idlewords
But then it got sucked in to TVTropes, and the world is saved.

------
jdimov10
Duplicate of
[https://news.ycombinator.com/item?id=12800228](https://news.ycombinator.com/item?id=12800228)

------
msh
What could ever go wrong with this....

