
Gwern’s AI-Generated Poetry - gbear605
https://slatestarcodex.com/2019/03/14/gwerns-ai-generated-poetry/
======
jadbox
So what happens when in the next decade the best poets, illustrators, and
architects are all AI (assuming there's double-blind competitions)? What
happens when humans are finally able to take their inner 'Daemon' and puts it
into the heart of the machine? In the near term, it means media producers will
start harnessing this machine creativity in generating news articles,
generating marketing plans and designs, and other product development. I once
thought that the last remaining jobs for people in the robot revolution would
be in creative fields... however, I'm quickly changing my mind about the pure
power of ML to automate all of humanity's work and even creative outlets.
Perhaps human art will no longer be a commodity (because markets will be
saturated by AI gen art), but rather as a pure zen/tao act of self-expression
for only oneself.

~~~
IdiocyInAction
I always thought it to be a bit anthropocentric to assume that art is a harder
problem than say, exploring meaningful theorems in math. Art is basically
learning what the human brain and sensory processing find pleasant, in certain
ways. Sure, I believe that creating very fine art with deep meanings might be
hard, but consumer-grade stuff? I can readily believe that that is pretty
automatable given enough training data. Journalists in my country already
mostly copy-and-paste from a central agency...

~~~
TheDong
> art is a harder problem than say, exploring meaningful theorems in math

I think it is not all that difficult to argue that math is art.

Sure, some questions posed by mathematicians (such as "Is god's number 20?")
can be proven by computers with rote brute force, but the actual creative
process of playing with a given system, adding and removing constraints, and
seeing what emerges is very much a creative and artistic process.

One of the easiest ways to see this is to read a good math paper. They're
rare, but good pieces of math can spark the same sort of feelings that a good
pieces of art does.

~~~
IdiocyInAction
Well, that depends on what counts as art I suppose. If you are reductionistic
enough, I'm fairly certain you could throw in almost any meaningful creative
human endeavor in there.

But I mean it in the sense: Sensory inputs that evoke certain emotions (awe,
fear, joy...) in the human brain - that should encompass most things people
immediately associate with the term.

------
2038AD
I'm entranced by this one:

    
    
      My heart, why come you here alone?
      The wild thing of my heart is grown
      To be a thing,
      Fairy, and wild, and fair, and whole

~~~
aijoe
Really sounds like it's declaring it's real and asking why it has come into
being alone in all the world

~~~
jcims
At some point in the future there could be sentient technology that we
communicate with.

It seems to me that there could be flashes of sentience in technology that
occur well before that time..before we understand that there's someone
listening. That would be a lonely lot indeed.

------
Udik
Ok, so it's impossible not to cite here Stanislaw Lem's short story "Trurl's
electronic bard", from the Cyberiad collection.

In the story, a poet AI has been produced, and it's put to the test by asking
it to compose poems on extremely specific subjects and following almost
impossible rules. For example:

"a poem about a haircut! But lofty, noble, tragic, timeless, full of love,
treachery, retribution, quiet heroism in the face of certain doom! Six lines,
cleverly rhymed, and every word beginning with the letter "s"!

Here you can find the requests and the machine's results:

[http://www.art.net/~hopkins/Don/lem/WonderfulPoems.html](http://www.art.net/~hopkins/Don/lem/WonderfulPoems.html)

I wonder how long till we'll be able to do it for real.

------
personjerry
Rhyme and rhythm are pretty small-scope when translated to mathematical
domain. Manually making lists of rhyming words and counting syllables from
markov chains, you could fit these requirements.

Meaning, on the other hand, is a much larger vector to handle, and that's the
real test of quality here. The OpenAI GPT-2 seemed to have meaning nailed down
-- these poems clearly do not.

~~~
the8472
> Manually making lists of rhyming words and counting syllables from markov
> chains, you could fit these requirements.

That would involve human labor. The NN learns that by itself from having
enough data thrown at it.

> The OpenAI GPT-2 seemed to have meaning nailed down -- these poems clearly
> do not.

This is derived from GPT-2-small. So we already know that the state of the art
is already better than what we see here.

~~~
gwern
> This is derived from GPT-2-small. So we already know that the state of the
> art is already better than what we see here.

And there is so much that could be done. I have a laundry list here:
[https://www.gwern.net/RNN-metadata#improvements](https://www.gwern.net/RNN-
metadata#improvements)

~~~
pmoriarty
I've just been reading through your methodology in the first part of your
article (dealing with prose works like Twain, Austen, etc), where you mention
you strip off the beginning of Project Gutenberg books, which contain
boilerplate.

I'd like to suggest that you also strip the ends of their books, as they also
contain boilerplate. In addition, I'd suggest stripping out introductions. The
early results you got from Shakespeare sounded like they may have been taken
partly from the intructions, which weren't written by Shakespeare at all, but
by much later authors.

I also noticed that you ran out of memory at one point and reduced the neuron
count as a result. You might want to consider doing some quick runs on AWS (or
one of their competitors), where you can get plenty of memory (and also faster
machines). That way you won't have to compromise your NN architecture for lack
of resources.

Something else to consider is using some other optimization techniques like GA
or GP to optimize the NN architecture or NN parameters, and also to maybe have
multiple NN's vote on the results. Such metaheuristic and ensemble techniques
have shown promising results.

Yet another thing to consider is using something called Dynamic Subset
Selection to effectively train on the most difficult portions of the training
data. I have not used this technique with NN's, but it's worked well with GP,
and saves a lot of time.

~~~
gwern
If I were to revisit those specific experiments, I wouldn't use AWS as it is
very expensive. Fortunately, I now have my own workstation with 2x1080ti
(which have ~5x more VRAM than the mobile GPU I was using at the time, IIRC).

There are a lot of hyperparameter optimization methods, but HO is only
worthwhile if you can afford a lot of runs and usually delivers relatively
small gains compared to scaling up your model/dataset. Right now, it seems
like it would be a better approach to continue scaling up the Transformer
and/or switching to Transformer-XL than it would be to attempt hyperparameter
tuning of GPT-2-small finetuning training.

------
Impossible
RZA should use the poem that repeats

 _The Emperor Wu (the great Wu)_

As a hook and claim the first hip-hop song by a major artist co-written with
AI.

------
pmoriarty
This reminds me of Stanislaw Lem's Cyberiad[1], in which an _" electronic
bard"_ is challenged to come up with _" a love poem, lyrical, pastoral, and
expressed in the language of pure mathematics. Tensor algebra mainly, with a
little topology and higher calculus, if need be. But with feeling, you
understand, and in the cybernetic spirit."_[2]

The bard does so brilliantly (though, of course, it's really Lem himself who
wrote it).

[1] -
[https://en.wikipedia.org/wiki/Cyberiad](https://en.wikipedia.org/wiki/Cyberiad)

[2] -
[https://www.cse.wustl.edu/~jbuhler/cyberiad.html](https://www.cse.wustl.edu/~jbuhler/cyberiad.html)

------
the8472
_> First two lines are perfect rhyme and rhythm, next four have no rhyme but
are close to the right meter, next few have almost random length, and by the
last one we’ve abandoned grammar and are making up nonsense words like
“wholubil”_

Is that an intrinsic issue of the NN or with how information is extracted from
it?

~~~
gwern
The NN is imperfect, the extraction method is extremely imperfect, but I think
the problem there is actually the dataset: the Alexander Pope material has an
unusual amount of irrelevant prose interspersed which I think screws up the NN
[https://slatestarcodex.com/2019/03/14/gwerns-ai-generated-
po...](https://slatestarcodex.com/2019/03/14/gwerns-ai-generated-
poetry/#comment-731081)

------
dre85
With all of these AI generated arts, it's not clear to me whether this is
using the training data as a kind of template and then somehow generating
entirely novel content or if it's essentially just a somewhat sensible re-
organization or re-quilting of the same components. In the case of poetry,
maybe just a jumble of the original verses. Based on my knowledge of machine
learning, it's more like the latter scenario, which somehow doesn't strike me
as something radical...

------
JamesBarney
I'm waiting for gwerns AI generated critical theory essays.

------
selimthegrim
Wonder what it’ll do with Edward Lear.

Also Ravana’s jaw? Lol. Try Raktabija

