
Show HN: DeepRhyme (D-Prime) – Generating dope rhymes with machine learning - sweezyjeezy
https://swarbrickjones.wordpress.com/2016/11/07/deeprhyme-d-prime-generating-dope-rhymes-with-deep-learning/
======
jcoffland
To the uninitiated these may seem like reasonable results because to the
uninitiated hip hop is just a bunch of unintelligible words strung together
anyway but these lyrics don't make much sense. There's no cohesive stream of
thought behind the lines. It's a semirandom juxtaposition of phrases and a
great example of one of the biggest limitations of deep learning softwares.
Even the best are only able to maintain very limited context. This is not just
a matter of scaling up to larger networks. Current methods require
exponentially larger networks to achieve linear increases in context handling.
So then why can our brains do it? The answer is simply that our brains are
more than neural networks.

~~~
Florin_Andrei
> _So then why can our brains do it?_

Because current NNs only simulate, like, less than 1 mm^3 of brain matter.
Someone writing lyrics for a song has millions of such tiny networks working
concurrently in their brain - and then there are higher-level networks
supervising and aggregating the smaller nets, and so on.

Current AI NN architectures are flat and have no high-level structure. There's
no hierarchy. There's no plurisemantic context spanning large time intervals
and logic trees. No workable memory organized short-, mid- and long-term. Etc,
etc, etc.

We're not even scratching the surface yet.

~~~
nickpsecurity
Further, the brain uses subsystems whose architectures are good at the
specific problem their solving with smooth integration that somehow often
preserves context. The approach of current ANN's seems to be equivalent of
taking visual cortext or other single subsystem then trying to apply it to
every other subsystem's area of expertise. It will do interesting things but
always fall short since it's architecturally not the right tool for the job.

It's why my money is on the humans for the upcoming Starcraft challenge. Like
poker, bluffing is integral to the process. AI's have been having a really
hard time with that in poker except for constrained games. Starcraft provides
enough open-ended opportunities that the AI will get smashed by clever human.
Hip hop similarly combines tons of references to pop culture, psychology of
what goes with what, coordination of presentation, and established musical
tricks. At least half of them ANN's suck at by design.

~~~
Florin_Andrei
There are two ways the AI could win at Starcraft:

1\. The game does not really depend on a larger context. What you see is what
you get. The "muscle memory" of a relatively simple ANI could therefore be
enough. This is partially in contradiction with what you said above about
bluffing, but I feel the contradiction is less than 50%.

2\. Simple "muscle memory" strategies should not be enough to win the game,
but the ANI's lightning speed reactions and its ability to see the whole game
at once are enough to outperform more sophisticated-thinking humans who are
slower and have tunnel vision w.r.t. the game. Basically the brute-force
approach.

I'm not placing bets, and I'm as curious as everyone else as to the result of
the contest. I'm just saying - if the AI does win, these are the ways it could
do that.

I'm using the expression "muscle memory", which is inadequate, because I have
no better way to express how current NNs operate. They are dumb at the higher
semantic levels. They only become powerful through colossal repetition and
reinforcement.

Watching current NNs being trained never fails to give me flashbacks from my
college days when I was practicing karate. We would go to the dojo, pick a
technique, and then repeat it an enormous number of times, to let the sequence
sink into muscle memory. I'm sure I still have some (natural) NNs in my brain
that still have got that thing down pat - I don't have to think about doing
the techniques, they just "execute" on their own. But there's no semantic
level here, it's just dumb (but blazing fast) automation.

~~~
nickpsecurity
It's possible but it's a lot more context-driven than you think with a human
involved. The bots do great against each other with their strategies and
muscle-memory stuff. Throw a human in the mix, they start noticing patterns in
how the units are managed, what strategies are likely, etc. They've exploited
these in the competitions to hilarious effect. Here's the main site on prior
work & results at the competitions:

[https://www.cs.mun.ca/~dchurchill/starcraftaicomp/reports.sh...](https://www.cs.mun.ca/~dchurchill/starcraftaicomp/reports.shtml)

Here's a few examples of how the muscle-memory approach, esp if focused on
unit vs unit, can fail against humans.

" In this example, Bakuryu (human) notices that Skynet's units will chase his
zerglings if they are near, and proceeds to run around Skynet's base to
distract Skynet long enough so that Bakuryu can make fying units to come
attack Skynet's base. This type of behaviour is incredibly hard to detect in a
bot, since it requires knowledge of the larger context of the game which may
only have consequences 5 or more minutes in the future. " (2013)

Note: At this point, they also sucked at building and expansion strategies
which surprised me since I thought a basic planner would be able to do that.
The constraints between rate of expansion, where to place stuff, what units to
keep/build, and so on get really hard. The other thing they weren't good at
was switching strategies mid-game based on what opponents are doing.

" despite Djem5 (pro) making the bots look silly this year... they were able
to defeat D-ranked, and even some C-ranked players. After the human players
have played one or two games against the bots they are then easily able to
detect and exploit small mistakes that the bots make in order to easily win
the majority of games... " (2015)

I don't have the 2016 result yet. It's clear they're getting better but
there's a huge leap between bots and humans. Gap seems to be context, reading
opponents, and expansion. Now, _if_ they can fix those, they have a chance of
taking down pro's combined with the machines inherent strength in
micromanagement and muscle-memory on specific attack/defense patterns.

Examples below of the AI's managing units in perfect formation & reaction
below. It's like an emergent ballet or something. The second one could be
terrifying if they can get it into real-world military. Figured you might
enjoy these given your position. :)

[https://www.youtube.com/watch?v=DXUOWXidcY0](https://www.youtube.com/watch?v=DXUOWXidcY0)

[https://www.youtube.com/watch?v=IKVFZ28ybQs](https://www.youtube.com/watch?v=IKVFZ28ybQs)

------
kastnerkyle
OP and others may be interested in this approach by Sony CSL from ~2012 using
constraint Markov chains for matching lyrical style with explicit rhyme
constraints, including (e.g.) Bob Dylan style with constraint of Beatles'
Yesterday [0].

This is the precursor to the work that brought on "Daddy's Car"[1], though the
techniques appear to have changed a bit reading their recent publications. I
find their paper on recent approaches for music synthesis with style matching
or other constraints [2] quite readable, and really good! I am hoping to build
out a version of what is described by the paper. There is also a very nice
talk by F. Pachet on this subject here [3], and his older videos are good too
(they should be in the recommendations of YouTube).

Still reading about DeepRhyme, but it seems great. Also a fantastic writeup!

[0]
[https://www.csl.sony.fr/downloads/papers/2012/barbieri-12a.p...](https://www.csl.sony.fr/downloads/papers/2012/barbieri-12a.pdf)

[1]
[https://www.youtube.com/watch?v=LSHZ_b05W7o](https://www.youtube.com/watch?v=LSHZ_b05W7o)

[2] [https://arxiv.org/abs/1609.05152](https://arxiv.org/abs/1609.05152)

[3]
[https://www.youtube.com/watch?v=5K4hn6cBUPU](https://www.youtube.com/watch?v=5K4hn6cBUPU)

~~~
sweezyjeezy
Hey thanks! I will check this out tomorrow!

------
gwbas1c
I seriously thought I was going to click on that link and see the
stereotypical cliche rap:

Yo yo yo

I'm Deep Rhyme and I'm hear to say

I figured out how to rap in a whole new way!

My name's D Prime and I'm an A-I

Maybe some day I'll calculate all of Pi!

Deep Rhyme in the house, yo!

(mike drop)

~~~
bbctol
To be fair, that would be a far more impressive result than this!

------
michaelbuckbee
Linked (but a little buried) in the post is this absolute standout of a
YouTube video about the rhyme scheme in Eminem's Lose Yourself.

[https://www.youtube.com/watch?v=ooOL4T-BAg0](https://www.youtube.com/watch?v=ooOL4T-BAg0)

Really drives home both how much more sophisticated hip hop has become and the
incredible degree of difficulty in trying to simulate such sophisticated
language structures.

~~~
dancek
Eminem has similar trickery going on in most songs and that's probably one of
the reasons of his success. Of course many other rappers do the same, and
surely Eminem isn't the inventor of funky rhyme patterns.

It's been 15 years but I used to analyze the patterns. Brings back memories.

------
to-mi
A rap lyrics generation algorithm with deep learning was also recently
published [1] in KDD conference and it also got quite a lot of publicity in
other media: see the associated website at
[http://deepbeat.org](http://deepbeat.org).

[1] Malmi et al.: DopeLearning: A Computational Approach to Rap Lyrics
Generation
[http://www.kdd.org/kdd2016/papers/files/adf0399-malmiA.pdf](http://www.kdd.org/kdd2016/papers/files/adf0399-malmiA.pdf)

~~~
sweezyjeezy
Author of DeepRhyme here. With respect to the authors of DeepBeat, what they
are doing is less ambitious. They are taking full existing lines out of a rap
lyric corpus and assembling it into a verse that rhymes and makes sense. There
is a paper that is very similar to what I did :
[http://www.emnlp2015.org/proceedings/EMNLP/pdf/EMNLP221.pdf](http://www.emnlp2015.org/proceedings/EMNLP/pdf/EMNLP221.pdf)
. It's hard to compare our models, because they don't give much output text.
They trained on 1% of the data I did, so I'm a bit dubious how successful they
could have been.

I am writing a followup post where I'm going to talk about previous work, I
hope no one takes any disrespect.

~~~
YeGoblynQueenne
>> It's hard to compare our models, because they don't give much output text.

The referenced paper doesn't give many examples of output, but the authors
have made the system available online on the url given above.

~~~
sweezyjeezy
I think you're talking about DeepBeat, they aren't tackling the same problem,
as I mentioned.

------
casoninabox
First, this is awesome. Great results.

I have been working on exactly this for the past few weeks. I went ahead and
produced a song and a lyric video from mine (I call it Lil' Markov). It
started out because I noticed patterns in Lil Wayne song where he'd say some
idiom and then rhyme it. It seemed so predictable I figured I could make a bot
that automated Lil Wayne's process...

From what I can tell our methods are very similar. I have a thing that allows
you to input a concept and it will try to rap around that concept and a
`whiteGuilt()` function (that's whats up with all the ninjas), but other than
that our process is 90% similar.

We should collaborate on something! Message me.

Version 0: [https://soundcloud.com/cason-
clagg/aesopxcasonxmarkov](https://soundcloud.com/cason-
clagg/aesopxcasonxmarkov)

Version 1:
[https://www.youtube.com/watch?v=fmcrjH6BKag](https://www.youtube.com/watch?v=fmcrjH6BKag)

Version 2: (No music yet, bit here's some sample lyrics based on the concept
"beef")

FIND MYSELF WITH MILLIONS THE PIGGIES THAT'LL SHARE HOPPIN' AROUND SATURN'S
RING I'M JUICED OFF DOG DARE

A DESCRIPTIVE LYRICIST LOUNGING IN LOVE LIKE YOU AIN'T EAGER TO KILL EACH ONE
OF THE DOVES

HE RAN FOR A BEAST HE'S A PARAGON OF YORE DIDN'T THINK OF HIS BLOOD ON THE
FACTORY FLOOR

------
adamsea
As a huge hip-hop head, I will just add that there are two aspects to
evaluating rhymes: How it sounds when delivered by the artist, and how it
reads. The best rap succeeds in both domains. Bob Dylan won the Nobel Prize
because he succeeds in both domains. In this case, we obviously can only
consider the second domain, how it reads. So we should evaluate the results
exactly how we would evaluate any other poem (while giving due respect to the
conventions and tropes of the form, ie, rap has its own classic metaphors and
themes).

------
uniclaude
Don't know how good you are with cache invalidation

But your naming game is tight, that's without a question

~~~
corey_moncure
I think you're on to something here! It might be a while before we train a NN
that can out-rap the pros. But can we raise the bar for applying the style to
alternative content domains to generate educational raps that aren't lame?

------
jeyoor
I wonder if approaches like this will work for poetry (or prose) from
different cultures. I'd be especially interested to see if some of these
techniques are effective on languages markedly different from English. (Maybe
Chinese and Arabic?)

~~~
DrNuke
For that, they should consider the underlying universal metrics (or meters)
instead of the local languages, along the path derived by Prof Nigel Fabb in
the UK [https://www.amazon.co.uk/Meter-Poetry-Theory-Nigel-
Fabb/dp/0...](https://www.amazon.co.uk/Meter-Poetry-Theory-Nigel-
Fabb/dp/0521713250) of which you have a review here
[https://web.stanford.edu/~kiparsky/Papers/fabb_halle_review....](https://web.stanford.edu/~kiparsky/Papers/fabb_halle_review.pdf)

------
daveytea
This is great. Well done to OP! I've been considering creating something like
this in the future as well.

You mentioned that you used the lyrics from 50,000 rap songs. Did you have any
filters for quality or certain sub-genres? What era were the rap songs from?
Maybe the 'quality' would be higher if the data set was narrowed to what you
consider quality hip hop lyrics.

Also from your post: "You can see that D-Prime does internally rhyme to a
small extent, but making the model better in this regard seemed like it was
going to be tough, and I didn’t pursue it." What were your main challenges to
improving/pursing better internal rhyme schemes?

~~~
sweezyjeezy
Hey man, thanks! I didn't try to filter to any subgenres, but I did try to
give the model as easy a time as possible by trying to stick verses which
stuck to the most common words (lowering the perplexity of the corpus). I
would call this genre 'mediocre', and may explain why I had such a hard time
with the sexism thing.

Internal rhyming / assonance and other stuff could be done by rewarding the
model for using it during the beam search. I suspect it's going to be hard to
tune how to set the reward so it still makes sense, it _only just_ makes sense
already. I also think at this point you would really need to sit down and
figure out what rhymes and what doesn't. I did the quickest, hackiest thing I
could think of, which was good enough for generating end rhymes, but if you
want to have internal rhymes and not massively constrain the model, I think
you need to work it out purely from analysing the syllables.

------
Karunamon
I have to giggle a little bit at the implications of taking source material
that's just about every negative *ist word it's possible to be and then using
ML to generate a version without that content.

~~~
farright
I think this is where were supposed to invoke the intersectionality fairy.

------
infinitone
This is awesome! I've been meaning to do something like this for awhile..
great that someone else finally did it.

Any chance you'd open source it?

------
YeGoblynQueenne
For a moment there I thought that thing had learned to rhyme on its own.

Then I figured out it's got some rules that help it along.

Phew. That'd have been scary.

------
david-given
I really want to drop in a Shakespeare play or two into the dataset and see
what comes out.

MC MacB comin for yo, Dun-king.

~~~
FroshKiller
Mad Magazine (or maybe it was Cracked) did a pretty good rap parody of Macbeth
years ago, maybe the late '80s or early '90s.

------
0xdeadbeefbabe
Somehow these AI projects always seem to demonstrate how real intelligence is
better.

~~~
singham
Baby steps. We all were babies once.

------
grandalf
It would be fascinating to try training it on some HRC and Trump speeches. I
am doubtful most people would be able to tell the difference between generated
remarks and the real thing.

------
david-given
...does enough chap-hop exist to form a decent dataset?

[https://www.youtube.com/watch?v=0iRTB-
FTMdk](https://www.youtube.com/watch?v=0iRTB-FTMdk)

------
ada1981
I'd like to see you team up with some AI Beat generators and then use text-to
speech and produce albums.

------
lifeformed
Next, you should plug this into a text-to-speech program and play it with a
simple beat generator.

------
aantix
Looking forward to the time

When Siri can spit

Some dope rhymes

------
singham
Next stop : Pun recognizer or pun generator.

------
mobiuscog
How will we know the AI versions ?

------
jgotti92
Awesome!

------
simplemath
D-Prime is an incredibly good AI MC name

Whoever thought that one up deserves free lunch in perpetuity

~~~
mrdmnd
No Free Lunch ;)

------
sjfiekzjndn
I'm sensing an impending gangsta job shortage. I wonder if machines become
better at being outlawz, then might these individuals pursue some form of
higher street education to keep up?

