
A.I. Duet: A piano that responds to you - dsr12
https://aiexperiments.withgoogle.com/ai-duet
======
krosaen
I like the concept and the framing of the demo is really cool, but the actual
music generated in response to me doing basic things like playing a blues riff
or a pentatonic scale was usually just a repeat of some fragment of what I was
playing. It felt like something that could be built using a basic heuristic...

~~~
svantana
Yeah either the examples from the video are cherry-picked, or something's
wrong with the deployment, because what I get is way worse than what's in the
video.

I really don't like this new trend of throwing Deep Learning at anything
without really checking whether a simple model (e.g. markov chains in this
case) would give equal results. Because using a simple model wouldn't be
"cool" or newsworthy...

~~~
sb8244
Here's a relevant excerpt from research I did on college into this. FWIW, my
research was specifically into genetic algorithms and generating similar
musical styles as the input music. It didn't perform particularly well but had
some surprises.

From my research, a Markov model could potentially be viable for this demo in
particular. However, it wouldn't be viable for longer generation of music
where you're trying to produce an actual song. The biggest problem being that
it doesn't have the concept of melody in it.

\--- research after here ---

A Markov model utilized by Herfort and Rehberger [17] uses the immediate
previous note in order to determine the next note against a set of
probabilities. This is a fairly simple algorithm as the set of rules is not
large, but it is extremely short-sighted by nature. This “first order” Markov
chain is not capable of determining a probability based on a large sequence of
pitches. To counteract this, “N-order” Markov chain models can be determined
to look more deeply into the past, but the complexity of the rule set
increases exponentially with added depth. Further, every aspect of music that
is to be modeled would require its own Markov model. For instance, there would
need to be a model for pitch, duration, etc. This leads to a potentially huge
Markov model rule set where all chains are independent of each other. Thus,
Markov chains are good at composing musical melodies on the micro level (such
as a measure) but fail to produce quality music at the macro level.

[17] Rehberger, Leopold. “Composing music using AI algorithms.” GRIN
Publishing GmbH. 2004. <[http://www.grin.com/en/e-book/108991/composing-music-
using-a...](http://www.grin.com/en/e-book/108991/composing-music-using-ai-
algorithms>).

Source:
[https://github.com/sb8244/GeneticMusicComposition](https://github.com/sb8244/GeneticMusicComposition)

~~~
JD557
Just to complete your post, one problem I always have with music generation
using "first-order" Markov Chains is that they get really confused by
children's music. For example, here's the start of "twinkle twinkle, little
star":

CCGG AAG- FFEE DDC-

So the tranistion matrix contails rules like: P(C -> C) = 50% P(A -> A) = 50%
P(F -> F) = 50% P(E -> E) = 50%

Which makes huge loops of the same note quite common. This makes for some
really uncanny music.

------
camtarn
Me: _plays a couple of minor chords_

AI: _long pause_ ; _holds down one random key for several seconds_ ; _boops
another random key_ "Nailed it."

After spending a few minutes playing, I did see evidence of it occasionally
tracking my chords (as long as they were very common - e.g. c major, a minor)
and tempo (but only matching tempo, as in vaguely matching how fast I was
playing, not actually playing in time with me). I played an extended chord
sequence and it made one or two nice contributions, but mostly it was just
random noise.

I wonder if this is due to lack of training data? Maybe it needs an actual
data set of people specifically trading musical phrases with each other,
rather than just a set of general music.

It's an amazing idea, and I can actually see it being quite interesting for
things like musical phrase creation - supply the first half of a phrase and
see what the machine comes up with for the second half, and tweak as
appropriate. But currently it's a bit of a case of over promising and under
delivering.

------
6stringmerc
Definitely a work in progress. Was like having a goddamn cat following my
fingers around the piano. No sense of rhythm or tonality whatsoever. Like, it
objectively made whatever I was playing sound worse.

It may "respond" but it's a lot more like throwing electricity through a dead
body than working with a competent duet partner.

~~~
Ajedi32
FWIW, it's not really meant to be playing alongside you. The idea is for you
to play a short tune, then wait for the AI to respond with a tune of its own
based off of yours.

~~~
6stringmerc
Well, FWIW as well I tried exactly that as well and it was like it could
barely follow what I did (rhythm was off) and then it went in other directions
that were not pleasing to the ear. Nor made much of any sense musically. So,
again, it's like a cat or child was mimicking my motions just enough to make
me dislike the result.

------
biot
This is fun. It actually feels like I'm taking turns with someone else on the
other end of the screen who takes what I play and sends back a riff on the
notes. The trick is (as the little intro shows) to only play for a few
seconds, don't play simultaneous notes, and then stop and let the AI respond.
Once the AI has finished, you can try again. If the AI starts playing before
you're done, you've screwed up and should make your little sequence of notes
shorter. Basically, the title of this is misleading... if you keep playing
expecting the AI to join you in a simultaneous duet, it'll sound like cats
fighting inside a piano.

Try playing the "Dukes of Hazard" horn: on your keyboard, press (without
overlapping the keypresses) "LJGGGHJKLLLJ" and let the AI respond. More often
than not, it'll come back with a vaguely pleasing riff on that sequence using
mostly the same notes and in the same tempo.

------
ssharp
My immediate reaction was to try and confuse it, so I played "Roll Call" from
the Thomas show (kids are obsessed !) It's a bit chromatic so I thought it
would have trouble, which it did.

Next, I tried putzing around on the white keys but made sure to never hit a F
or F#. I wanted to see if it would try and guess which key I was playing in, C
or G major. It seemed to guess C most of the time, though one time I tried
hinting at G by starting out on G, it got confused and starting playing both F
and F#.

I was just clicking with a mouse, so rhythm was hard to judge but overall this
thing did a really bad job "complimenting" my playing!

~~~
jff
You can use your keyboard, the middle row maps to the white keys and the upper
row maps to black keys. Works better than a mouse for tempo, IMO.

~~~
coroxout
Are these standard keys for some other piece of music software?

I keep getting confused because I'm so used to the old-style tracker keys (I
keep hitting J for Bb and getting B, which isn't helping my C minor improv).
Also one octave isn't great, lag is killing the rhythm, and it keeps not
waiting for me to finish.

Neat idea though.

(I'm curious to know whether the training set for the version shown in the
demo is the same as for the live web version, as the demo sounded way better
than anything I'm getting out of it - though obviously they would have
cherrypicked the results for the video.)

~~~
fryguy
It seems to be the same keyboard mapping that Reason uses:
[https://youtu.be/7maQ1zjvb-M?t=28](https://youtu.be/7maQ1zjvb-M?t=28)

------
jpalla
If you are interested in real-time generative music, check out IRCAM's project
SoMax/OMax:
[http://repmus.ircam.fr/omax/home](http://repmus.ircam.fr/omax/home)

Sample (software is piano trained with Bill Evans corpus):
[http://www.dailymotion.com/video/x2j39hf_somax-with-remi-
fox...](http://www.dailymotion.com/video/x2j39hf_somax-with-remi-fox-at-the-
ircam-open-doors-june-2014_music)

------
wufufufu
I played the A-section for Heart and Soul (the song from the movie "Big") and
its response was enough to tell me it was not going to be playing actual
duets.

I'd be interested in a version that has learned accompanying portions to many
popular melodies. If I could play the right hand part of a piece and it played
the left hand I'd be very impressed -- also this would actually be useful
software for aspiring pianists.

------
amelius
I think Google can learn something from Microsoft here: [1]

[1]
[https://www.youtube.com/watch?v=3oGFogwcx-E](https://www.youtube.com/watch?v=3oGFogwcx-E)

------
dominotw
just tried this on my midi keyboard( web midi is awesome!) but the "AI music"
response was very underwhelming and sounded bad.

I would be interested if "AI" can help me with different timbres based on the
physical environment that music is playing using various sensor like air
pressure, humidity, room resonance, maybe even heartbeat, perspiration ect.
Make that a new dimension of information coded into the song along with pitch,
timing, amp and timbre.

------
ovulator
I'm not sure if it is a reflection of my own lack of understanding of music,
but it sounded like chaos to me. Garbage in garbage out though.

------
cyberferret
I was excited to play with this, but ultimately left disappointed. I was
hoping that my initial melodies would be expanded upon and that I could build
a series of riffs that would become increasingly complex but based on the same
starting melody, but in the end, the responses from the AI seemed to be fairly
random, and didn't seem to be built upon the previous sequences at all. The AI
also changed keys a couple of times, which was disconcerting.

------
catnaroek
To whoever implemented this AI: Can you actually play music yourself? Your AI
can't. I tried to play “Satori Maiden” (link:
[https://www.youtube.com/watch?v=0c9bea_thaM](https://www.youtube.com/watch?v=0c9bea_thaM)
), and your AI only replied with unmusical nonsense.

~~~
hellofunk
Go easy man. It's an "experiment" for a reason.

~~~
catnaroek
I'm opposed to this trend of trying to replace actual understanding of the
problem at hand with massive amounts of computation. If you want to write a
program that writes music, then learn how music is actually written first.

\---

@hellofunk: If you want the ML algorithm to actually learn how to write music,
at the very least, the programmer must be able to express the goal of
producing musical sequences of notes.

~~~
jgalloway___
It's opensource bro. Translate your problem to a requirement and impress us.

~~~
catnaroek
Why would I? While I use computers to play music all the time, I don't think
using computers to write music is a good idea in the first place.

\---

@jgalloway___: I'm not sure if I'd call it an “expectation”. I just dislike it
when people throw random shit at the wall and see what sticks. When you write
music, you should first _think_ how it's going to sound, convince yourself
that it's plausibly good-sounding, and _only then_ actually play it just to
make sure it sounds good. If you do it the other way around (first play random
stuff, then filter out what doesn't sounds good), you're a wanker.

\---

@ohitsdom: I'm totally fine with improvisation driven by _ideas_ , not random
computation. Coming up with ideas on the fly is good. Passing noise as genuine
ideas is not good.

~~~
ohitsdom
That's fine that it's not your style, but experimentation is a huge source of
new music and reinvented ideas. Dismissing any music that wasn't strictly
planned seems a little silly.

~~~
TheOtherHobbes
You can grade experimentation exactly on its ability to source new music and
ideas.

This experiment does neither particularly well.

------
jgalloway___
For those looking for harmony try letting the machine lead .. if you mimic
it's style and complement with a hint of your own rhythm it can be satisfying.

------
stevehiehn
This is cool! I've been trying to use ML to automate the entire production
pipeline. Choose chords, synths, loops, mix and add FX.
[http://signalsandsorcery.com/](http://signalsandsorcery.com/)

Im finding extracting audio features from loops the hardest part. The
selection of notes is actually not as difficult as choosing a good timbre for
the notes.

------
rrherr
Style transfer and generative adversarial networks have worked well with
images. How can these be applied to music?

~~~
increment_i
It does seem like an interesting edge to explore doesn't it? ML techniques
combined with spectral analysis could yield some interesting outcomes. The
datasets wouldn't be too hard to find either - most people have pretty
extensive digital music libraries.

------
cocktailpeanuts
I know this is meant to be a "cool experiment", but from trying this out for a
while my impression is:

I could just build a random sequence generator and tell the world "I made this
with tensorflow" and people would believe. I mean, none of the response
sounded like music. Which is fine, but then again, what is the machine
learning for, if it just sounds like random keystrokes?

~~~
sordina
I think that this response is confused. What is your point? That people can't
differentiate random notes from a duet? That people would believe that
anything associated with tensorflow was imbued with sophisticated behavior?

How does that relate to this experiment? It's clearly not just playing random
notes in the example video...

------
y04nn
Make me think about the movie "Electric Dreams"

[https://youtu.be/aVE8taDWmMc](https://youtu.be/aVE8taDWmMc)

------
jrowley
I want to hear one trained on Monk, Bill Evans, etc.

~~~
zhemao
I get the feeling this one was trained on Cecil Taylor.

------
algomusic
Even more sophisticated duets with a computationally creative computer: CIM
Sessions 2017.
[https://www.youtube.com/user/QCGInteractiveMusic/videos?sort...](https://www.youtube.com/user/QCGInteractiveMusic/videos?sort=dd&shelf_id=0&view=0)

------
algomusic
Even more sophisticated duets with computationally creative computer: CIM
sessions 2017
[https://www.youtube.com/user/QCGInteractiveMusic/videos?sort...](https://www.youtube.com/user/QCGInteractiveMusic/videos?sort=dd&shelf_id=0&view=0)

------
stevehiehn
I've had the most success in my own experiments using many dedicated Models.
The first and most important model is a NN that generates a chord progression.
After that is much easier to select good notes!
[http://signalsandsorcery.com/](http://signalsandsorcery.com/)

~~~
tyingq
Markov chains might do better for progressions :)

~~~
stevehiehn
I have tried just using N-grams which is pretty good but it doesn't have any
structure. My next experiment will be to use the 'word2vec' algorithm the same
way it it used for NLP. Meaning first create relationships between chords then
create sequences with RNN.

------
jgalloway___
As an experiment I think this is well done. It's fun to mash the keys and see
what happens. If there was a feedback loop to register if the user was happy
or displeased with the results via facial recognition or other input with
expanded training data it could become quite interesting I think.

Thanks for this!

------
shams93
I'm getting some good results but I'm also a clasically trained musician so if
you haven't studied music it may not give you a pleasing result but if you are
a trained musician you may get more mileage.

------
dnprock
I think it'd sound more interesting if the computer compliments what I play.
For example, if I play in C key, the computer tries to guess the chord
progression and play the rhythm that matches what I'm playing.

------
return0
I believe ML can do better than this.

------
erinjerri1678
I tried this at Google Tensorflow Summit and it was fun!

