
Making music using new sounds generated with machine learning - gk1
https://www.blog.google/topics/machine-learning/making-music-using-new-sounds-generated-machine-learning/
======
bluetwo
Repost from yesterday.

[https://news.ycombinator.com/item?id=16576587](https://news.ycombinator.com/item?id=16576587)

~~~
gk1
Weird, usually HN will show an error or make a redirect when submitting
something that's been posted, but it didn't happen this time.

~~~
bluetwo
Same article but on a different URLs, both Google sites.

------
telesilla
Aargh come on Google, how about you do some research first with some long-
standing audio research institutions. Cross-synthesis isn't going to sound
particularly new because it's been around for decades.

Case in point: AudioSculpt manual, IRCAM 1996 (p62) (Second edition!)

[http://homepages.gold.ac.uk/ems/pdf/AudioSculpt.pdf](http://homepages.gold.ac.uk/ems/pdf/AudioSculpt.pdf)

Machine learning is gonna give us some awesome new sounds but you've got to do
your research first on what's been done.

~~~
8bitsrule
Yep. Simply put, what makes traditional instruments popular is the richness of
their sounds, and how they enable the musician's expressiveness. Great
traditional musicians don't switch between instruments, they mine the sonic
possibilities of the one they're holding. That personal thing touches the
audience, and reputation is built on it.

Of all the computer synths I tried out back in the day, Absynth was a
standout. BUT it's hard to 'get physical' with a computer keyboard ... let
alone a finger on a tablet. When the technology gets in the way, musicians run
away.

------
danilocesar
It's super cool, super smart, super fresh, but still sounds like those
synthesizer in 90's. I was really expecting something new when I opened the
link...

~~~
pknopf
Yeah, it seems like a lot of work and technical know-how to make something
that doesn't really sound new.

Seems more like a hipster-ish take on making music. "I only make music on
instruments that I can build myself."

~~~
operatorius
well you have pure data, max/msp, reaktor, etc for that. These experiments
with ML can approximate the soundwaves generated by particular instruments
where the soundwaves produced are very complex, but other than that there are
tons of software that can help you design any sounds you like. Machine
learning is like the micro thing in the 70's-80's companies are trying to put
it everywhere they can despite there are no benefits to the existing
convenient products/solutions or not incorporating ml at all but still
labeling the product. i.e. quoted from wiki: Microwaves are a form of
electromagnetic radiation with wavelengths ranging from one meter to one
millimeter. I see nowhere the 10^-6 meter waves being used in our commonly
used kitchen appliance

~~~
TheOtherHobbes
There are any number of ways to approximate the soundwaves generated by
particular instruments that work better than this.

There are also any number of ways to make cool new sounds, even using old
tech. (My current favourite is Aparillo by SugarBytes, which uses FM but
sounds absolutely amazing.)

This Google project seems to be a technology demonstrator made by people who
seem curiously unaware of the domain they're trying to work in.

There is a _lot_ of competition in this space and this project is doing
nothing remotely new, including the ML element.

All of which might be forgivable if the sound was unbelievably awesome... but
it isn't.

------
KriptoNYC
This is FFT-Based resynthesis, and a not very good-sounding implementation at
that. It’s understandable this is the technique that would be chosen as that
the format of the data used lends itself to averaging/blending. However, after
the bombastic headlines, I am truly disappoint.

~~~
mkj
Why do you say it's FFT based? The paper
[https://arxiv.org/pdf/1704.01279.pdf](https://arxiv.org/pdf/1704.01279.pdf)
appears to be generating temporal samples, the FFT based one is their baseline
comparison.

------
triplesec
I have it on very good authority (a fine music engineer friend) that this ten
year old project still remains the standard for an AI music generator. And
look at the people who used it
[https://en.wikipedia.org/wiki/Hartmann_Neuron](https://en.wikipedia.org/wiki/Hartmann_Neuron)

~~~
mmjaa
I once had a long discussion with Axel about the Neuron, which went like this,
around and around in circles:

Me: "So .. how does it work?"

Axel: "It doesn't matter how it works, musicians don't care, they just want to
hear something when they do something.."

Me: "Sure, okay .. but how does it work if I am a musician who does want to
know.."

Axel: "Real musicians don't want to know."

Me: ...

I'm honestly not convinced he knew how it works, either! :P

------
kubiiii
Somewhat related (released in 2002) the Hartmann neuron.
[http://www.vintagesynth.com/misc/neuron.php](http://www.vintagesynth.com/misc/neuron.php)
Never had the "chance" to put my hands on it.

------
acobster
Despite the requisite HN cynicism, this is really, really cool.

This is a bland marketing demo _of course_ they're not going to get to the
really interesting parts. Oh, this combo of instruments didn't really sound
that good...yeah, that's because they chose an arbitrary combination to
demonstrate _that they could_ , not because they're claiming that every new
sound will be amazing. An important aspect of any new medium is the ability to
make novel mistakes.

Music synthesis has been a thing for a long time now, so _of course_ there are
going to be sounds that you can get through some other technique. That doesn't
mean there can be no surprises using a similar (but still novel) approach.

Andrew Huang did a video a while back about nsynth (just playing with the
algorithm, before the nsynth super) and lo, despite nsynth's absolute dearth
of musical merit, he actually manages to make something cool.

Edit: link to that video:

[https://youtu.be/AaALLWQmCdI](https://youtu.be/AaALLWQmCdI)

~~~
nitrogen
_This is a bland marketing demo of course they 're not going to get to the
really interesting parts._

Isn't the point of marketing to _emphasize_ the interesting parts? If the demo
isn't compelling, who will take the time to make something that is?

------
_nrvs
This is such a cool project, especially the principles behind it (open
hardware!).

As with anything, there's prior art, so everyone should _chill_ :)

Being passionate about synthesis, I have to mention vector synthesis which has
a similar vibe in terms of interaction. More like simple mixing between sounds
but nonetheless really neat and powerful.

[https://en.wikipedia.org/wiki/Vector_synthesis](https://en.wikipedia.org/wiki/Vector_synthesis)
Yamaha TG-33
[https://www.youtube.com/watch?v=8DK7K5sFqWg](https://www.youtube.com/watch?v=8DK7K5sFqWg)
SCI Prophet VS
[https://www.youtube.com/watch?v=1lJL3blZKVM](https://www.youtube.com/watch?v=1lJL3blZKVM)
Korg Wavestation
[https://www.youtube.com/watch?v=i1fokDelaxM](https://www.youtube.com/watch?v=i1fokDelaxM)

------
brwsr
Clickbait? Where is that music?

Didn't hear one single interesting sound out of that box that can be used to
create music. Most producers and musicians already went back to analog because
it sounds so much better.

I would be really impressed if some machine learning algorithm can only
generate a kick that sounds better than the original analog TR808/909.

------
maroonblazer
I studied audio engineering and music synthesis in college and accumulated
racks of synths. I rarely composed a complete piece of music but rather became
consumed with 'sculpting' sound. After graduating I sold all the gear and
bought an acoustic guitar and microcassette recorder. Constrained to just
voice and guitar I was MUCH more productive.

Perhaps I'm naive or 'old school' but I'm struggling to believe we've
exhausted all possibilities of what can be done with just 12 notes (and their
octaves) and rhythm subdivided into 32nds or 64ths.

In other words I'm more interested in how ML can help us discover new
combinations of notes (melodically and harmonically) and rhythms rather than
new kinds of sounds.

~~~
kubiiii
Even when it comes to electronic music, most of the stunning new sounds we
come across are achieved by hacking or tweaking simple systems : over use of
compressor, overuse of LFO for dubstep basses, bitcrushing, use of half
defective samplers producing slightly out of tune sound, prepared pianos...

------
amelius
What I'd like to see is a deep-learning project which uses a brain sensor like
an EEG. It could learn what kind of music the listener likes (and under what
circumstances). This could be the first step, and a useful tool in itself.
Then in the next step it could learn to generate new music which the listener
likes, using the previously generated model and/or using a feedback loop
involving the user.

~~~
bitL
Do you really think a single simple metric like EEG would tell you much about
musical preferences (not mentioning type of music to synthesize) given 100B
neurons in average brain? For the first part you could try a RNN in an hour;
not sure it would give you any meaningful results though.

~~~
amelius
Well, I'm guessing that a simple metric could tell whether the user likes or
dislikes the music they are currently listening to. Advanced brain scans can
show activity in the pleasure center of the brain, so perhaps an EEG can do
something similar based on a learnable multivariate function.

You could view it as supervised learning, but with the user having only a
passive (listening) role.

------
diydsp
There is a tendency to judge and predict the commercial success of this (and
every) music project.

As a long-time player and designer of digital music instruments in academic
laboratory and underground contexts, i can say it's almost impossible to
produce a hit directly from a laboratory. It has to go through a commercial
layer first. Then an artistic layer.

That's not a dig at anyone or anything. Music is a projection of life's
experiences into sound... A lab can inject some tech into the music world, but
we won't know for approximately 1-2 decades whether any instrument or
technique is a hit or it.

FM Synthesis, formalized by Chowning at Stanford ,comes to mind. His initial
implementations were idiosyncratic. His math was an organization of community
knowledge. It took Yamaha's need to make sound cards and keyboards to direct
all this effort those sounds into the chips and instruments we ended up buying
and loving in the 80s. And it took artists and producers to bring out the best
of the techniques. Also, you'll find when the tech passes through the artistic
layer, there are many more mundane variables than the technology, such as re-
sale value and ability to get them re-paired.

For example, if you think these sounds are non-compelling, I dare you to look
up compositions by Chowning himself! [1] They are nothing like Samantha Fox's
"Touch Me," [2] a famous example of the Yamaha DX-7 FM synthesizer.

The point of research like this is not really to inspire music fans so much as
to inspire the next generation of commercial instrument designers and
eventually artists and producers.

[1]
[https://www.youtube.com/watch?v=988jPjs1gao](https://www.youtube.com/watch?v=988jPjs1gao)

[2]
[https://www.youtube.com/watch?v=W1btg3mpEOc](https://www.youtube.com/watch?v=W1btg3mpEOc)

------
telebone_man
I appreciate there's a sense of logic that manufactures the sounds that this
produces.

But, in every practical sense, the sounds are seemingly random to the
listener.

Abstract expressionist art can be enjoyed regardless of the fact it's visually
just a bunch of random scribbles.

But the sound this machine produces has to then be sculpted - lest we listen
to a droning tone.

Would be cooler if you could describe the timbre of a sound you want to hear,
and it produces that...

"I want a woody.. tinny.. percussive sound" (out pops some kind of
glockenspiel marimba hybrid) or "I want a breathy sounding noise that sounds
synthesized with the tone of tenor vocalist but a punchy distorted entry".

------
zo7
Does the hardware do anything interesting, or does it only process midi input
and provide an interface to mix between generated sounds? From their
instructions on creating new sounds, it looks like they're pre-computing the
generated sounds: [https://github.com/googlecreativelab/open-nsynth-
super/tree/...](https://github.com/googlecreativelab/open-nsynth-
super/tree/master/audio)

------
torgard
The 'fnure' reminds me of an orchestral stab:
[https://youtu.be/w0qnBU7fWKo](https://youtu.be/w0qnBU7fWKo)

------
_Marak_
Does anyone know how to compile the the open-nsynth algorithm into a VST that
could be used with a DAW? I couldn't find this yet.

source: [https://github.com/googlecreativelab/open-nsynth-
super/tree/...](https://github.com/googlecreativelab/open-nsynth-
super/tree/master/app/open-nsynth/src)

------
pasta
About real new sounds: I got the feeling all sounds are already generated.
This AI might generate variations of sound we already know, but could there be
any new sounds we never heard before?

I got the feeling that since like 1920 until now all sounds are generated and
we don't experience any 'new' sounds anymore.

Maybe AI could be used to explore sounds we really never heard.

~~~
kevincennis
You can combine sine waves in an infinite number of ways, so I think it's safe
to assume there are plenty of sounds nobody has ever heard before.

To some degree, it's about how you group things. If you believe a guitar and a
banjo are similar enough to qualify as the "same" sound – then no, there's
probably not too much left to discover. But if small subtleties in timbre are
fair game, then there's basically a limitless supply of new sounds.

------
plaidfuji
You know what would be a really useful audio tool that may be amenable to a
machine learning approach? An AI-mastering algorithm. Training set: pre- and
post-mastered audio files. IMO the biggest thing holding back indie producers
is that final step that turns your home recording into radio-ready music with
requisite "loudness".

------
modernerd
Google's self-assembly guide for the “Open Nsynth Super” is here:

[https://github.com/googlecreativelab/open-nsynth-
super](https://github.com/googlecreativelab/open-nsynth-super)

------
908087
I was really hoping the buzzword kamikaze would leave synthesis and music
alone for at least a few more years.

