So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning?
Trying to measure the sadness of lyrics with an algo is tantamount to the early scene in "Dead Poets' Society" where the students are supposed to measure the greatness of poetry by plotting it on a chart.
It's demonstrated in the data: one of the most depressing songs of their entire oeuvre is "Fitter, Happier" but because it's completely metaphorical (or sarcastic), the algo rates it as one of the happiest.
This will always be a big problem in AI. Measuring the literal meaning of words is easy. Measuring the semantics and context of words is a completely different matter.
Neither am I. But 10 years ago I would have said I had no idea how an AI was supposed to recognise objects in a picture, and now they do it better than humans. Tech moves fast.
Recognising objects in pictures is something that, in principle, a machine could do better than humans, because there is ground truth.
I think it's not possible in principle for a machine to do better than humans at recognising human emotions, because a human can't exactly be wrong at this. (Other than edge cases such as not knowing the right name for an emotion, perhaps.)
But suppose a sample of 1000 humans answered a survey on emotional content of a song and a machine was able to predict the distribution within some tolerance. In a sense, this would be superhuman performance, but even if we don't like that interpretation, it would still be very good performance. And it is possible in principle. So the argument that humans sometimes disagree doesn't prevent a machine doing well on the task.
Another possible output for this type of task is an opinion or interpretation and a justification of it. A machine might feasibly be considered "good" if it output an unpopular interpretation with a good justification.
Great point! The same type of argument is going in the AI subfield of computational creativity. A computational system might produce a piece (of art or music, say) which nobody initially likes, but could provide a justification/interpretation which might be convincing. In some ways this would be a big improvement over a pure Turing test-style judgement of the output itself. I think Margaret Boden has written this idea into her definition of computational creativity.
EDIT: "The ultimate vindication of AI-creativity would be a program that generated novel ideas which initially perplexed or even repelled us, but which was able to persuade us that they were indeed valuable.", from Boden, Creativity and Artificial Intelligence, 1998. I believe, based on her other writings, that she thinks of it as a sufficient, but not a necessary condition, for AI creativity.
I thought before writing "always". But I do think it will be difficult for the foreseeable future, because as @Grue3 already pointed out, it's ambiguous even to humans.
The central problem of getting a machine to understand metaphor, irony and the like is going to be Hard by any definition.
Being ambiguous to humans doesn't make it an ill-defined task. A machine which perfectly mimicked the judgements of just one human would be doing pretty well, well enough that we couldn't justify saying "judging emotion is a big problem for AI" unless we said the same thing about humans.
Having said that, I agree that it seems a hard problem for the foreseeable future.
> So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning?
That's a bit harsh, but I agree.
Words & music are both key to understanding a song's "mood". Only looking at words can completely strip the irony/sarcasm off a song (as can only paying attention to the arrangements and melody/harmony.)
How would such analysis even work on artists like Dylan?
I think you will find wildly varying perspectives on whether a given song is depressive or not. To my surprise, when I read the YouTube comments on Radiohead's track Nude, I found many of the commenters found the song depressing, because of the lyrics:
"Don't get any big ideas, they're not gonna happen."
But, as far as I can see, this is only depressing if, first, you expect all your big ideas to happen and, secondly, you associate a big idea happening with happiness. I mean, why would a big idea not happening be depressing, unless you believe this big idea will make you happy?
I quite like the reminder that big ideas usually don't happen, unless you put in an earnest effort. And that the expectation of happiness from a big idea is the only thing that can cause depression if it doesn't work out, which it often doesn't.
In my experience, big ideas come out of happiness, not the other way around.
Thom Yorke once said that after touring for OK Computer he was at a local pub and some guy stopped him and said, "No Surprises" is the most miserable song he'd ever heard and why had he written it? At the time Thom said he had to concur with that assessment of the song. I can't remember what interview this was from but I used to have the recording of it where he introduces the song this way.
> The title line: "fifteen steps and then a sheer drop"
This song is obviously about Thom Yorke's frustration with the Mario Party minigame "Shy Guy Says". Think about it, 'first you reel me out then you cut the string', 'won't take my eyes off the ball again' (ball, in this case, meaning shy guy's flags). It makes perfect sense.
They're missing In Rainbow's Disk 2, which includes the song "4 minute warning" which I'd consider the "most" depressing Radiohead song: https://www.youtube.com/watch?v=VtotpiSL700
I think this is an example of something machines can't quite do better than humans yet, though it's an interesting problem to solve (trying to analyze how people in general will feel about something).
Personally I'd give it to either How to Disappear Completely, or Exit Music (For a Film).
Exit Music might be their most bitter song, but I'd say Street Spirit or How to Disappear are the saddest.
I wonder how many people would have an immediate answer at the ready, and how many would have to ponder and classify ( assuming similar familiarity with the Radiohead catalogue)
“‘Street Spirit’ is our purest song, but I didn’t write it…. It wrote itself. We were just its messengers… Its biological catylysts. It’s core is a complete mystery to me… and (pause) you know, I wouldn’t ever try to write something that hopeless… All of our saddest songs have somewhere in them at least a glimmer of resolve… ‘Street Spirit’ has no resolve… It is the dark tunnel without the light at the end. It represents all tragic emotion that is so hurtful that the sound of that melody is its only definition. We all have a way of dealing with that song… It’s called detachment… Especially me.. I detach my emotional radar from that song, or I couldn’t play it… I’d crack. I’d break down on stage.. that’s why its lyrics are just a bunch of mini-stories or visual images as opposed to a cohesive explanation of its meaning… I used images set to the music that I thought would convey the emotional entirety of the lyric and music working together… That’s what’s meant by ‘all these things are one to swallow whole’.. I meant the emotional entirety, because I didn’t have it in me to articulate the emotion… (pause) I’d crack…. Our fans are braver than I to let that song penetrate them, or maybe they don’t realize what they’re listening to.. They don’t realize that ‘Street Spirit’ is about staring the fucking devil right in the eyes… and knowing, no matter what the hell you do, he’ll get the last laugh…and it’s real…and true. The devil really will get the last laugh in all cases without exception, and if I let myself think about that to long, I’d crack. I can’t believe we have fans that can deal emotionally with that song… That’s why I’m convinced that they don’t know what it’s about. It’s why we play it towards the end of our sets. It drains me, and it shakes me, and hurts like hell everytime I play it, looking out at thousands of people cheering and smiling, oblivious to the tragedy of it’s meaning, like when you’re going to have your dog put down and it’s wagging it’s tail on the way there. That’s what they all look like, and it breaks my heart.”
So many possibilities, but my mind went straight to Pyramid Song. I think there is no way to objectively answer something like this. Even if you get some kind of majority in favour of one particular answer, that doesn't make it "true".
> and then used the tidytext package to break the lyrics into words,
> eliminate common "stop words" like 'the' and 'a',
> and count the number with negative sentiment
Is 'not' a "stop word"? How did it count negations after "stop words" removed, i.e. "I am not forgotten". Positive, negative?
Would be interesting to see how it handled REM or Dubstar, both of whom have some really upbeat tunes, with really dark lyrics - "The One I Love", for example, with "a simple prop to occupy mind ", or "Not So Manic Now", a happy bouncy pop song about assault...
'Depressing' has a variety of facets made up of sadness, textual content/lyrics, and (at least for me) the placement of the title within its respective album context.
Anecdote: When I previewed Kid A via AudioGalaxy back then (it would not be released for another two weeks in my region) I thought the stylistic choices of 'Everything in its right place' were some kind of encoding error (quite frequent back then) so I trashed the whole album after spending hours on the download.
Imagine my surprise when I listened to the album from the physical album I bought a fortnight later :)
I got my first cassette tape player in 1982 and Prince's "1999" album to go with it. The lead track starts with distorted vocals, and I thought my player was malfunctioning. The intro is only about ten or fifteen seconds, so unlike you, I didn't trash the tape.
Closer to topic, listening to the song "1999" as a kid, I thought it was a cheerful party song. After Prince's death, I listened to the song as an adult and realized it's about nuclear annihilation.
While great to see, this also highlights just how far we are from where data science needs to be. The amount of code necessary to just pull down and reformat data before any real analytic work is done is always staggering; I see it in my work but it's even painful in a fun example like this. I know, 80% of time in prep, 20% in analysis, yadda yadda, but I hope we can turn some of this magical AI into making data transforms easier in the future so we can actually get to the analysis portion faster, along with more accessible "tidy" data to get to the analysis portion faster.
We'll always have to code our way out of some wacko data situations, but I look forward to (and dream of!) better and better libraries and approaches to getting prep prepped faster.
If you're going put each album on its own point in the X-axis, could you not just label those axis instead of using colour-coding and requiring us to look up the corresponding colour in a legend (that's not ordered in the same way).
Is there a way to have Spotify users create the mapping, e.g. by using song selection to detect their mood and finding the cluster of Radiohead songs that are only ever listened to in a depressive mood?
Note how listening to all suggestions mentioned in the HN comments but only to few or none of those recommended in the blog post is a good indicator of what you think about the power of ML/AI.
How on earth is Street Spirit not even on the list?
Yorke himself had called it "our purest, saddest song.", and went on to say:
I can’t believe we have fans that can deal emotionally with that song. That’s why I’m convinced that they don’t know what it’s about. It’s why we play it towards the end of our sets. It drains me, and it shakes me, and hurts like hell every time I play it, looking out at thousands of people cheering and smiling, oblivious to the tragedy of its meaning, like when you’re going to have your dog put down and it’s wagging its tail on the way there. That’s what they all look like, and it breaks my heart. I wish that song hadn’t picked us as its catalysts, and so I don’t claim it. It asks too much. I didn’t write that song.
True Love Waits is a sweet, quiet, sad little song about yearning and the fear of loss.
If, like me, you read that quote and wondered what exactly the song is about, then here is another fragment of the quote that might help:
They don't realize that 'Street Spirit' is about staring the fucking devil right in the eyes... and knowing, no matter what the hell you do, he'll get the last laugh...and it's real...and true. The devil really will get the last laugh in all cases without exception, and if I let myself think about that to long, I'd crack.
Irony and taunting really confounds sentiment analysis. I mean, street spirit ends in "immerse your soul in love" repeated 2X but is still incredibly bleak.
I didn't know that Spotify has an API. I see mention of a rate limit in the docs, but I can't find the actual limit. If I want to sort the tracks from albums released in a given year (say, ten to fifty out of a thousand-album collection) by Spotify popularity, will the limit get in the way?
Agreed. I'm not sure why the original link is necessary at all. It doesn't add anything and seems to have been written for no reason other than to write a blog post. The author refers to Radiohead's first album as 'Honey Pablo', so it doesn't look like he actually paid attention to the original article.
Trying to measure the sadness of lyrics with an algo is tantamount to the early scene in "Dead Poets' Society" where the students are supposed to measure the greatness of poetry by plotting it on a chart.
It's demonstrated in the data: one of the most depressing songs of their entire oeuvre is "Fitter, Happier" but because it's completely metaphorical (or sarcastic), the algo rates it as one of the happiest.
This will always be a big problem in AI. Measuring the literal meaning of words is easy. Measuring the semantics and context of words is a completely different matter.