
How Well Can You Hear Audio Quality? - sinak
http://www.npr.org/sections/therecord/2015/06/02/411473508/how-well-can-you-hear-audio-quality/
======
kazinator
This requires a proper, double-blind "A X B" test. Sample X is guaranteed to
be the same article as either A or B, which are different. You, the subject
are asked whether sample X is identical to A, or B.

Double-blind means that the person administering the test doesn't know whether
A is the compressed file and B the uncompressed or vice versa, and whether X
is a copy of A or B.

Without a properly conducted test, users who want there to exist a difference
between 320 kbps that they are able to hear will convince themselves that they
can hear it.

But likewise, those who believe they shouldn't be able to hear any difference
will convince themselves that they don't.

The A X B test eliminates this, because though you know that X is exactly the
same article as A or B, you don't know which, and you don't know which of A
and B is the higher fidelity one. If you believe that there is no audible
difference, you can at best randomly guess at the identity of X. If a large
number of subjects are tested and they all believe that A and B sound the
same, the distribution of their identifications of X will be consistent with a
random binary choice. If there are some subjects who can in fact tell the
difference, that will show up in the data as bias toward the correct
identification of X.

Those who think they have "golden ears" and convince themselves they can hear
something they in fact cannot are called out by this test procedure, by their
failure to actually identify X better than a random guess.

~~~
rectang
NPR botched the protocol in other ways. The samples have to be sync'd, and the
tester to be able to switch between them at will -- with a small crossfade to
prevent the glitch from throwing you off.

An old buddy of mine, Dan Dugan, hung two pink sheets at an Audio Engineering
Society conference on opposite walls to illustrate this point, asking people,
"Are they the same color?" It would be trivial to distinguish them side by
side, but accuracy is greatly reduced when they are distant in time or space.
Human perception is all about edges, not about absolute measurements.

~~~
Udik
I don't agree. What you propose would be the test "are you able to tell the
difference between the compressed and the uncompressed sound when you compare
them directly"? While this test is "are you able to appreciate the difference
between a compressed and an uncompressed piece of music"? If the two seconds
it takes you to switch from one version to the other are enough to prevent you
from spotting the difference, I'd say that whatever difference there is is
likely to be irrelevant to the listening experience.

------
bd
This was surprising. With all that poking fun of audiophiles, I expected there
would not be much of a difference.

I got 5 out of 6 correct, and the one I missed was pretty near miss (I picked
at random between 320 kbps and uncompressed sample). And these were quite
clear choices, many times I just needed few seconds: 128 kbps sounded worse
every single time, 320 kbps vs uncompressed was a bit harder, but still pretty
noticeable if I paid attention.

It wouldn't probably make a big practical difference for a typical "background
noise" listening, but it may have impact if you just want to sit back, relax
and focus on music (lower bitrates for me sounded "muddled", losing details in
high frequencies).

BTW I'm no audiophile, no special audio gear, just cheap (but decent) 9 EUR
in-ear headphones plugged into a notebook.

~~~
Joeboy
> With all that poking fun of audiophiles, I expected

> there would not be much of a difference.

The fun poking is mostly about things like 192kHz sample rates and thousand
dollar power cables. It's uncontroversial that people can hear mp3 artifacts.

~~~
72deluxe
192kHz sample rate are unnecessary. In truth, 96kHz is common for live too (as
it is half the latency of a 48kHz system).

48kHz and 24bits is what most audio is recorded at nowadays, then downsampled
with aliasing for CD quality audio.

~~~
Joeboy
> 96kHz is common for live too (as it is half the latency of a 48kHz system).

I don't really understand this, but I guess what it means is that the first
sample of a digital signal gets from some place to some other place a ninety-
six thousandth of a second more quickly. I'm not sure why that would matter.
It's the time it takes sound to travel 4mm, which seems inconsequential, and
the overall shape will be the same phase.

In many cases (like, if there's a computer involved anywhere), a high sample
rate means you need _higher_ latency to avoid the risk of underruns.

~~~
72deluxe
If you've got to send it from stage to the mixing console at the back and then
back to the stage or wherever the amps powering the line array is, then
latency becomes more important? What if the console is half a km away?

Also, if the sound processors are performing calculations in the idle time
between samples (like calculating FIR filters or something like that) or in
the idle processor time remaining after processing the sample, a higher sample
rate will mean the calculation gets done faster (and is therefore audible
faster). Else you'd change a setting and wait to hear it (and it would be
noticeable perhaps), I guess?

~~~
Joeboy
> What if the console is half a km away?

Then a ninety-six thousandth of a second's worth of latency seems pretty
irrelevant? Even if you were decreasing the latency by using a higher sample
rate, which you aren't.

The signal coming out of a FIR filter will come out at the same time whatever
the sampling rate. I guess it's conceivable, if you have no buffering
whatsoever, that the very first sample will come out slightly quicker, but
that is honestly irrelevant. The overall signal will have the same timing at
either sample rate. Unless you've had to introduce more latency to cope with
the demands of the higher sample rate.

~~~
72deluxe
I meant that the FIR might be calculated between samples, so the higher the
clock the faster the calculation.

------
arh68
I did this yesterday; I picked 320 / WAV / WAV / 320 / 320 / 128\. I cannot
distinguish 320 from WAV, so I was mostly just trying to spot the 128 and pick
one of the other two. I found it a bit fatiguing to listen to so much new
music.

Also, it's hard to know if I'm listening to a WAV of some 808 drum loop or an
_actual_ recording of a snare/cymbal encoded. The classical piano seemed
easiest to spot the 128.

I listened through IEMs plugged into a Macbook Air in a quiet room and I
wouldn't say it was easy.

~~~
uulbiy
Similarly, I got 3 wavs and 3 320s. I could distinguish the 128s but not
between the other two. Please note that I did this wearing studio headphones
(shure srh840).

~~~
eludwig
Same here 3 of 6. I always spotted the 128, but the others were a coin toss.
This was on an iPad (chrome browser) with Bose over-ear phones.

------
jreed91
Want to quickly win? Have a slow connection and pick the one that loads the
slowest.

~~~
noobie
I knew my Third World citizenship would be of some use one day! haha

------
codeulike
I feel sorry for people who can tell the difference between high bitrate mp3
and wav because they then have to spend extra on headphones and pono players
and amps and whatnot just to elimate the perceived difference that the rest of
us are happily oblivious of.

~~~
hzhou321
It is not really about people IMHO but about their headphone and amplifier.
Better equipment require more expensive up-keeping, that is usual.

~~~
scosman
Exactly. Most people could hear it with pro equipment. It's a dangerous cycle.
Buy better gear, hear more, repeat.

------
JohnBooty
This test reinforced what I've found in the past:

1\. I _can_ reliably tell the difference between 128kbps mp3s and higher-
quality files

2\. I _can not_ tell the difference between 320kbps CBR mp3s and uncompressed
originals. (Or between ~256kbps VBR mp3s, 256kbps iTunes Plus mp4s, and
uncompressed originals... though those weren't a part of this test)

~~~
stordoff
About the same as what I've AXB'd in the past as well. Most tracks at ~256kbps
AAC are the same to me as the uncompressed version (some doesn't compress that
well so still arifact, but are rare). Around 128kbps is fine, but I can pick
out which is which. Lower than ~100kbps is awful - the arfiacts are instantly
obnoxious and I hate listening to it.

It doesn't really apply to streaming, but I keep my music as lossless files
despite that. It removes any questions of "Could this sound better?" and I can
transcode to a device-appropiate format without compounding the quality loss
(desktop has lossless files, laptop has a 256kbps copy, mobile devices have a
128kbps copy)

------
72deluxe
Very interesting! I was relieved that I managed to pick out the uncompressed
WAV in most of them even on MacBook speakers - I failed with Jay Z and
Coldplay as the telltale snare and cymbal artefacts are hidden under layers of
compression already (and the "snare" on Jay Z's song is a sample anyway). The
real tell-tale song was Suzanne Vega's a cappella song, where the MP3 versions
sound like she's got a terrible snotty cold.

Modern songs have much compression and hard limiting which makes everything
distorted (you can hear distortion on the Coldplay sample I think towards the
beginning, unless I am mistaken?). Other offenders for brick wall limiting are
most Chili Peppers albums, Paul McCartney's Memory Almost Full, and California
Breed's (sadly) one and only album. Jason Bonham's snare gets lost for 75% of
the album.

~~~
thesumofall
Interesting that you were able to distinguish the 320 MP3s from the
uncompressed files. Couldn't make out any differences at all on decent
headphones. Even the 128s were pretty good esp. for the more modern mainstream
music where the quality is questionable from the beginning :)

~~~
lgunsch
Haha, yeah. Multi-band compressed till the waveform is a bar-code.

~~~
lgunsch
Why was I downvoted? It's very much true.

~~~
72deluxe
It is indeed very true, with compressed samples put through more compression
and then limited. Look at the waveforms of some tracks and they'll be like
square waves.

Rush Vapor Trails is the worst offender, perhaps?

------
coldtea
For most people it's like:

In self-assured comments in forums? Very well.

In actual, properly conducted, A/B tests? Not so much.

------
Zigurd
I'm old enough to be losing auditory acuity, and I really despise golden-ears
equipment reviews, especially of speaker cable, but I find the qualitative
tests for compression technologies unsatisfying.

Taking a gang of schlubs out of a shopping mall and playing audio, video, or
still images to them and asking for an opinion could be OK in some contexts,
like "Can you tolerate us putting this low-bit-rate codec in your mobile
phone?" If the answer is "Huh, can't tell" then go ahead.

But is a JPG of an Ansel Adams print still a work of art if 90% of those same
schulbs can't tell the difference?

------
squeaky-clean
4/6\. I could definitely tell which was the 128/320/WAV for the 4 I got right,
but on the Katy Perry and Neil Young, all 3 were indistinguishable to me.

But still, this is when comparing small sections of songs directly to one
another, and being explicitly told there are quality differences. The 128
didn't sound bad, there's just some difference in transients and high
frequency content (the hi-hats on the Jay Z track were an immediate giveaway
for me).

Maybe a better test would be only having one audio track per song (instead of
3) and having to choose if it is 128/320/WAV. I wonder if anyone could
distinguish the difference there reliably.

I've gotten a new phone 3 times in the past year (I'm clumsy...), and each
time, I go a month or two with Spotify set to the default quality instead of
"Extreme Quality", and I never notice until I listen to an album I know
extremely well. WAV is great, but mp3 is still pretty good, even at 128kbps.

------
KeytarHero
Tidal has a similar test[1] (though only between 320 and lossless - no 128
track). Now, it's trying to sell you their high-quality streaming option, so I
would take the results with a bit of a grain of salt as there are a number of
ways their bias could have affected the results (for example, they could have
cherry-picked tracks that didn't encode to mp3 as nicely). However, the format
is much better than this one: the two play in sync, and you can seamlessly
switch between A&B as it plays. I found it much easier to find subtle
differences between the tracks that way over having to restart the track each
time.

But more importantly, that test format preloads the tracks. In this one, some
of the lossless tracks took several seconds to load for me, which completely
killed its double-blindness.

[1] [http://test.tidalhifi.com/](http://test.tidalhifi.com/)

------
uslic001
First time around only got 1 out of 6 but all the others I chose were the 320
wav using cheap over the ear headphones straight out of my desktop. The second
time I got 3 out of 6 correct using the same headphones but using a cheap DAC
and chose the 320 wav files for the other 3. I would love to do the test with
my higher quality DAC and my higher quality over the ear headphones but I do
not have access to them at this time. I rip all my CD's to FLAC at home and
then convert to 192kH MP3 to load on to my phone. The only time I listen to
the FLAC is when I use my higher quality DAC and higher quality headphones. I
have bought 3 albums from high definition websites and on only one (Van
Morrison Astral Weeks) can I tell the difference between the high definition
version and the CD version in blind testing. Given this I have not purchased
more high definition CD's.

------
hansjorg
Monty from Xiph/Mozilla wrote a great article about why 24-bit/192kHz format
(which is what Neil Young's Pono delivers) makes no sense:

[http://xiph.org/~xiphmont/demo/neil-
young.html](http://xiph.org/~xiphmont/demo/neil-young.html)

~~~
KeytarHero
Good article, but irrelevant. This is about lossless vs 320kbps mp3, not
24/192k vs 16/44.1k

~~~
hansjorg
Well, it's the next frontier in digital audio for those who claim to have
golden ears.

------
petercooper
The digital audio system here in the UK is terrible. They divide up the
available bandwidth so much to cram in more channels the stations sound worse
than FM (and they use MPEG2 audio too, which makes it worse). Last week, they
crammed in a special Eurovision channel and reduced several other channels
from 128kbps to 112kbps to cope. 1xtra (the station I listen to) suddenly
sounded so "dull" and slushy.. even just with a 16kbps drop! I had no idea
what was going on so asked on DigitalSpy and learnt the above.

Long story short, my takeaway is that perhaps it's not just about absolute
levels of quality, but also what effect minor effects in bitrate can have on
the underlying codec.

~~~
72deluxe
Is this with a satellite feed or Freeview? I have Sky (basically Freesat
because I won't want to pay Sky any money thanks) and the quality is abysmal,
even for video. The audio is compressed massively and LOUD all the time, and
the video is blocky as can be, with snowy or rainy scenes being
unintelligible. You can notice on some adverts that the small text at the
bottom is completely unreadable.

~~~
petercooper
No, DAB (for people overseas, DAB is a terrestrially transmitted digital audio
service - imagine FM radio but digital). If I'm not in the car, I'll listen
online instead (notably, BBC stations sound better streamed over 3G but this
has big UX implications when driving).

The quality on Sky used to be substantially better than DAB when I used it 10
years ago, but I don't know about it now. That said, I have Sky TV and the
picture quality is variable. Most of the channels I watch are fine and all HD
channels are very clear.

~~~
72deluxe
Informative, thank you. I am glad to know that Sky is decent quality on the HD
channels. Strangely they believe that you spend your entire life watching TV
and can't comprehend you doing anything else (like typing comments on HN at 9
in the evening.....)

A chap at work listens to DAB over some big Questeds and the glitching and
artefacting from Radio 2 (ugh I hate hearing that all day) is irritating (down
to poor reception). Those Questeds sound great if proper audio is put through
them though! It seems a waste to shove DAB Radio 2 and Jeremy Vine and his
argument "show" through them.

~~~
petercooper
Sadly the only way you'll get a better experience from your coworker's DAB
radio is to convince them to switch to Radio 3. It's the only DAB station in
the UK with a 192kbps bitrate (although only 256kbps is considered to be
better than FM under perfect conditions).

(Well, technically, DAB+ would make things a ton better as it uses AAC
instead, but progress is moving forward very slowly in rolling it out..)

------
gtrubetskoy
I think this experiment is slightly flawed, because what compression will do
depends on the original recording. For me, the difference was most apparent in
Susanne Vega and Katy Perry. Mozart and Neil Young were a complete shot in the
dark, I could hear a slight difference with Coldplay.

I suspect this very much depends on how the analog recording was digitzed in
the first place, if there was an analog recording to begin with. A sample from
a CD is not the same as one made from a vinyl or a master tape.

Bottom line - there definitely is a difference, but in some cases it's hard to
tell.

------
istvan__
I can hear the difference between mp3(320) and flac/alac easily on my stereo
both in the car and in my room. The funny part is that you can easily
recognize mp3 compression based on high it trims the high frequencies and the
high hats are affected by it quite a bit. You can try different lossy formats
but i guess it is just damage control. The only reason I use alac and not flac
is because iPods only work with alac (and it is limited to 16bit, 44100Hz) and
my car only supports iPods :(

~~~
imaginenore
Or you think you hear the difference because you actually know which files you
play. Do a double blind test and you will fail. Professional musicians fail at
320 vs original tests.

~~~
istvan__
Not to mention that I am more of a sound engineer who has a lot to do with
sound quality while musicians care less about that, I know several musicians
who could not setup even basic sound system. It is also pretty common that DJs
(and other types of musicians) require the roads to help them to achieve the
best sound of their music. The musician is in charge of the music, while the
sound engineers are in charge of the sound. Quiet different subjects.

------
virmundi
Apparently I prefer 128. My definition of better was crispness. I found the
uncompressed to be muddied, at least coming through a Bose Color Soundlink.
128 was bright and clear.

~~~
JohnBooty
I'm not one of those people who automatically trashes Bose equipment, but Bose
does a lot of processing/EQ/"coloration" to music to make it sound more
pleasing.

There is absolutely nothing wrong with preferring that! Nearly all consumer-
oriented speakers/headphones do this, because there are other models
specifically intended for monitoring use. However, it does make Bose equipment
poorly suited to engineering-type applications such as recording work,
comparing mp3 compression algorithms, etc.

~~~
virmundi
Good point. I do wonder if Bose washes out the middle.

------
ta92929
My ears suck :-( and I tell myself I'm an amateur musician too.

------
pingec
Similar test: [http://test.tidalhifi.com](http://test.tidalhifi.com)

------
adamc
I consistently picked 320 over WAV, fwiw. I could definitely hear the
difference but always preferred the 320.

------
scosman
With a dac and decent, but not pro, headphones I could pick out the WAV, but I
was always unsure.

