
Android Audio Latency - kawera
http://www.androidpolice.com/2015/11/13/android-audio-latency-in-depth-its-getting-better-especially-with-the-nexus-5x-and-6p/
======
wawi
It is really an interesting aspect of the whole Android versus iOS "war", that
indeed Google did not up-front plan to make the OS actually useful for
realtime Audio/synthesis/processing; but instead, Android has had more of a
'data-center'-like design, wherein the kernel and its onboard assets is more
of a 'managed host' than it is 'realtime embedded' sort of system ..

That the Android designers are having to play catchup with this speaks a lot,
I think, of how significant design-decisions were influenced by the territory
from which the originated - the web, managed systems - and the realm which
they hoped to capture - the embedded system in your pocket. Android was not
designed up-front to be an amazing media system; iOS, was. Thus, iOS has more
synth-plugins and a vibrant, thriving Audio-Application eco-system (seriously,
its million-dollar) whereas Android has a lot of new players, but so far: not
such a great playground, really, for "Musicians".

In fact, while iOS has really impinged upon the Music-Instrument (Digital)
industry, Android has yet to make a dent.

So it is an interesting time to be having new solutions to what have been,
relatively, frustrating and troublesome aspects of Android realtime Audio, as
a developer.

~~~
agumonkey
Was iOS evolved from the iPod OS ? a lot ? a little ?

~~~
on_and_off
I have no idea of the internals of the iPod. As I understand it, iOS benefited
immensely from the work already done on OsX. I don't even know whether they
planned for it or just benefited from a mature module already in place.

~~~
sirkneeland
IIRC the very first iPods actually had licensed much of the under the hood
software.

I could be confusing them with Palm OS, which did the same for the kernel of
their first Pilots...

------
rcthompson
OK, what about the absurd latency for responding to hardware buttons on
headphones and the like? My experience across 3 different Android phones has
been that when I press the button on my headphones to pause whatever I'm
listening to, it will respond about half the time, and when it does respond,
the latency is randomly somewhere between 1 second and ten minutes. This is
consistent across several sets of headphones, both wired and Bluetooth, and
even true of the pause button that appears on the lock screen, which I'm
assuming uses the same mechanism to signal the app.

~~~
mikeash
I feel like nobody cares about responsiveness anymore. (If they ever did.) My
TV takes 20 seconds from pushing the power button before it's ready to show a
picture. My Bluetooth headset takes a couple of seconds after flipping the
power switch before it actually powers on. My iPhone (and I'd really expect
better from Apple!) often takes 5-10 seconds to respond to a press of the Home
button.

All over the place, apps and devices get faster and faster, and take longer
and longer to actually do what you tell them to do.

</rant>

~~~
BookmarkSaver
TVs are digital had come with all sorts of other features that require
processing. This used to be handled by "secretly" leaving them on all the
time. This would waste a decent amount of power, and as brands seek better
power ratings they actually turn off now. It's the price you pay for HD. I
also suspect that in general Bluetooth headsets always had a similar startup
latency.

~~~
mikeash
I don't buy it. Nothing about "processing" or HD means it has to take twenty
seconds to boot. It's the price you pay for TV programmers thinking a full-
sized Linux counts as an embedded OS.

~~~
simoncion
Given what I understand the power of the machine inside these TVs to be,
there's no reason a large-enough Linux shouldn't boot in a few seconds.

The manufacturer could get even smarter and use something like CRIU [0], along
with opportunistic hybrid suspend [1] to dramatically speed up start times.

[0] [http://criu.org/Main_Page](http://criu.org/Main_Page)

[1] Which would work like this: A little while after you power your TV off, it
suspends to RAM, and disk. If you power your TV on, the system TV restores its
state from RAM. If AC power goes out, when AC power is restored and stable,
the TV performs a background operation that uses the on-disk suspend image to
get back to its hybrid suspend state.

~~~
jschwartzi
The problem is that these features don't work out of the box. Any feature is
going to be judged by whether the increased time-to-market and R&D cost is
justified by additional sales. It's worth noting here that nobody buys TVs
based on how quickly they turn on.

~~~
simoncion
> The problem is that these features don't work out of the box.

What?

 _No_ feature works out of the box. Engineering effort has to be expended to
make something work when designing and building _any_ new product. Did you
maybe misphrase what you were trying to say?

> It's worth noting here that nobody buys TVs based on how quickly they turn
> on.

[citation needed]

A big part of _my_ criteria for purchasing a display device is device and
control responsiveness.

My non-technical family members who care about TV made "time between the time
you turn the thing on and time it becomes usable" their #3 priority when
purchasing their _second_ "smart TV". [0]

It's pretty clear that many (most?) techies are _really_ disappointed in how
shitty, slow, and poorly designed most "smart TVs" are.

[0] #1 Picture quality. #2 Overall value per dollar.

------
exDM69
Given the title, I would have liked to see some in-depth analysis on why the
latency is as bad as it is? Which parts of the software stack are causing it?

Even though the Linux kernel is less than ideal for audio, it can still
provide a latency low enough for music, ie. 10 ms or less.

This is an unbearable situation. There are lots of interesting music apps on
the iOS that I would like to use, but only a handful are available on Android.
And even if they are available, they are not necessarily usable.

~~~
rogerbinns
> Which parts of the software stack are causing it?

There is a diagram lower down in the article with the components (ALSA,
audioflinger etc).

[http://www.androidpolice.com/wp-
content/uploads/2015/11/nexu...](http://www.androidpolice.com/wp-
content/uploads/2015/11/nexus2cee_Android-Audio-Path-Latency-Superpowered-
Audio700px.gif)

~~~
exDM69
The infographic looked so much like an ad that I skipped over it.

So this is one of the "good" Android audio stacks, getting 36ms latency. But
half of that is "wasted" in user space, in AudioFlinger as well as the app
ring buffer (which seems pretty big to me). I'd like to see the comparison to
a "bad" Android device.

Doing some internet searches, I found someone who had plugged in PulseAudio to
an Android phone and got 20 ms latency instead of the 170+ ms latency from
AudioFlinger on the same device [0]. That's pretty huge.

Ideally, there should be no user space middleware component processing audio
in between, this could half the latency (according to the diagram), bringing
the latency to acceptable levels. But ALSA isn't really intended for routing
audio on the fly conveniently (e.g. when plugging/unplugging headphones), so
there is some room for improvement too.

Overall, the situation is pretty sad. Linux can do better than that but a
modern multimedia device (desktop or mobile) has several audio outputs and
hot-plugging, which makes the situation rather difficult using ALSA only. And
the userspace audio components just aren't great.

[http://arunraghavan.net/2012/01/pulseaudio-vs-
audioflinger-f...](http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinger-
fight/)

------
SwellJoe
The overall latency (not audio, but the responsiveness, in general) of my
Nexus 7 2013 got _much_ worse in Marshmallow. It now takes a second or two
after clicking anything to get a response. I'm not sure what happened, but
battery life also dropped immediately after the OTA update to 6.0. I don't
have a lot of confidence that audio latency has improved, given the overall
increased sluggishness of the device. I have a few music apps (Caustic,
several DJ things, a couple of classic style trackers), but haven't tried them
lately...partly because latency has always been such an issue that it wasn't
worth trying to do stuff on it.

Nonetheless, I avoid Apple products on principle, and also enjoy playing with
audio stuff (so much so that I started an audiovisual company a few years ago,
which I only recently extricated myself from, leaving it in the hands of one
of the co-founders), so I'm pleased to see the issue getting attention. I'd
love to be able to make real music on my tablet. I have a Gameboy that I use
for "on the train" music tinkering, because it has an immediacy that is lost
in most modern devices (not merely latency...just the general composition
process on something so simple it's impossible to get bogged down in bullshit
like selecting the perfect snare sample or something). It's be cool to be able
use the Nexus 7 for that purpose.

~~~
6stringmerc
Uh, while I think it's perfectly reasonable to like using a Gameboy for music
tinkering, that whole "immediacy" thing you cite about composition process is
essentially what got solved with Propellerhead's Figure. I've had it since the
first version and it's grown into my personal, portable drum/bass/synth
sketchpad and recommend it to anybody serious about composing on the go. Just
because you don't know what's out there doesn't mean it's not out there, for
what it's worth.

~~~
SwellJoe
Figure is an iPhone/iPad app. There is no Android version.

~~~
6stringmerc
I know, which means that until Android gets its garbage latency issues sorted
out, having principles a la "No iDevices" simply cuts off the nose to spite
the face.

~~~
SwellJoe
That's your position, which I have not made any assertion about. My position
is that I don't buy Apple products. That is not to say I believe Google is
vastly better, from an ethical perspective, but, Android is more open. Until I
can buy a reasonable Firefox phone in the US, Android will have to do.

~~~
6stringmerc
If I understand correctly Apple products can be "jailbroken" if you don't want
to buy a new product and primarily want to use it for actual artistic
endeavors. I was simply contradicting your assertion that music software isn't
up to par with quick composition, because it is, and to assert otherwise is
simply ignorant. In your case, willful, so that's just your own problem, as I
noted.

~~~
SwellJoe
Which part of my comment led you to the conclusion that I was ignorant? I
specifically mentioned that I'd ruled out Apple devices on other grounds,
which, I would think, would be sufficient evidence that I'm aware of the
capabilities of those devices. I also mentioned that I founded an audiovisual
company; one might conclude from that statement that I've recently worked in
the field professionally. One might draw some conclusions from that, as well.
I'm just not sure how one would draw the particular conclusions you've drawn,
based on the comment that I made.

------
vanous
Direct link for the superpowered.com latency testing app:
[http://superpowered.com/latency](http://superpowered.com/latency)

------
ZenoArrow
IMO the biggest reason to be optimistic that this will get this fixed is VR.
GearVR had to rely on its own audio stack, but if VR takes off in a big way
then I'm sure Google will want to support in stock Android, and that'll only
work if they re-architect the Android audio stack to support low latency
audio. This could end up fixing the issue for desktop Linux too.

------
on_and_off
If you want an in-depth explanation (AP mostly skims the surface) :
[https://www.youtube.com/watch?v=d3kfEeMZ65c](https://www.youtube.com/watch?v=d3kfEeMZ65c)
is a must-watch. Obviously, it does not cover the work accomplished these past
two years, but it gives a good idea of the challenges of audio latency on
Android

------
Polyphonie
You can watch this I/O session from a few years back:

[https://youtu.be/d3kfEeMZ65c](https://youtu.be/d3kfEeMZ65c)

It probably provides the best insight to the problem Google engineers were and
probably is still facing in regards to high performance audio on Android. Even
when the latency is brought down, there's still other audio related
issues/glitches (like buffer underruns).

Even now that Android is measurably better than it was previously, things like
buffer underruns can still affect the overall usability. Increasing the buffer
(to fix the underruns) render the device less suitable for such things as
using it with external MIDI controller. You can see from Superpowered's audio
latency measurements:

[http://superpowered.com/latency#cta](http://superpowered.com/latency#cta)

that the buffer for iOS is half that of the best Android (the dual-core Nexus
9)- 64 instead of 128 for Android.

I

------
ericmo
> (from the article) [...] It did unquestionably require more time to code in
> lots of low-level functionalities that were simply missing from Android's
> SDK — and even after all that work, the end result was still far from ideal.

That being said, it's not clear whether the improved latency on newer
platforms can be achieve without using Android NDK. I haven't used it, but I
know Android 4.0+ supports OpenSL through NDK, and I'm guessing that if you
want low latency you need OpenSL. By skimming through the docs, it looks like
that's the case, isn't that so?
[http://source.android.com/devices/audio/latency_app.html](http://source.android.com/devices/audio/latency_app.html)

So, maybe it does solve the latency issue, and I would gladly use NDK because
I like C++ stuff, but I think most developers wouldn't want to do it.

------
coupdejarnac
I have a virtual hearing aid app for iPhone(Enhanced Ears) that simply is not
possible right now with Android. I've had potential customers clamoring for an
Android port, but I don't want to develop an app that has crippled
performance. The audio coming in to the user's ears needs to match moving
lips.

------
kabdib
A few years ago I got into a project that involved pretty tight control of
audio latency, in a system with a fair amount of legacy.

"How hard can it be?"

At the end of a year, imagine someone who has spent a year in the Alaska
wilderness, strangled three grizzly bears with his bear hands, lived on
berries and nuts and fish caught in frigid glacial streams, and generally been
up and down the audio stack in the OS and a couple pieces of firmware,
involving audios APIs, a USB software stack that we re-wrote, a close
examination of USB hardware and how chipsets are fucked up, and how exactly
I2C microphones really work. My wife described me as "broken". It took a while
to recover.

How hard can it be? Isochronous, low-latency audio can be pretty hard.

------
vlaskovits
Hi all -- Patrick from Superpowered.com. We're working on an update to our
Android 10 ms Problem article with new data --- quick question: What's not
clear about Android latency? What questions would you like answered that you
haven't seen answered online?

------
mikecb
Given that the android tv devices are (imo) more important in this aspect than
a phone or tablet, I would have liked to see what latency was on things like
the nexus player, shield tv, sony tv (baked in), apple tv, and roku maybe. I
assume they would be similar, but perhaps they made some special modifications
that will later be incorporated into AOSP?

~~~
WalterSear
I don't understand why you think that round trip latency matters at all on tvs
and media consumption devices.

~~~
mikecb
I misunderstood that the issue was input-to-output latency, and was instead
thinking of audio/visual sync issues.

------
yeureka
I have been writing a game on my spare time that has an audio sequencer for
interactive music.

It currently runs on iOS and OSX.

Initially I thought about doing an Android version as the code is all in C++
and it would be easy to port, but if the latency is as bad as described here I
might not bother.

------
SCdF
> The fact of the matter is that sound (along with everything else we know of
> in the Universe) travels at a finite speed. In regular conditions, a sound
> wave propagates at around 340 meters (or 1100 feet) through the air in a
> single second. This means that even for plain old physical instruments,
> there exists an audio delay between the moment the instrument is played and
> the instant the sound reaches the musician's ear.

You heard it here first folks, in a few years Apple is going to come out with
the revolutionary "Cochlea sound", which has such a small delay your ears are
unable to perceive that there is any delay at all (while listening from 10-12
inches from your ears of course)!

It will be magical and delight us, and something we never realised we needed
until we were told we did. It will become the new standard and a household
name, and we'll all have to catch ourselves when we say it and try to use some
obnoxious generic term instead (LoALPS perhaps: Low Audio Latency Per Sound).

