
Oboe: A C++ library for low latency audio in Android - el_duderino
https://android-developers.googleblog.com/2018/10/introducing-oboe-c-library-for-low.html
======
hyperpallium
> Oboe provides the lowest possible audio latency across the widest range of
> Android devices

The "lowest possible" on android might not be low enough for musical
instuments (<10-15ms). It's revealing that they _don 't_ reveal latency for
any devices at all. (If I sound cynical, this isn't the first low-latency
offering for android.) It's just "low" _for_ android.

funfact: consoles also have terrible audio latency. Rhythm games like
rockband/guitar hero don't actually need this, they need only relate it to the
audio playing (and also config compensation for input latency). The latency is
shocking if you use the mode where you free-"play" the instrument, and is why
actual musicians often a tough time with these games. The surprising thing is
that delayed sound effects from games in general aren't so jarring.

FWIW iphone audio latency also isn't good enough, despite being far better
than android. That's how bad android is.

EDIT 7ms claimed in screenshot at
[https://github.com/google/oboe/blob/master/samples/hello-
obo...](https://github.com/google/oboe/blob/master/samples/hello-
oboe/README.md) \- doesn't include input latency: _average latency of the
audio stream between data entering the stream and it being presented to the
audio device_

EDIT they know they need latency figures _" Publish a list of Android device
latency"_
[https://github.com/google/oboe/issues/235](https://github.com/google/oboe/issues/235)

~~~
sirwitti
Audio latency might not event be the main issue here: Touchscreen latency is
far worse which makes any input almost useless for someone trying to actually
play notes. Note that changing parameters of filters, synths,... is fine
because changing these does not require the same precision as playing.

Btw if you're looking for a subjective number for tolerable latency I'd say
~5ms. This could be output latency if you're playing notes via midi or round-
trip latency if you're processing incoming sound and output it again. In my
experience you don't get that acceptable latency on notebooks with the onboard
sound cards either :)

~~~
veli_joza
5ms is bit on the low side. If you're playing from guitar amplifier that's 4
meters from you, you get 12 ms latency just from sound propagation (and most
amps introduce additional latency). As extreme example of what the brain can
compensate, church organ players have to deal with 100ms+ latency and they
still manage to play Bach.

~~~
jcelerier
> church organ players have to deal with 100ms+ latency and they still manage
> to play Bach.

but they don't play Bach on pipe organs with a drummer

~~~
stalfs
No, they play it with a choir.

------
romwell
Does this mean we will _finally_ get decent musicmaking apps on Android?

A decade after iOS, I guess it comes in from the "better late than never
department", but I am happy. Apple seems to have lost its focus on the
creative usage of their devices, and it has somewhat stagnated (compared to
the initial explosion), plus iOS apps tend to be pricey.

Apple's decision to not include the headphone jack effectively puts the kibosh
on their realtime audio / musicmaking apps. The latency on AirPods is
atrocious. You press a key on a virtual keyboard, and count to three before
you hear a sound.

Yes, this is probably solved by a dongle. Eh, I'd rather carry my Yamaha
Reface DX around than think about that -- and the developers are going to have
a FunTime(tm) explaining this to the users. (The sole reason musicians love
iDevices so much is the JustWorks(tm) aspect - well, that just went out the
window).

This Android API just might start a new wave. The video appeals to the synth
nerd in me. The gear, the T-Shirt, the code samples doing the right thing.

I am looking forward to what this will bring us.

~~~
puzzle
Raph Levien worked for years on improving Android audio latency. I had a great
chat with him a long ago about emulating the DX7, APIs and so on. I thought
they already managed to get numbers to reasonable levels (after adding tighter
requirements to the Android certification suite, among others) and even gave a
talk at I/O. I'm surprised he's not involved in this, but then this is C++;
he's been really into Rust for a while and perhaps back to working on
typography alone — assuming he's still at Google, on the Android team.

~~~
raphlinus
Wrong on both counts. However, I'm very much back in the audio synthesis game,
and will give a talk at next month's SF Rust Meetup on the topic.

~~~
wwweston
Would love to hear you're planning to put up video and/or slides -- that way I
don't have to drive up from Los Angeles. Or get around the waitlist of 30
people for a full event. :)

~~~
raphlinus
Yes to both, and also plan to write blogs about the content, and documentation
for the library as it matures. I'm gratified to see the interest!

------
pjmlp
Typical Android NDK library from Google, instead of being added to the NDK,
Android Studio and respective documentation, we get a github repository and
"good luck" pat in the back.

And maybe if they have enough buffer on their 20% we get some of the reported
issues fixed.

Like cdep, Play Services for C++, FPL libraries...

------
gugagore
The proliferation of apps for musicians on iOS (drums, guitar amps) is
explained by the iOS's longstanding API for low-latency audio. I hope we can
expect the same kinds of apps, with usable latency, for Android now too.

~~~
veli_joza
I contacted one of Oboe authors and I reached conclusion that this doesn't
achieve anything you couldn't already get with openal-soft lib. You can expect
some 20ms of audio delay, plus 16-33ms of touchscreen latency .

If you are curious about latency, check out my android app "Hexpress" (on play
store) that gives you drums and other instruments to play with. I measured
roughly 50 ms of tap-to-sound delay, not consistent across devices. That lands
it within 'cute audio toy' territory, which is why I stopped investing time.

~~~
f_m
That's too bad: on my OnePlus 6, the latencies for your app are great. It's
the first time I'm drumming in an Android app, and am not bothered by latency.
Thank you for having the source on Github, I'll amuse myself this weekend
checking the code.

------
evozer
I looked at the implementation of the openStream() function:

    
    
      AudioStream *AudioStreamBuilder::build() {
          AudioStream *stream = nullptr;
          if (mAudioApi == AudioApi::AAudio && isAAudioSupported()) {
              stream = new AudioStreamAAudio(*this);  
    
          // If unspecified, only use AAudio if recommended.
          } else if (mAudioApi == AudioApi::Unspecified && isAAudioRecommended()) {
              stream = new AudioStreamAAudio(*this);
          } else {
              if (getDirection() == oboe::Direction::Output) {
                  stream = new AudioOutputStreamOpenSLES(*this);
              } else if (getDirection() == oboe::Direction::Input) {
                  stream = new AudioInputStreamOpenSLES(*this);
              }
          }
          return stream;
      }  
    
      Result AudioStreamBuilder::openStream(AudioStream **streamPP) {
          if (streamPP == nullptr) {
              return Result::ErrorNull;
          }
          *streamPP = nullptr;
          AudioStream *streamP = build();
          if (streamP == nullptr) {
              return Result::ErrorNull;
          }
          Result result = streamP->open(); // TODO review API
          if (result == Result::OK) {
              *streamPP = streamP;
          }
          return result;
      }
    

Doesn't this leak memory if streamP->open() fails? I'm also surprised they are
using new instead of unique_ptr, especially since one of their listed benefits
is "Convenient C++ API (uses the C++11 standard) ".

~~~
lossolo
Yeah, seems like memory leak to me too. streamP is leaked if open fails.

And this in modern C++ in year 2018:

if (result != Result::OK) { goto error2; }

------
Wehrdo
I used Oboe (and its predecessor, Howie) to make a self-balancing robot on
Android. We were able to use the headphone jack (RIP) as a low-latency output
to send serial data to an Arduino[0].

Android Studio has also finally improved NDK support enough that writing large
chunks of your app in C++ isn't painful anymore.

[0]:
[https://davidawehr.com/blog/audioserial/](https://davidawehr.com/blog/audioserial/)

------
iandcoleman
Unfortunately, this is unlikely to improve latency on devices which already
don't have 'good latency'. Reviewing the linked doc, this is a 'high level'
API whos goal is possibly more about streamlining the API across multiple
recent releases, while I'm sure, addressing latency as best it can. The
problem is that this API still sits on top of other lower (and already
existing api's) aaudio for current releases and opensles for older. A large
part of Android audio latency is down to the HAL layer that each vendor
provides. Andoid as a framework offers support for varying degrees of latency,
but it's up to the vendor supplying the audio HAL to support this. This is
exposed through the low_latency prop check (or 'professional' in recent
releases). If the vendor HAL doesn't support 'low latency' through reduced
buffer sizes and efficient (or no) locking etc, there's nothing a high level
API like this can do, other than bypass the HAL and talk to ALSA directly, but
this breaks the whole architecture.

------
The_rationalist
It may be interesting to recall that collabora did port the linux professional
JACK audio server to Android and it has _far_ less latency than audioflinger
(btw in french audioflinger phonetically mean to destroy the audio (quality),
which is kinda appropriate).

Sadly no OEM ever cared to my knowledge to integrate this work in their ROMs.
I also openened an issue on the android bugtracker to feature request
replacing audioflinger with alsa/pulseaudio. They never answered nor closed
the issue but I haven't checked since 2 years.

Edit: here is the android feature request
[https://issuetracker.google.com/issues/37073168](https://issuetracker.google.com/issues/37073168)
Maybe if you comment/upvote it it may gain traction.

Digression: If you are interested in audio latency, I recall that Firefox has
better webaudio performance than Chrome.

------
saagarjha
Unrelated, but there's a code example on that website:

    
    
        AudioStreamBuilder builder;
        AudioStream *stream = nullptr;
        Result result = builder.openStream(&stream);
    

Why even bother initializing stream to nullptr if it's going to be overwritten
anyways by AudioStreamBuilder::openStream?

~~~
FigBug
You'll probably get a warning about using an uninitialized variable if you
don't. Compiler probably can't tell the variable will only be written and not
read.

It's also just good practice to initialize your variables.

------
nshgraph
Unfortunately this won’t help with the terrible hardware level latency on many
Android devices.

~~~
iandcoleman
It's not the hardware per see, ALSA could certainly be improved by vendors,
it's the intermediate layers that cause the problem. Android needs to enforce
low latency upon vendors, and not continue to give them the easy option of
avoiding it.

------
xtrapolate
Audio latency issues on PCs are largely alleviated thanks to better softwate
(device drivers that shorten the path to/from the hardware - such as ASIO4ALL)
combined with better hardware (soundcards interfacing over
PCI-E/Firewire/Thunderbolt/USB) at affordable prices. Plenty of audio cards
today will offload computation off the CPU via discrete DSPs.

On Android devices (and smartphones in general) - hardware would be the
limiting factor.

~~~
AstralStorm
Not even close. You can have 1 millisecond latencies with a well configured
kernel (PREEMPT), directly talking to ALSA. I'm talking both Qualcomm and
Exynos CPUs.

It just takes a SCHED_FIFO task with forced CPU affinity. Android does not
make it easy to get one.

There are some other hardware issues, like audio input and output using
separate clocks on Qualcomms, but that's at most one extra buffer.

Speaking of this, you can have sub-millisecond latencies on Beagleboard. These
devices in phones are vastly more powerful.

~~~
xtrapolate
> "It just takes a SCHED_FIFO task with forced CPU affinity. Android does not
> make it easy to get one."

Are you aware of any non-RTOS that "makes it easy to get one"?

> "Speaking of this, you can have sub-millisecond latencies on Beagleboard."

I did not make myself clear, so I'm taking the blame here.

I'm mostly interested in the use case. If it's just capturing audio alone,
this number makes sense, and no fancy hardware is necessary.

The moment we're introducing some-kind of processing, or even logging the
stream to disc, buffering becomes necessary and latency is introduced. Assume
audio stream read by a "user-mode" service, then redirected out through
headphones - are we still talking sub-millisecond latencies?

~~~
endorphone
Audio is an _extraordinarily_ low demand task, and hardware is not the issue,
and hasn't been for a couple of decades (even DSP offloading has virtually
disappeared). A smartphone has many, many magnitudes of excess performance to
handle extremely low latency audio. The iPhone has had a 7ms latency for many
generations, and the 7ms is not some intrinsic limit but is simply a decent
balance between low power usage and ensuring no glitches.

The problem on some platforms is one of architectural choices and
prioritization. On Android we know they started with the already arguably poor
latency Linux audio foundation of ASLA, then layered on and layered on
(flingers and HLAs and user-mode transitions), each layer adding its own ring
buffers.

Even on Windows, on the fastest PC known to man, audio has generally poor
latency (because it's architectural) which is why audio software makers have
their own hardware->application drivers (ASIO).

Low latency audio was not important to the project, and they dug themselves in
so deep that for many years we've been hearing recurring "We've finally solved
that latency issue" claims.

------
telesilla
Wonderful! Latency has been a massive blocker for audio on Android - this
makes me want to break open the SDKs again and tinker.

------
fulafel
So what are the latency numbers for some common devices? (apart from the 7 ms
visible in the hello oboe screenshot)

~~~
iandcoleman
'low_latency' used to be sub 20ms, however 'professional' was introduced a
while ago to replace this, with 'low_latency' becoming 40ms

~~~
iandcoleman
Sorry, Actually, it's 45ms. From the docs: android.hardware.audio.low_latency
indicates a continuous output latency of 45 ms or less.
android.hardware.audio.pro indicates a continuous round-trip latency of 20 ms
or less

------
resters
Can anyone recommend the best way to do low latency audio on windows and osx?

~~~
adzm
For Windows, ASIO! [http://www.asio4all.org/](http://www.asio4all.org/)

~~~
raphlinus
I think wasapi has more or less superseded asio and has more support from
Microsoft.

If you're programming in Rust, you can use the cpal crate, which wraps all
these. I'm not sure cpal has astonishingly good performance in all cases, but
I plan to work with the authors. It doesn't have an Android back-end yet, but
such a thing should be possible.

~~~
resters
Excellent. Trying to do sine wave generation, on and off at low latency.

------
mohitmun
Would this make it possible to build active noise cancelling using software?

~~~
taneq
I wouldn't think so. If your cancelling signal is half a period out, you'll
double the volume instead of cancelling it. That suggests the required latency
is on the order of microseconds, not milliseconds.

------
adamnemecek
You guys should also check out AudioKit
[https://github.com/AudioKit/AudioKit](https://github.com/AudioKit/AudioKit)

~~~
izacus
I don't see an Android version mentioned there?

~~~
aikah
Because there is none, this is for Apple devices. For android you have that
SDK, though I didn't check the licensing terms:

[https://superpowered.com](https://superpowered.com)

------
alexforster
The title should probably mention "for Android". It's misleading without that
context.

~~~
sctb
Thanks, updated!

