
Apple's CVDisplayLink Doesn't Link to Your Display - jordwalke
http://thume.ca/2017/12/09/cvdisplaylink-doesnt-link-to-your-display/
======
jchb
If you use OpenGL or Metal with triple buffering (the blog post mentions
MTLView which is a Metal view) I believe this doesn't matter. You'll only be
using the CVDisplayLink timer events to generate frames spaced at a even
interval - which is equal to the display vsync interval, depending how you
configure CVDisplayLink. Then, the actual presentation of those frames _will_
happen in sync with the actual display vsync.

See Metal Triple Buffering
([https://developer.apple.com/library/content/documentation/3D...](https://developer.apple.com/library/content/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/TripleBuffering.html#//apple_ref/doc/uid/TP40016642-CH5-SW1))
and MTLCommandBuffer presentDrawable.

For OpenGL there are similar mechanisms, eg. NSOpenGLCPSwapInterval.

~~~
wtallis
If you're aiming for the lowest latency possible, you will still want to have
access to the actual display refresh timing information. But these days with
variable refresh rate monitors growing in popularity, this gets pretty
complicated.

~~~
izacus
Does Apple even support any of the variable refresh rate standards?

~~~
tvararu
Not sure about Apple, but on a hackintosh you can install Nvidia Geforce
drivers that allow you to turn on GSync.

~~~
paxswill
Which is kinda buggy last I checked, crashing if more than one display was
attached.

------
trishume
tfw you do a bunch of research you're proud of and write an article that
reaches the HN front page, and then someone chimes in on Twitter that actually
the research is flawed and only applies to multiple displays, and you have to
update the post about how wrong you were. :/

I'm just sad about all the people that already read it and won't see the
update so will go away with incorrect knowledge.

~~~
pcwalton
Interesting. So if I'm reading this properly, CVDisplayLink _is_ still a timer
--it's just that it tries to synchronize itself to the "vblTime" if there's
only one of those to synchronize to.

If I'm not mistaken, then, it would seem you can do the same thing yourself by
just blocking on CGLFlushDrawable() and setting up a timer on the refresh rate
as soon as it returns. (I guess that's similar to what mstange suggested in
the comments.) At least CVDisplayLink tweaks timer parameters in order to get
the timer to fire more reliably.

------
Someone
The documentation
([https://developer.apple.com/library/content/documentation/Gr...](https://developer.apple.com/library/content/documentation/GraphicsImaging/Conceptual/CoreVideo/CVProg_Concepts/CVProg_Concepts.html))
doesn’t mention vertical refresh interrupts.

It also says _”In the past, synchronizing your video frames with the display’s
refresh rate was often a problem, especially if you also had audio. You could
only make simple guesses for when to output a frame […] which didn’t take into
account possible latency from user interactions, CPU loading, window
compositing and so on. The Core Video display link can make intelligent
estimates for when a frame needs to be output, based on display type and
latencies. […] If for some reason the processing takes longer than expected
(that is, the display link’s estimate is off), the video graphics card can
still drop frames or otherwise compensate for the timing error as
necessary.”_.

So, disassembling the code wasn’t strictly necessary.

Combining this with this article, it seems they try to call your program as
late as possible, possibly to decrease latency between the time you have
influence on what is displayed and the time it is displayed. Could that
_especially if you also had audio_ be the reason they do that, as it might
make it easier to keep audio in sync with video?

~~~
jmull
> it seems they try to call your program as late as possible

Right. The point of the API is to trigger rendering as late as possible before
the next frame swap, not immmediarely after the last frame swap, as the author
expected.

To do so, it _has to estimate_ how long your app’s callback will take to
render a frame. (E.g. if it expects your callback to render a frame in 10ms it
might call it 11ms before the next frame swap.)

It seems the heuristic it uses to estimate the duration of the callback
execution is assuming a fairly constant duration. The documentation seemed to
suggest that. But as the author has inconsistent rendering duration.

So, e.g., if it estimates 10ms and therefore calls back 11ms before the frame
swap but you end up taking 20ms, you’re going to drop a frame. If it then
adjusts and calls back 21ms before, but you render in 1ms, you could render
two frames in the space of one displayed frame.

Anyway, just thinking through it aloud.

I think the author may not want to use this API for his purpose, though
perhaps by adjusting his rendering so that it always takes the “max” rendering
time might let its estimates be correct and then things would smooth out.

~~~
trishume
Author here, that would make sense, except it doesn't appear to be doing that.
I also don't see anything in the docs suggesting that it would. My test app
takes 1ms to render yet, and the callback returns within 100us, but it's
consistently calling me 7ms before vsync.

I'm also not actually working on anything that uses this right now, I was just
curious. I'm just warning about the dangers if you have inconsistent rendering
time.

------
adamnemecek
There’s quite a few approaches, it’s hard to figure out which one is
appropriate for this one. I think that enabling setNeedsDisplay and
dispatching commands asynchronously might do the trick? With asynchronous
dispatch I’ve actually been getting 120 FPS. But it’s very much case by case
basis and I might not be understanding the use case.

IIRC this project Gets to 120 when you drag your mouse
[https://github.com/jtbandes/metalbrot-
playground](https://github.com/jtbandes/metalbrot-playground)

~~~
trishume
I have a lot of avenues to test but unfortunately testing each one requires
setting up a sample app with signposts and recording a bunch of runs in
Instruments, so there's a bit of work to do if I want to figure out the right
way.

And do you have a 120fps+ monitor? If not I'm not sure what you mean by 120fps
unless it's broken or not vsynced at all and painting twice per frame. If you
have a proper way of waiting on vsync 120fps is just a waste of power with no
benefits and you don't want that.

~~~
adamnemecek
Haha welcome to gpu programming. You should hit up Warren Moore on Twitter
(@warrenm), he’s a real life savior.

Re: 120fps, I questioned it myself but the difference is perceivable. Idk if
something is broken or not I was confused too. Re power consumption, that’s
why you enable setneedsdisplay.

~~~
trishume
Maybe the reason that 120fps is perceivably better is that your draw calls
don't line up with vsync, so doubling the number of frames eliminates the
dropped frame effect in @jordwalke's diagram!

And thanks for the Twitter recommendation, I followed him.

------
mrpippy
Bring back VBL tasks.

[http://mirror.informatimago.com/next/developer.apple.com/doc...](http://mirror.informatimago.com/next/developer.apple.com/documentation/mac/Processes/Processes-74.html)

~~~
revelation
I like the Linux DRM atomic interface, they recently added native fences so
you can pass in a synchronization object from the GPU for when it's done
rendering and the flip can proceed and you get back a fence that is signalled
once the flip is done. Basically all asynchronous now but it's still synced to
the hardware flip, no timer bullshit.

IIRC it's somewhat inspired by Android SurfaceFlinger.

~~~
exikyut
Hmm. I wouldn't mind finding out how that actually works.

One question though. Is this driver-dependent in such a way that it only works
on newer hardware? For example on the 10+-year old machine I'm using right
now, could I play with this?

~~~
simcop2387
Should work with anything that uses the open source drm drivers as I
understand it.

~~~
exikyut
Interesting. Thanks.

------
danra
Wow, this is bad. If the author is reading this, is there any chance this is a
regression in recent OSes? I remember a WWDC video from quite a few years ago
showing exactly why it’s important to use CVDisplayLink which is linked to
when your display needs new frames, showing a similar illustration to the one
shown above about how otherwise you lose frames.

~~~
lobster_johnson
It's not bad, it's by design:
[https://news.ycombinator.com/item?id=15890910](https://news.ycombinator.com/item?id=15890910).

------
stmw
Interesting turnabout from classic MacOS, where the VSync interrupt was both
synced and really important. It was the way you got lots of non-graphical
processing done, too - lots of event timers & the like went in there.

------
TwoBit
You can synchronize to actual vsync on Windows.

~~~
isatty
Very helpful, thank you.

