
Ray Tracing with Metal [video] - pjmlp
https://developer.apple.com/videos/play/wwdc2019/613/
======
Impossible
It's a shame that Hackernews posts about graphics API features are so often
derailed by discussions about Vulkan adoption. I'd be fine with the discussion
if it was in context (a critique of Metal or even Vulkan) but threads about
new Metal API features and even threads about WebGL and WebGPU are
increasingly filled with the same fly-by responses about Apple's lack of
support for Vulkan from people that have never used the API and are not
graphics engineers. It would be much more interesting to have discussions
about the merits of different approaches to raytracing in Metal vs. DXR vs.
Vulkan than read the same tired half baked concerns about the existence of
Metal, although it's a good chance for me to give Pjmpl more karma I guess. No
one complains that Apple doesn't implement the entire Android API for iOS, why
is it different with Metal?

Edit: The talk isn't even about new API features or raytracing hardware. The
only reason it's "with Metal" is because it's a WWDC talk, it could be called
"implementing GPU raytracing with compute shaders" and apply to Vulkan,
DirectX, OpenGL or any other modern graphics API. Another good talk along
these lines is this one by Matt Swobda
([https://youtu.be/ZrFkKrw1uGI](https://youtu.be/ZrFkKrw1uGI)).

~~~
Reelin
> why is it different with Metal?

Because the GPU ecosystem has historically been fractured. Vulkan looks like
it has the potential to _finally_ end that, but Apple has a walled garden and
doesn't seem to be on board.

But I agree, such things aren't all that relevant to the article being
discussed here.

~~~
Impossible
Have they? OpenGL was really the only option on most platforms for years. Even
Windows has always supported OpenGL even if driver support from various
vendors wasn't always good and official Microsoft support lapsed. Consoles
have always used propiertary APIs, and before someone brings up PS3 and PS4
OpenGL implementations they are wrappers over low level APIs, so if they count
then MacOS and iOS do support Vulkan via MoltenVK (there is even an official
SDK on LunarG). If anything we've seen more fragmentation in the last 5-6
years due to the brief introduction of Mantle, followed by Metal and Vulkan.
Some of this was necessary because OpenGL and DirectX development had
stagnanted.

~~~
Reelin
> we've seen more fragmentation in the last 5-6 years due to the brief
> introduction of Mantle

I thought Mantle was what became Vulkan?

> Consoles have always used propiertary APIs

I certainly don't expect console manufacturers to adopt Metal, but at least
there's the _possibility_ of (eventual) unification with Vulkan.

> OpenGL ... support from various vendors wasn't always good and official
> Microsoft support lapsed

And Windows was the main PC gaming platform, so developers used DirectX
instead of OpenGL as a result.

On OSX, Metal became a thing; as a result, last time I checked OpenGL was
stuck back at version 4.1 (that's > 5 years behind everyone else).

I don't see Android, iOS, or Linux adopting DirectX. I don't see Windows or
Android adopting Metal. If you're hoping for unification, Vulkan seems like
quite literally the only possibility. Add to that Vulkan support from a
variety of hardware vendors other than AMD and Nvidia as well as officially
planned Vulkan-OpenCL interop and I'm actually somewhat optimistic about
things.

~~~
pathartl
To add to this, I can see Microsoft adopting Vulkan. I don't understand why
Apple has taken this approach. Well, I understand why _Apple_ did, but I feel
like most companies in their position wouldn't invent a new API.

------
Abishek_Muthian
One application which has impressed me with metal implementation even on low
end HW (intel HD 620) is iterm2[1] terminal emulator.

IMO, it is a perfect example of how normal screen drawing if made using HW
accelerated API could make a huge difference. This is especially true with low
end hardware e.g. if Raspberry Pi (Broadcom/Videocore) had implicit support
for OpenGL from the start by default like Apple has done with Metal, there
could have been a decent desktop experience which is not possible with current
SW rendering on low power CPU.

GPU rendering talk is getting lost with discussions about High end graphics
cards or GPGPU computation, but I think Metal has brought back the focus to
general screen drawing and judging from how developers like George Nachman
(iTerm2) was able to take advantage of it; Apple seems to have done good with
it.

[1]:[https://gitlab.com/gnachman/iterm2/wikis/Metal-
Renderer](https://gitlab.com/gnachman/iterm2/wikis/Metal-Renderer)

~~~
Teknoman117
If you've ever played with Wayland on a raspberry pi, it's an amazing
experience. It doesn't feel slow anymore. Now, apps still run slowly if they
are doing anything particularly compute intensive (chromium), but it's very
"responsive".

~~~
Abishek_Muthian
Yes, these are my experience. Please suggest if I had missed something which
could provide a better experience.

My configuration to minimize variables which diminish Pi desktop experience -
USB SSD boot, Zswap, Btrfs on Arch Linux 64bit (for better support for the
latter 2).

1\. Enabling VC4 driver at boot, enables OpenGL support to Pi.

2\. For Wayland compositor, I used Enlightenment (Gnome, GDM didn't launch). I
enabled HW rendering via OpenGL for all default rendering. Basic windowing
experience is fast, as expected from HW rendering. Few default applications
like terminal (Terminology) if used carefully(not resizing windows, not
dragging etc.) works. But any other applications which use OpenGL like
screenshot (or) applications which use XWayland chromium, firefox failed to
launch. So, my conclusion is that this setup works only for basic windowing
i.e. file transfers and careful use of console.

Perhaps Wayland via SW rendering (likely the one mentioned by you) could
provide a crash free experience, but that defeats the purpose of getting
general screen drawing HW accelerated (to feel responsive)?

------
shmerl
What's the story with Apple still refusing to support Vulkan natively?

~~~
AHTERIX5000
They don't need to. Also Metal isn't really 1:1 comparable to Vulkan in terms
of abstraction level.

~~~
shmerl
Same way MS didn't need to support common HTML/JavaScript since they had
ActiveX?

If Apple would have acted properly, instead of their usual "eat our lock-in"
mentality, they could provide Vulkan support, and then build higher Metal-
style abstractions on top of it. But no, it's Apple for you. Eat Apple only
Metal or get lost.

~~~
dmitriid
Apple has all the right to not think about Vulkan and invest in their own
implementation.

\- First release of Metal is June 2014. The Khronos Group began a project to
create a next generation graphics API in July 2014. First release of Vulkan is
2016.

\- Metal is C++. Vulkan is C. Yes, that may matter

\- Metal is specifically optimised for a specific set of hardware with known
capabilities. Vulkan aims at everything under the Sun

\- Vulkan is headed by a consortium that wasn't known for its great decisions
in the past, and Apple might want a faster turnaround on features and
capabilities

\- It's just business

~~~
shmerl
All of that is not a justification for their sick lock-in.

\- It was explained above, that Apple knew well all along, that there was an
effort to make a common low level API. Well before they started Metal. So they
purposefully refused to participate.

\- C was a proper choice for Vulkan (short of choosing Rust let's say), since
it makes it easier to provide bindings from any language than let's say using
C++, making bindings to which is way more painful.

\- Optimization for hardware happens on low level compilation, so it's a lame
excuse not to support Vulkan.

\- Faster or not, Apple could observe the result (which was good) once it was
ready, and support it. They refused (because Apple).

\- Business or not, lock-in is a nasty attitude that's aimed at taxing
developers. There is no no need for development tools lock-in, it's a sick
anti-competitive methodology, for which Apple is quite infamous.

~~~
dagmx
When Metal was initially developed and even when it was released, Vulkan
wasn't even a proposed project yet. The only comparable thing was Mantle from
AMD (which later evolved into Vulkan). At the time NVidia were still saying
use OpenGL with AZDO techniques and on mobile, you were stuck with OpenGL ES,
with a multitude of issues.

Optimization of existing code happens at a lower level but if you can gear an
API for optimal use of your hardware and existing APIs, you make it easier for
developers to write code that will have a better shot at being optimized. Not
all code optimizes the same way.

Also what you view as taxing developers, can also be flipped around as being
able to make an API that is a lot easier for the developers on their platform.

And indeed, metal is probably the easiest of the modern graphics APIs to get
started with, with a very light learning curve even compared to OpenGL. All
while not sacrificing performance and expressiveness for more involved
development. Vulkan meanwhile remains very inaccessible to many developers
even with graphics backgrounds.

Let's not also forget that apple isn't an outlier for lockin, and actually are
quite open. OpenCL, Clang, webgpu, swift etc are all very open and cross
platform. Meanwhile even things like directx are locked to windows.

The real goal isn't lockin. It's providing the best API to developers to let
them get in quickly while also optimizing for their hardware. Lockin is an
unfortunate side effect but they're also completely succesful in meeting their
actual goals.

~~~
shmerl
_> The real goal isn't lockin._

That's an excuse, since the result is lock-in. I.e. if Apple so much wanted
Metal - let them have it. But how does that prevent them from supporting
Vulkan, for those who don't want to waste resources on duplicating work and
want to reuse their existing Vulkan codebase on Apple systems? But no, Apple
__forces __you to use Metal. So the claim that it 's for developers' benefit
won't fly. It's for Apple's benefit very clearly.

WebGPU is also a counter example, since Apple there try to prevent adoption of
common formats like SPIR-V.

When it comes to sabotaging interoperability, Apple are always among the
first.

~~~
dagmx
Apple does all their driver and API development in house. Supporting Vulkan
would mean a lot more effort on their part. Unlike windows and Linux where
support is handled by the GPU vendor instead.

Why support Vulkan, which is arguably harder to use and takes more resources
of their internal development teams, when they have an API that provides all
the benefits and is arguably a better API? Especially when the major off the
shelf engines already support metal. It's diminishing returns at this point.

Personally I'd be quite happy to have Vulkan support on mac, but if you take a
step back from your inalienable position that it's a conspiracy, there's
actually very sound reasoning behind it.

You may not agree with the decision, but you weaken your argument when you're
trying to force everything to fit a black and white narrative. The reality is
much more nuanced.

~~~
banachtarski
> Apple does all their driver and API development in house

Not OP but the games and film industry believe this here is the problem. The
additional effort to support common APIs is an artificial self-imposed problem
resulting in worse software.

~~~
dagmx
I'll definitely agree here that it's got its share of pros and cons, and
having worked in both realtime and film work, it is difficult.

But on the flip side, it does fit in with apples ability to develop very
tailored platforms and make very easy to use APIs.

So while I often find myself on the wrong side of the double edged sword, I do
also feel their decisions make sense as well. And honestly, having done
dx9-12, OpenGL and Vulkan, I won't argue against how easy to use metal is.

~~~
banachtarski
I'm sort of mixed on how easy Metal is. Having to support Objective-C and/or
Swift as part of the build toolchain is sort of a headache and Metal actually
isn't low-level enough for me to support a lot of the optimizations I can do
in DX12/Vulkan and especially doesn't come close to console APIs (which you'd
think should be possible given that they can specialize for their hardware).
My feeling is that for driver APIs, it's ok for the API to be lower level
since users (and other developers) generally won't need to deal with that
anyways.

------
legohead
Apple, please add a time multiplier to your video player. There's no need to
watch a demonstration like this at 1x time.

~~~
tuacker
Apple is using just a bog standard <video>-tag, so this should be a general
plea to browser vendors (including Apple/Safari) to add that functionality.

~~~
olliej
I vaguely recall the video element already having this functionality - I
remember an engineer working on it making the playback rate be negative. The
performance was not spectacular :)

~~~
pfranz
I'm not surprised and I don't think it's the fault of browsers or the
implementation. Delivery codecs have a lot of optimizations around playing
forward. There are a whole other class of video codecs designed around the
needs of video production; scrubbing, frame-by-frame, playing backwards, etc.

~~~
erikpukinskis
Frame-by-frame and slow-mo should be fundamentally easier than
scrubbing/backwards. Nothing about a forward oriented codec would make frame
advance or slow-mo harder. You still get a series of image buffers eventually.

~~~
pfranz
When going frame-by-frame you tend to go back and forth. You can preload and
cache, but that leans a lot more on your player and hardware. Yes, you do get
every frame but some frames may be very "muddy" because of compression. I'm
not saying it's impossible, it's just not an optimized use-case that's trivial
with other codecs.

Seeking is another example. It's a hard requirement for video editing, and
desirable for users but often compromised for more preferable things like
bandwidth and file size. For example, if the data-rate is variable, you don't
really know where 3minutes 20seconds into the file is without hunting around
(which is trivial if the data-rate is constant). MP3 supports a 1-100% lookup
table at the front of the file, but this isn't very granular for long files.
Even if you can find the spot in the file, you'll probably snap to an i-frame
unless you decode the whole segment.

I'm sure these things will get better as codecs mature and constraints on size
and processing power ease up.

------
mtgx
It's very disappointing that Apple hasn't adopted Vulkan. It's not like they
are completely against standards either. I believe they even joined the FIDO
Alliance recently.

With Apple adopting Vulkan, it may actually stand a chance against DirectX and
the majority of game developers might start developing for Vulkan first. And
Apple would benefit greatly from that, just like every other Vulkan supporter
would.

~~~
vondur
Does any OS natively support Vulcan? Or do you get the drivers from a third
party?

~~~
Avamander
Pretty sure AMD on Linux can do Vulkan with just second-party stuff.

~~~
ken
The "second party" is the user. I don't think any users are writing their own
Vulkan drivers.

~~~
Avamander
My mistake, thought about from the user perspective.

------
robert_foss
Apple doesn't sell any products with NVidia RTX hardware.

Yet still they feel the need to re-invent the wheel and write an API for it. I
guess they really want to maintain that competitive moat.

~~~
sleavey
Is this not something to do with the new Mac Pro?

~~~
robert_foss
The new Mac Pro only ships with AMD GPUs which do not yet support raytracing.

~~~
chrisseaton
Of course you can ray trace on AMD, or Intel, or an abacus. It’s not some
magic it’s just maths you can run on any general purpose computing device.

~~~
robert_foss
You can raytrace by hand too, but that doesn't change the fact that there is
no dedicated hardware support for it on any hardware shipped by Apple.

Raytracing is a not a problem that is well suited to GPUs since there are lots
of data dependencies, which means parallelism is severely limited.

~~~
chrisseaton
> that doesn't change the fact that there is no dedicated hardware support for
> it on any hardware shipped by Apple

Nobody in this thread or anywhere else said there was.

> Raytracing is a not a problem that is well suited to GPUs since there are
> lots of data dependencies, which means parallelism is severely limited.

...no it's the classic example of an embarrassingly parallel problem - that's
literally the term people use because it's so easy to parallelise. Each pixel
is completely independent with no data or control dependency between them and
they can be rendered in parallel.

~~~
Jasper_
Embarrassingly parallel in theory is different from embarrassingly parallel in
practice. One of the major differences is cache -- both instruction cache and
data cache. You can, in theory, run 100 rays at once on 100 different threads,
but since each ray goes to a completely different part of the scene, it will
be loading different geometry, running different shaders, and fetching
different textures.

Even in path tracers that aren't using the GPU, one of the relatively recent
huge improvements was "ray sorting", which tries to collect rays into bundles
that use the same shader and roughly the same area of textures to improve on
cache behavior. It brought huge speed increases.

One of the big limitations of NVIDIA's RTX right now is that it does not
support ray sorting.

~~~
0-_-0
Interesting tidbit about ray sorting: Imagination Technologies had a GPU
design in 2014 that had support for ray tracing, including a Coherency Engine:

[https://www.anandtech.com/show/7870/imagination-announces-
po...](https://www.anandtech.com/show/7870/imagination-announces-powervr-
wizard-gpu-family-rogue-learns-ray-tracing)

But there wasn't enough interest in GPU path tracing yet.

