
Feasibility of low-level GPU access on the Web - kvark
http://kvark.github.io/web/gpu/2018/02/10/low-level-gpu-web.html
======
c-smile
This situation reminds me days when we, at W3C HTML5 WG, were trying to sneak
SQL specification into HTML5 "umbrella". And one particular flavor of it - the
SQLite's SQL as it was defined at the moment of writing. Haven't got through
for many good reasons as we know.

As of GPU exposure to the Web ...

It makes sense only when we will have stable and unified GPU abstraction. As
for now DirectX.12, Vulcan and Metal are close but different.

Like the WebGL that is (more or less unified) OpenGL. But even that looks too
foreign to HTML/CSS/script - immediate mode rendering in architecturally
retained display model of web documents.

And conceptually: HTML5 umbrella is large but not infinite. 3D rendering is
too far from HTML "endless flat text tape" model.

I remember those <applet> days when browser was used as a component delivery
platform for stuff that does not fit into DOM and structured yet styled text.
That was conceptually the right way "to grasp the immensity".

These days, with WebAssembly, we have another incarnation of the <applet> idea
and I think GPU functionality belongs to it rather than to HTML. GLSL to be
expressed in WebAssembly bytecode terms but not in JS.

Web standards have to be backward compatible and I doubt that current still
ugly GPU paradigms will survive on the long run. Like tomorrow someone will
come with practical voxel based system instead of current abstract vector
ones, what will we do?

~~~
colordrops
HTML and CSS are slowly losing ascendancy as the main render surface for web
apps. The canvas provides a standard pixel buffer for both JS and compiled
applications, making HTML's limitations no longer a restriction.

~~~
stu_k
Except that rendering everything to canvas is not accessible to people with
certain disabilities.

Also mentioned recently here
[https://news.ycombinator.com/item?id=16347216](https://news.ycombinator.com/item?id=16347216)

~~~
vincentriemer
At least until we get the AOM: [https://github.com/WICG/aom/blob/gh-
pages/explainer.md](https://github.com/WICG/aom/blob/gh-pages/explainer.md)

------
bobajeff
I have this idea that if they split off WebAssembly, WebGL and WebAudio into a
external format for applications it would be easier to implement than a full
browser.

~~~
avaer
I've gotten pretty far doing exactly this (only-those-apis browser in Node):
[https://github.com/modulesio/exokit](https://github.com/modulesio/exokit)

Not released yet, but there's basic Windows, macOS, Linux, Android and iOS
support. Contributions welcome!

~~~
bobajeff
Your ideas is actually really cool (very tech noir).

But that's not what my idea is. My idea isn't really a browser. It's just a
separate format/mimetype (.app or .game) for a app runtime.

Small projects could implement it. We can have it embeddable as a object in
browsers. Clients other than major web browsers (Gopher, Dillo) could embed
it. Embedded devices (Roku) could include support for it. It could even be
used in a physical media like SDcards or DVDs.

~~~
avaer
Awesome, thanks for the clarification.

Here's what I'm trying to reconcile: if the idea is to break off the awesome
subset of multimedia web tech, because it would be easier to implement than a
full browser -- I 100% agree! -- then why is there a need for a new
mimetype/format?

Many things could be done better than the current HTML-as-laundry-list
approach, but if we invent a new format, that adds the extremely difficult
problem of getting everyone on board, rewriting things for it. The brilliance
of asm.js (which gave birth to WASM) was that everyone had already implemented
it before it existed.

~~~
bobajeff
Like you say it would be a subset. It's all just standard WebAssembly. The
APIs of course would need to be made callable from WebAssembly but that's
already planned for in browsers anyway.

Until browsers supported the mimetype they could just be served as .wasm
files. Not that the mimetype is important at all I just think it more clearly
states it's intended use.

------
Animats
What problem does this solve for the end user?

Few sites even use WebGL for anything interesting.

~~~
na85
Most of these fads seem to be driven not by what's good for the user but
what's good for the advertiser and the developer.

Low level GPU access will just make it easier for them to run montero miners
on my browser.

~~~
Animats
There's an ICO for that.[1] Really. They want to offer browser based virtual
reality which mines tokens in the background. The initial token sale is in
progress now. The virtual reality system appears to be total vaporware.

[1] [https://55cancri.io/](https://55cancri.io/)

------
matt_m
Well, as a small developer I'd love to see a higher-level, easier-to-use API
(than Vulkan) that can be used as a modern, cross platform replacement for
OpenGL. If it could be used both in a browser and standalone even better. If
it was available outside of a browser (even as a Vulkan/Metal wrapper), I
think it could become a no-brainer replacement for where OpenGL ES is used
today.

I understand that engine developers can get some (small?) percent improvement
from an ultra-low-level API that exposes more platform specific details. But
they almost always support multiple APIs natively, and already use the low
level ones where available. There may be a small performance benefit (over a
higher level API) in the browser, but I don't think the browser is a likely
target for ultra heavyweight apps / games, that usually are multi-GB downloads
anyway, in the near future. And keep in mind that if smaller software sticks
with WebGL because of API complexity, that might be a big performance loss for
the user.

But a higher level API would have great benefits everywhere! Right now the
only real cross-platform graphics API is OpenGL ES 3.0, which is becoming more
and more obsolete, with little sign of the situation changing. No compute
shaders, no AZDO, etc. Any move beyond that feature set now requires multiple
APIs, shaders etc. And the easiest way for a small developer to get those
features is still to skip Vulkan and to stick with GL ES 3.1+ on
Win/Linux/Android and choose Metal on Mac/iOS. Of course on the web for those
features there are no options at all.

I think if a new API was available that was easy to use and truly cross
platform (including web), it would be the obvious first API to implement for
all new graphics software. And this would be a much larger benefit than an
unknown performance improvement that is accessible mostly to engine
developers.

~~~
zanny
If you are making a commercial product Vulkan + MoltenVK should be perfectly
serviceable as a target. There are also free software attempts to make a
wrapper.

At this point, so long as Apple is in your target market, there will never be
an everywhere API because Apple does not want there to be one. The whole point
of Metal is to make your life harder so developers currently writing for Apple
first are less likely to port their software elsewhere.

~~~
matt_m
Yeah that seems like that might be the best option right now, and it doesn't
look like it will be too expensive either. What I probably would like more is
the reverse, a Metal wrapper for Vulkan, since it seems so much easier to get
started with Metal. Too bad Metal is a Swift/Obj-C API so it's not
straightforward how to make it cross platform.

I don't know how much bad faith I want to assume on Apple's part, since there
are probably some legitimate technical reasons why they don't want to support
a lower level API (and they came out with Metal first). Vulkan is such a pain
to use anyway that it's probably mostly used by engine developers who
generally don't have a problem supporting multiple APIs like Metal / DirectX
etc.

But at the very least, it would be nice if they upgraded their OpenGL version,
since they already support that and it's only them holding back some of the
newer features.

~~~
pjmlp
Vulkan is such a pain that one of the GDC 2018 talks is "Getting explicit: How
hard is Vulkan really?".

[https://www.khronos.org/news/events/gdc-2018](https://www.khronos.org/news/events/gdc-2018)

~~~
Tobba_
Vulkan is almost 1:1 equivalent to "modern" OpenGL except the need for a more
advanced allocator. You had to write the exact same thing in OpenGL if you
wanted decent performance, except through an entire translation layer.

I don't think anyone who actually worked on a serious engine thinks it's
somehow harder to use. Sure, there's more boilerplate and it might get pretty
difficult to port something to it, but that's something else entirely.

~~~
pjmlp
Except one also gets to compare Vulkan with Metal, DX 12, LibGNMX, LibGNM ,
NVN, all of them with a bit more developer friendliness and tooling in mind.

Which I guess, it is where the resistance is coming from, hence the need for
such presentation.

------
slavik81
I don't understand why this is a priority when WebGL is still so rough. Maybe
we wouldn't need a new API for performance if WebGL worked better. There seems
to be lots of room for improvement. My WebGL programs were much slower and
were harder to write than the native versions of the same programs.

We should also probably sort out the native low-level APIs before setting the
standard for the web, because otherwise we're building on top of a big mess.
Though, my impression is that the WebGPU initiative is basically just another
battleground in that struggle. I don't have any faith that this is being done
for the good of users. It's just strategic ground to capture.

~~~
stillkicking
Because WebGL is an evolutionary dead end, for a variety of reasons. The
initial idea was to track OpenGL ES, but that isn't really true anymore.
Because of Windows, WebGL has to stick to a subset that can be easily
translated to Direct3D. Because of GPU process sandboxing, anything that flows
back from GPU to CPU is a huge synchronization problem. On top of that, mobile
GPU drivers continue to suck badly, which means most of the extensions that
were supposed to expand WebGL's scope into this decade are still out of reach,
with 50% or less support in practice.

On the flipside, the native low-level APIs have all diverged. Vulkan, D3D12
and Metal each made different decisions, so it's pretty much inevitable that a
4th standard will have to be created to unify them. It will probably be higher
level than any of the 3, and it will still be subject to strong sandboxing
limitations.

Personally I think the big issue is that people stare themselves blind on the
traditional graphics pipeline. Modern renderers have evolved past this, with
various compute-driven and/or tiled approaches common place now. They're a
nightmare to implement against current APIs, because a small change in
strategy requires a rewrite of much of the orchestration code. The job of
figuring out how to map your desired pipeline onto the hardware's capabilities
should be the job of a compiler, but instead people still do it by hand. Plus,
for GPU compute-driven graphics to be useful for interactive purposes beyond
just looking pretty (i.e. actual direct manipulation), you absolutely need to
be able to read back structured data efficiently. It's not just about RGB
pixels.

There's an immense amount of potential locked inside, but the programming
model is a decade or two out of date. Only AAA game companies and the vendors
themselves have the resources to do novel work under these constraints.
Everyone else has to throw together the scraps. Even the various attempts at
LISPy GPU composition fall short, because they don't attempt to transcend the
existing pipeline.

~~~
hasahmed
I would like to hear more about this. I was quite surprised how difficult
things were when I started dabbling in Opengl and I thought that there has to
be a better way. I know that there are libraries that build on top of Opengl
and the like, but then always its a sacrifice of the power that you could
have. It seems weird to me that it is so difficult because conceptually it
seems to me that the model could be closer to the CPU/Memory model that
everyone is already familiar with. You just have some RAM and some
processor(s) that are going to do some computations right? Although I guess
what really makes it a mess is that there needs to be a connection between
what the GPU and the CPU are doing. I don't know, I was a bit surprised by how
difficult it was. Perhaps I just don't understand it well enough.

~~~
Tobba_
To attempt to explain (desktop) GPU architecture: you don't just have memory
and a bunch of individual cores on a GPU like you would on a CPU. You've got
memory, texture sampling units, various other fetch units, fixed-function
blending/output units, raster units, a dispatcher and then a ton of processing
elements. These are all things the programmer need to set up (through the
graphics API). Each of those processing elements runs several warps
(wavefronts in AMD terminology), each which contains 32 or 64 threads (vendor-
dependent), that all have their own set of registers. The warp holds the
actual instruction stream and can issue operations that occur on all or some
of those threads. Branching is possible, but pretty limited unless it's the
same for every invocation. So the programming styles/models are incompatible
from the start.

Then the real problem is, since all shader invocations share those fixed-
function units, if you need to reconfigure them to use a different set of
textures, buffers, shaders, etc you have to bring the whole operation to a
complete halt, reconfigure it and restart it. And, contrary to popular belief,
GPUs are the exact opposite of fast - each shader invocation takes an enormous
amount of time to run, which is traded for throughput. Stopping that thing
means having to wait for the last pieces of work to trickle through (and then
when starting back up, you have to wait for enough work to be pushed through
that all the hardware can be used efficiently), which means a _lot_ of time
doing little work.

So if you're trying to deal with the above, any notion of keeping things
separate and clean (in terms of what the hardware sees, anyways) immedietely
goes out the window. That's why things like virtual texturing exist - to let
you more or less pack every single texture you need into a single gargantuan
texture and draw as much as possible using some God-shader (and also because
heavy reliance on textures tends to work well on consoles). Then you also have
to manage to make good use of those fixed-function units (which is where tiled
rasterizers on mobile GPUs can become a problem), but that's a relatively
separate thing.

Also: transfering data back and forth in itself isn't necessarily that bad in
my experience (just finnicky), it's usually the delays and synchronization
that gets you.

------
swix
Canvas and webgl is lovely in the way it quickly lets you get something on the
screen... Pixel pushing on canvas is, easy and accessible.

It makes me wonder if anyone has created some sort of port to a standalone app
with no browser involved where you could use javascript/canvas-api/webgl to
draw pixels on a canvas-like surface... without the fatness of the browser.
Just spawning some window, that would be a lovely scripting/game-dev
environment, maybe with some sdl-bindings or whatever. Anyway just rambling,
does such a project exist? Anyone knows? :)

~~~
icebraining
I suggest [https://love2d.org/](https://love2d.org/)

~~~
sitkack
I think you meant this, love2d running in the browser.

[http://tannerrogalsky.com/mari0/](http://tannerrogalsky.com/mari0/)

------
z3t4
i think the web browser is the modern terminal and it should make use of all
capabilities of the device, including hardware accelerated rendering. i think
the current webgl is too low level though and only needed if you want to make
your own 3d engine. if it was a higher level 3d renderer it would be easier
for browser wendors to make secure and optimize, and easier for developers to
use.

~~~
rimliu
And I start to think that web tech is becoming a fast food of software
development.

------
jcoffland
For the Folding@home project we would love to have GPU compute access widely
available from the browser.

~~~
gambiting
How many people do you think are actually interested in running the client in
browser? I have Folding@Home on about 5-6 machines and I can't imagine a
situation where I could run a web client but not a full client.

~~~
RandomInteger4
Chromebooks?

~~~
ComputerGuru
Those are pretty woefully underpowered for folding..

------
fulafel
WebGL 2 already gives you pretty good GPU gfx & compute. The use is limited by
implementation quality in browsers. Let's give it time.

~~~
Marat_Dukhan
WebGL 2 is based on OpenGL ES 3.0, it doesn't give you compute. Compute
shaders were added in OpenGL ES 3.1

~~~
fulafel
The OpenGL compute shader feature is not needed for doing GPU compute. It's
just another type of shader that is not connected to other GL rendering that
may be happening at the same time. People have been going GPGPU with the
traditional types shaders for a long time. And WebGL 2.0 is a huge upgrade
from 1.0 from a GPGPU point of view.

Also, a compute shader extension may be coming[1] for WebGL 2.0, it's
mentioned in some meeting slides at least:
[https://www.khronos.org/assets/uploads/developers/library/20...](https://www.khronos.org/assets/uploads/developers/library/2017-gdc-
webgl-webvr-gltf-
meetup/2%20-%20WebGL%202.0%20@%20Silicon%20Valley%20WebGL%20Meetup%20-%20Mar17.pdf)

~~~
Marat_Dukhan
It is possible to perform _some_ computations using OpenGL ES 3.0 / WebGL 2.0,
but many types of operations (e.g. anything that involves random-access
writes) are impossible, and many others (anything that normally requires
shared memory) are very inefficient. Programming GPU through WebGL 2.0 is akin
to programming desktop GPUs pre-CUDA: it is too intricate to take off.

Compute shader extension for WebGL 2.0 would be cool, but it would require to
port a large part of OpenGL ES 3.1: OpenGL ES 3.0 / WebGL 2.0 doesn't include
even random access buffers (SSBOs)

~~~
fulafel
I agree that "too intricate" is the other main problem in WebGL uptake. But
we're not even seeing WebGL versions of textbook GPU applications that are
straightforward to implement with the tools WebGL 2 gives.

For example, here's WebGL compatibility stats from a site that has counters on
technical web sites and graphics programming web sites:
[http://webglstats.com/](http://webglstats.com/) \- As you can see, WebGL 2
compatibility is only at 40%, despite having been enabled in stable
Firefox/Chrome for over a year. WebGL 1, a 7 year old standard, is now at 97%.
(And even for the nominally WebGL-enabled browsers, users often report browser
or OS crashes, so the percentages are upper bounds).

From an armchair quarterback position, if I wanted to effect GPU compute
uptake, I'd work on compiler tech and tools targeting WebGL GLSL.

------
Retr0spectrum
Maybe GPUs need to be fundamentally redesigned to allow multi-user access with
enforceable security boundaries. I'm sure this would be highly non-trivial to
implement, but it would be great for other scenarios like shared access to GPU
compute resources in "the cloud".

~~~
monocasa
That has existed for over a decade. In fact that's sort of the point of
Vulkan, et al.; with all user code being behind an MMU context, it's safe to
provide a more console like 'bare metal' API. There are bugs, but the
infrastructure is all there. The end goal is that you can only crash your own
process.

The issue with this in the browser is that the API isn't part of the
JavaScript/WebASM, and just exposing it in the same way will allow you to
subvert the sandboxing of the VM>

------
guidedlight
There is an upcoming Xbox game (Crackdown 3) that uses web-based GPU
processing to share compute amongst players in a multiplayer game.

~~~
andrewmcwatters
What does that even mean? It's an Xbox, they all literally have the same
variations of hardware. There aren't 8000 different build specs, there's at
most, what 8? How could compute sharing provide any possible enhancement in
experience at all? If anything, it's eating frame budget and it shouldn't.

~~~
stordoff
When it was first announced, there was talk of off-loading certain tasks to
(presumably) Azure. That may be what OP is thinking of.

I'd also wonder if you could share the power of any Xbox One Xs connected to a
multiplayer session, given the gap between an original Xbox One and the X is
rather large (seems like it'd be far more trouble than it's worth though).

------
visarga
Hmm... I wouldn't want a random webpage to start mining bitcoin on my
computer. How would we protect against that?

~~~
ramses0
Net Neutrality, right? It doesn't matter what the page does, it only matters
that you are allowed to visit any page on "The Inter-Web" that you like,
right?

Safari has "tab paused/reloaded due to high power consumption" (good), Chrome
has "auto-mute tabs" extension (which I have turned on), I have "tab
suspender" extension installed.

Basically, I'm trying to be a responsible consumer:

\- this tab would like access to your hard drive files

\- this tab would like access to your video camera

\- this tab would like access to your microphone

\- this tab would like to play sounds

\- this tab would like to download more than 5mb of data

\- this tab would like animation/movement

\- this tab would like to use your CPU a lot

\- this tab appears to be using a lot of your battery

\- this tab would like to use your GPU (at all)

\- this tab would like to maintain state > 24hr

the_internet.js is actually potentially really hostile (suck 9999mb at full
speed, ddos@1.2.3.4, while(1){alert(1)}, mine_bitcoin( $hacker_wallet )), and
I am much in favor of treating it as untrusted by default (low access, limited
# of cpu cycles) until "trusted" (ie: android permissions swap: i give you the
executable, you give me the permissions).

YouTube? Yes to whatever they ask.

ShadyWebsite.1234.some-random-domain.ru? You can d/l 300kb and can't do
anything else (ie: web 1.0/no-script).

While each "tab" in a web-browser attempts to provide "safe" access to the
computer resources, it is still not "permitted" access to computer resources.
Whatever the browser defines as "safe" is 100% ok, which has to work equally
well for WASM-unreal-tech-demo-castle as well as cnn.com.

I'd prefer cnn.com only had 300kb download, no external domains, no sound, no
battery, no cpu, etc.

As I _trust_ cnn.com more (to the same level as youtube), I would then permit
sound by default, permit large downloads, permit gpu, permit
animation/video/etc.

------
happyguy43
How does this interact with WASM?

------
sitkack
Couldn't many of the GPU operation be accessible via functions over typed
arrays? How does this relate to using SIMD from JS?

------
baybal2
It has same downside as WASM. Running precompiled bytecode is a bad for
security as it always been.

It was no more than a year when a remotely exploitable WASM hole was exposed
(derivatives of Spectre and co.) Knowledgeable people told that ISA level hole
that can be exploited remotely over the web will be "a one minute global IT
disaster" if somebody would resort to propagating it through a big adnet or
paid traffic scheme.

As for WebGL as it is now, there were numerous sites on my memory that
froze/crashed/rebooted both Linux and Windows systems, which means that the
prime suspect there was a buggy shader as it is the only thing resembling raw
instructions that can be passed to gpu through webgl.

~~~
Felz
Weren't Meltdown/Spectre exploitable via Javascript?

~~~
baybal2
Actually, the POC required both sharedbuffer object and ASM js (actually an
even worse thing than a bytecode)

