I think they tried it already in the original transformer paper. THe results were not worth implementing.
From the paper(where Additive attention is the other "similarity function"):
Additive attention computes the compatibility function using a feed-forward network with
a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is
much faster and more space-efficient in practice, since it can be implemented using highly optimized
matrix multiplication code.
Can someone elaborate on how this translates to the actual performance of GPU. What is the performance jump (will this allow to run 4k x 4k per eye VR headset) when compared to current GPU generation ?
Today Even with Nvidia 1080 cards running 2k x 2k per eye headsets is not an easy task and beyond a smooth VR experience.
It's right there in the article: 1.35x. That sounds like raw speed though, with actual changes probably depending more on what people do with the new technology.
" The headline PPA values that Samsung is announcing are also impressive: compared to 7nm, 3GAE will offer 1.35x performance, 0.5x power, with a 0.65x die area."
...
Samsung states that these performance numbers are based on using larger width cells for critical paths where frequency is important, and smaller width cells for non-critical paths where power savings are crucial. Technically Fmax of the widest cells is listed as 1.5x, while power at Fmax is 0.6x. Power at iso-performance is where the 0.5x number comes from."
I don't think 35% at fixed frequency makes sense; 35% is the frequency boost (with power savings too).
Nearly every existing VR game runs perfectly smooth with a 2080. You won’t get 90fps 4K even with a Ti card, but for FHD we’re well served, price aside.
GPU scales really well with transistor count and You can expect about double the performance with best of 7nm, and likely another double for 3nm. So you are looking at 4x the performance.
Of course this does not take into account about TDP, Clockspeed, Memory Speed, Cost of Die Size etc. What is technologically possible may not be economically possible. We are going to need much faster memory, GDDR7? or HBM3, how much would those cost?
And a 3nm ( Whether that is from Samsung or TSMC ) Nvidia GPU will likely be 2022 or 2023 at the earliest.
More transistors require more energy. GPUs are already at 300W. Doubling the number of transistors means double the power consumption which means 600W. Smaller transistors are more energy efficient but they can't reduce the energy needed per doubling by 50%. Therefore you end up with a 400W GPU. That's still too high. Energy efficiency is now far more important than raw transistor count.
Edited a bit, Nvidia has always been late ( or waiting for it to mature ) to leading node. So in reality we are looking at 2024 / 2025 for mainstream part.
I tasted kofola as a child and I did not liked it all. To me it tasted like a weird mixture of tea and coca-cola and I found it disgusting. Coca-cola tasted much better. Later as an adult i tried kofola couple of times and again did not like it. As the time went by somehow I get used to it, and now i go for kofola over coca-cola everytime. I dont expect anyone trying it for the first time to like it.
From the paper(where Additive attention is the other "similarity function"):
Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.
reply