
Supercomputing on Nitro in AWS Cloud - fanf2
https://ieeexplore.ieee.org/document/9167399
======
WoahNoun
The Monday Night Infrastructure keynote from re:Invent was all about the
changes required to support true super-computing use cases inside a placement
group. The nitro virtualization layer is a huge part of it (the other being
the networking between hosts within a placement group).

[https://www.youtube.com/watch?v=GPUWATKe15E](https://www.youtube.com/watch?v=GPUWATKe15E)

------
stonogo
This seems like Amazon trying to make excuses for being the only cloud
provider who thinks they can pursue HPC workloads without Infiniband. I can't
help but notice that this paper consists of nothing but network bandwidth
testing, and their latency testing was bulk-only. Also, IEEE Micro is a weird
place to publish real supercomputing work.

Does Amazon have a plan for tightly-coupled large-scale software on AWS? Is
there an exascale team at AWS?

edit: I found a thread on Twitter[1] where it is explained to us that the
protocol in question will remain proprietary because nobody is smart enough to
keep up with Amazon engineering, and Amazon engineers' convenience is more
important than standards.

This is extremely disappointing -- it turns the entire fabric into a black box
and slams the door on meaningful repeatability for large-scale modeling
efforts. It's so frustrating to see all this effort fenced off behind 90s
Business Dude mindsets.

[https://twitter.com/_msw_/status/1297231949895831553](https://twitter.com/_msw_/status/1297231949895831553)

~~~
wmf
OTOH, why should we want an Infiniband monoculture (which is also a
Mellanox/Nvidia monopoly)? Why not let AWS try their weird stuff and let it
fail on its merits?

~~~
stonogo
Breaking the infiniband monoculture is _why_ I am sad they're sitting on this
protocol. I would like to deploy it to my cluster in my datacenter for
testing. I cannot, and that sucks.

Even if SRD is the greatest thing ever, it doesn't do me any good to replace
an infiniband monoculture with an AWS monopoly. I can't even use public-cloud
computing for about two thirds of my workload, so this is a non-starter.

~~~
wmf
Even if the protocol was documented you couldn't use it without hardware which
AWS won't sell you.

~~~
stonogo
We'll never know. I have access to network hardware that is extremely
configurable via FPGA.

"What hardware does this protocol require for successful deployment" is just
one of the many questions that are currently only answerable via Amazon
trademarks.

~~~
wmf
I have a feeling that anyone competent enough to implement SRD in an FPGA
could also invent their own similar protocol.

~~~
stonogo
That may be true, but I'm not funded to develop similar protocols. I _am_
funded to _evaluate_ them, which, again, I cannot do because of how they're
handling this work.

------
ZeroCool2u
"Instead of preserving packets order, SRD sends the packets over as many
network paths as possible, while avoiding overloaded paths."

From my naive perspective, this sounds like a great way to end up overloading
all the paths that aren't currently overloaded. I suppose maybe this works out
when you have global information about the network, unlike the regular
internet though?

~~~
fanf2
AIUI the sender manipulates bits in the packets that the switches use for ECMP
(equal cost multipath) to shard the traffic across the available paths. The
protocol is designed for very fast feedback about path latency so it can avoid
building up queues in switches; if any path seems congested then the sender
stops using its ECMP value. There isn’t any duplication of packets (unless
they are lost) nor does it rely on global information (because that is far too
slow).

(Based on this discussion
[https://twitter.com/_msw_/status/1297223835519815681?s=21](https://twitter.com/_msw_/status/1297223835519815681?s=21)
as well as the paper)

