
Elastic Network Adapter – High Performance Network Interface for EC2 - jeffbarr
https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/
======
illumin8
Does anyone know what type of hardware they are using to accomplish this? Is
it just your standard Intel X520/540 10GbE adapter, or is there some secret
hardware they have access to?

FWIW, saturating line rate 10GbE has been problematic at times in the past due
to challenges like single-threaded drivers, which could only run on 1 core at
a time, even if there are 128 logical cores in a system (like the X1
instances). There are also challenges offloading VXLAN encapsulation to
hardware at 10GbE line rate, but it seems like AWS has figured that out.

~~~
netguy6
Since it offers 20Gbps to the VM, i'd guess it's a 25GbE NIC, since those and
accompanying switches have just come out.

VXLAN in the NIC (and the switch) at full line-rate offload has been a thing
for at least 18-24 months.

Most Modern NICs also support all of the features listed here, like checksum
offload, Receive steering, multi-queue, etc. Check out Mellanox CX4/5, Intel
X710, QLogic, and Netronome if you want fancy stuff. They aren't even that
expensive anymore.

Their Linux driver has support for every current and future standard:
1/2.5/5/10/25/40/50/100/200/400G, nice future proofing.

They also have a DPDK driver:
[http://dpdk.org/browse/dpdk/tree/drivers/net/ena/base/ena_co...](http://dpdk.org/browse/dpdk/tree/drivers/net/ena/base/ena_com.c)

~~~
illumin8
Thanks, but my guess is that it's actually a bonded 2x 10Gbps link, which
would also give you 20gbps, and cost a lot less per port.

~~~
netguy6
With current per port costs and economics, that doesn't make much sense. The
$/Gbps/Watt improves with every spin of the chip, and every increase in serdes
bandwidth.

10GbE (64x10G) - ~$4,500, $70/port

10GbE (32x40G) - ~$6,000, $50/port

25GbE (72x25G) - ~$7,000, $100/port (Seen as low as $5k)

25GbE (32x100G) - ~$9,000, $75/port (Seen as low as $7k)

2x 10GbE = $100

1x 25GbE = $75

Not to mention the extra cabling costs, and complexity.

------
tevenian
Huh -- why is Amazon introducing ENA as a paravirtual virtual device? They
already provide Xen netdevs to VMs.

The Xen PV device would be able to use the (more efficient) xen bus/event
channels for interrupts vs. ENA (MSI-x, by looking at their driver on the
Linux netdev mailing list).

The Xen PV device already also supports multiple queues for xmit, see
netback_init() in netback.c.

2\. The ENA LLQ interface (pushing short packets via CPU write-combiner
writes) only makes sense when you are writing to a physical
PCI-e/Hypertransport/QPI device.

I suspect that this ENA NIC is at least partly implemented in silicon/firmware
rather than dom0 s/w.

Perhaps something like Azure's smart NICs? (SIGCOMM 2015 keynote)

------
jewel
20gpbs is fast enough for a raw 4K video stream at 60 frames per second in
ABGR format (e.g. what you'd get out of a video card). That's slick.

------
xchaotic
Tl;dr 20gbps virtual Nic

~~~
illumin8
Oversimplified - it's also a multi-threaded driver with multiple queues. This
should hopefully allow you to fully saturate the 20gbps available to you.

