
Packets per Second Limitations in EC2 - mbarlocker
https://www.bluematador.com/blog/ec2-packets-per-second-guaranteed-throughput-vs-best-effort
======
kev009
A more plausible explanation is that the xen networking path is simply
expensive, the intel VFs are limited by queue count and silicon (i40e isn't a
great ASIC), and the Annapurna part is really an ARM64 NPU. NPUs have been
abandoned by most silicon vendors and have a tragic history. It's simply hard
to make NPUs work right at attractive price/power/performance and at high
speed versus fixed function scatter/gather I/O units coupled with general
purpose CPUs running software network stacks. The only benefit Annapurna gives
EC2 over a software device model is a hard security boundary of effectively
another computer inside the computer for Nitro metal as a service. I think
this is one reason why EC2 is limited to 25G while 100G has been commodity for
a long time.

Here is a demonstration of a software stack that can scale toward hardware
limits without relying on a particular vendor
[https://www.slideshare.net/SeanChittenden/freebsd-vpc-
introd...](https://www.slideshare.net/SeanChittenden/freebsd-vpc-
introduction). This approaches 100G line rate for large packets which is what
it was optimized for. I don't know PPS at low packet size but do know what
would be required to optimize that use case and it could be done pretty
quickly.

~~~
discodave
Interesting theories on the EC2/Annapurna situation.

Do GCP, Azure, or any other cloud providers offer 100G networking?

~~~
virtuallynathan
Not to the instance, AFAIK. Google Cloud maxes out at ~20Gbps, and I think
Azure does ~40Gbps.

~~~
p0rkbelly
It's really about flows as well, not necessarily total throughput.

AWS Nitro allows 5G/bit per flow. And then maxes out at 25G/bit. I know GCP
does something similar.

Also, pretty sure that is false regarding Azure, they have a small
availability of Infiniband, but, that is not on their general compute platform
and has a narrow use case/many restrictions. Azure has had the worst
networking performance from my experience and only had 10GbE NICs (it's been a
while though)

------
ttul
Amazon also limits DNS queries - probably in a well meaning attempt to prevent
DNS amplification attacks from originating within AWS. And I mean DNS queries
across their network whether or not they hit Amazon's DNS servers. This is
_any_ port 53 UDP traffic.

[https://www.sparkpost.com/blog/undocumented-limit-dns-
aws/](https://www.sparkpost.com/blog/undocumented-limit-dns-aws/)

~~~
greglindahl
I wonder if this is related to connection tracking?

~~~
john37386
By default, Amazon uses stateless firewall. It means that by default it's not
tracking connections.

~~~
spydum
I think you may be mistaken. Security Groups are stateful:
[https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit...](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html)

As suggested, it's very likely they hit the connection tracking limitation:
[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-
ne...](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-
security.html#security-group-connection-tracking)

I've personally witnessed teams hit this specifically for DNS (usually for
internal, where you have explicitly permitted src/dst).

~~~
john37386
Yep I mistaken Security groups with Network Acl. Thanks. It's in the best
practices to not track dns for big systems.

From powerdns
[https://doc.powerdns.com/recursor/performance.html](https://doc.powerdns.com/recursor/performance.html)

------
toast0
Note that if your traffic hits the ec2 connection tracking security groups,
you will also hit per instance limits on the number of tracked connections
[1]. As far as I know, they don't come out and say they have a limit on the
number of tracked connections, but they do, and it scales by instance type --
better to adjust your rules so the traffic is allowed in a stateless manner.

I don't know, but wouldn't be surprised if connection tracked packets are more
limited than packets that aren't tracked.

[1] [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-
ne...](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-
security.html#security-group-connection-tracking)

~~~
greglindahl
That sure sounds like it's being processed by the standard Linux firewall. In
which case, yeah, if you have (my favorite example) a web crawler operating on
the general web, you'll hit serious limits.

~~~
toredash
There is a limit of you have a Security Group attached with a rule that is
-not- 0.0.0.0/0\. So for anything that is public / heavy utilized, the
recommendation is to open the service up to 0.0.0.0/0.

------
rbranson
The (undocumented) PPS limitation on EC2 instance types before they added SR-
IOV NICs is around 150K PPS. If you had your own full machine — usually the
top size of a given instance class, but no guarantee — this would be pretty
consistent. But it was a shared resource. This made running memcache clusters
really painful on EC2, given that they’d easily get limited by packet
throughput before CPU or bandwidth. With modern instance types it’s much
better!

~~~
ra1n85
????

Where are you getting these numbers? For what instance types? For what
protocols? This is just wrong.

~~~
edoceo
GP said it was undocumented (I presume empirical, but would like more details)

Do you have a more accurate dataset? What are your observations?

I'm assuming this affects loads of HN readers and I too am interested in what
the facts are.

Side note: you may be getting downvoted because a source, or other details are
lacking. I too get downvoted for posts that lack these details.

~~~
samstave
I took him saying "just wrong" as being "im not OK with this", but i could be
incorrect.

However, with that said, i have always found that calling your rep and asking
about specific un documented limits is the fastest way to get to the bottom of
per-instance/account/vpc/whatever limits.

Just as there are limits that can be changed if you agree, in writing, that
you will be financially responsible for whatever the impact is (e.g. when you
could tell them that you wanted spot price limits adjusted for you to be able
to better bid above the scaling factors that were in place(not sure if this is
still the case))

Some limits are global and cant be changed/negotiated, but other undocumented
limits....

------
Rapzid
I guess technically it is, but I hesitate to call the need for PPS limitations
in the DC as "over subscribing".

Connect a single server to a network and it's oversubscribed. That's a bit
hyperbolic, but even some beefy networks can be seriously burdened by just a
single server spamming UDP packets without some sort of QoS.. Especially if
they are bypassing user space and using the kernel to just replicate a bunch
of packets onto the wire :)

I'm probably a bit biased from having spent time setting up linux TC on xen
hypervisors for this very reason; and I think we even settled at 50k pps for
the per vm limit too..

~~~
blazespin
They should still publish the limits. If they are reasonable, customers won't
mind.

------
RA_Fisher
The distributions are mixtures. This is my favorite package for modeling
those:
[https://cran.r-project.org/web/packages/gamlss.mx/index.html](https://cran.r-project.org/web/packages/gamlss.mx/index.html)

------
ramshanker
One thing I was looking for the entire article weather these limitations are
imposed by the Instance or the TOR networking gear.

Maybe random reallocation to a rack with newer switches will guarantee better
baseline PPS.

~~~
rbranson
It has to do with hardware support for virtualized networking. Instances
capable of SR-IOV or with the ENA can sustain millions of PPS. It makes a huge
difference.

~~~
ra1n85
Not correct. Instances are rate limited based on their type and protocol.
Enhanced networking support increases total potential performance, but there
are artificial limitations put in place.

------
shaklee3
Does anyone know if this applies to the dpdk ENA driver?

~~~
dgemm
What difference would the driver make?

~~~
shaklee3
I'm assuming the limitation is in the smartnic, and perhaps a different driver
is tuned differently.

------
patrickg_zill
Eventually people will realize how AWS overcharges for what they deliver. But
of course there's nothing wrong with pricing yourself higher than the lowest
cost option...

This is perhaps already being seen in a piecemeal fashion as people compare eg
S3 storage prices with other companies' prices.

------
john37386
EC2 throttles everything by default and PPS is no exception.

What can you do when your system needs more bandwidth, cpu, ram or any other
kind of resources?

You can either scale vertically... which is not bad at beginning of a project,
but sooner or later you will hit the ultimate limit.

Or

You can scale horizontally. Which means that you have enough nodes or
instances to bypass those limits and make sure your projects grow well over
time.

Netflix runs on EC2 and they probably generate billions of PPS from Amazon.
For sure it's several millions PPS a d they seem to not hit the limits
mentionned in the article.

~~~
spydum
What is it you think netflix runs on AWS? Content distribution is served from
their Open Connect CDN, not AWS.. last I understood, most of Netflix cloud
workloads were analytical/DWH, and services.. Not generally billions of PPS,
and certainly not to single instances.

