
Serving 6.8M requests per second at 9 Gbps from a single Azure VM - profquail
http://www.ageofascent.com/azure-cloud-high-speed-networking/
======
SEJeff
They had me until the moment where I read "While our game servers are built
more like HFT Quant trading servers" yet they are in the cloud... They clearly
have no idea what they're talking to regarding finance, where latency matters
a whole lot more than throughput.

"HFT Quant Trading Servers" optimize things down to the micro and even
nanosecond level from the hardware to the bios all the way up to the
networking interconnects and applications + c libraries. Optimization to this
level is simply impossible from the cloud, where even with VT and SRIOV, the
performance penalty is simply far too high (and measurably so!).

Disclaimer: I'm a technologist who has worked in the "HFT Quant / Electronic
Trading" Industry the past 8 or so years. I literally build infrastructure
like this for a living.

~~~
benaadams
Yes, the analogy to HFT trading is a bit clumsy - sorry - what we're saying is
that we've borrowed techniques from HFT (especially at the decisioning end)
rather than saying our system is suitable for HFT (which it absolutely isn't).

Disclaimer: Author of article

~~~
mc_hammer
its probably fair.

gamers "flip their shit" when ping is not steady, and are only happy at ~50ms
response times. a 200ms response in video games is usually considered
"unplayable".

~~~
btym
In any split-second game (and this article literally advertises "real-time
twitch combat"), 200 ms is unplayable. With two players of equal skill, it
comes down to "who can react quicker", and the 30 ping player will win every
time.

~~~
vvanders
Totally depends on the type of game and how the design is structured to deal
with latency.

See my reference to SubSpace farther down, it's a highly skilled game that can
be easily played on a ~250ms latency. The large majority of the game is
predicting where the other player will be in ~0.75s and setting things in
motion ahead of time to intercept them.

Any game based on prediction and designed for smooth re-integration of the
game state can be done on latent connections. Heck there's even been some
really impressive stuff in the fighting game space involving re-winding
gamestate to resolve hits ~0.25s in the past(the original Counter-Strike does
this as well, although not as well which is why you'd sometimes see people
rubber-band back around a corner when hit).

------
vvanders
TCP for twitch game? Ugh.

Cool that you're getting a ton of throughput but you don't use UDP for
throughput, you use it so that the one dropped packet doesn't back up all of
your time-sensitive data. You _want_ to drop packets that are out of order
since you're using dead-reckoning to keep the game state psuedo-in-sync.

Seriously, there's a reason everyone uses UDP(or IPX!) since the days of doom.

~~~
benaadams
Its a constraint of the browser technology stack; not only do we have to use
websockets for real-time (TCP/IP) but we also have to go over SSL/TLS as
intermediary hardware/software (routers, proxies etc) still routinely break
websocket connections otherwise.

(Disclaimer: Author of article)

~~~
vvanders
I really hope your game designers know how to mask latency. If you're going
over TCP/IP then expect to need to resolve a game model that can sustain at
least ~750ms of latency.

I totally understand the browser constraint part but you're going to hit some
serious issues once you start having real-world clients that have >3% packet
drop.

It gets even worse when you start adding different discrete layers on a single
TCP connection(chat, larger game state not related to twitch parts, etc). Each
one of those channels has the chance to drop a single packet and then your
whole world comes to a stop until you can roundtrip that single packet that's
unrelated to the realtime info you care about.

FWIW most stacks end up implemented channels over UDP with
unreliable+unordered, reliable+unordered and reliable+ordered for the
different latency needs of the specific channel.

I've done a fair bit of this stuff in a production environment and have to
echo my original comment. TCP is great for lockstep but if you're doing dead-
reckoning it will fall apart on anything other than a LAN.

Edit: I also don't mean this as a dig, just that I've seen many teams go
through this and it's usually at a point where it's too late to make any
changes.

~~~
benaadams
Not saying its easy and we have discovered the reason most MMOs have long wind
up animations for actions...

The other layers (chat, non-space game state) are handled by discrete
microservices on other connections to other hosts - to keep the space flight
data slim and connection dedicated in use.

Can always give it a try, our next public playtest is on Saturday 6 Feb 8pm
GMT/UK | 12pm Pacific | 3pm Eastern US at
[http://www.ageofascent.com/](http://www.ageofascent.com/)

Just don't try it on a phone; give us a chance with the latency ;-)

~~~
vvanders
I wish you the best of luck but I really think you've got a poor technology
fit for what you're trying to do.

Games like SubSpace/Continuum were doing 300+ player twitch based on UDP based
connections over 56k dialup back in '97\. Any sort of packet loss(and
subsequent TCP exp backoff) is going to weak havok on your latency.

~~~
twodave
Holy cow, talk about a flashback. I played the crap out of that game for about
a decade.

~~~
dnr
Add me to the HN/Subspace club: I played quite a bit of it starting from the
end of beta (~'97) and also got pretty involved in the bot/modding/development
scene that rose up after VIE left the picture. Some of the best gaming
experiences I've ever had, and I learned a lot on the development side too.

------
nacs
The article has small bits of useful info but the article and site reeks of
submarine (Microsoft Azure) advertising for me..

Their homepage has a testimonial/review-blurb of the game as the very first
thing from a Microsoft "White paper".

Below that there are 2 videos, one of which is an "Interview Rob Fraser -
Microsoft".

And of course the article itself focuses on Azure with another 12 mentions of
Azure and how it runs Windows, .Net and C# so well (I'd hope so).

Also, the 7 mill requests/second aren't real world numbers but just a software
benchmark.

Oh and the benchmark was run on an instance that has 16 cores, 3TB of SSD
storage, and _224 GB_ of RAM. That's more like a dedicated server than a
single cloud server instance (and waay more expensive than dedicated at _$3061
per month_ per instance).

~~~
brazzledazzle
Microsoft may have given them free or heavily discounted access in exchange
for being able to use them to advertise. That was apparently the case with
TitanFall.

------
yread
The actual code they're using

[https://github.com/benaadams/benchmarks/blob/managed-rio-
exp...](https://github.com/benaadams/benchmarks/blob/managed-rio-
experimental/experimental/ManagedRIOHttpServer/RIOServer.cs)

------
ckluis
If possible - they should try and fix the C#/windows techempower
implementations which are not anywhere near their results.

~~~
benaadams
(Author of article) Working on it, currently the #1 contributor to AspNet
Core's new webserver, and it is doing about 2.5M rps on equivalent techempower
hardware

[https://github.com/aspnet/KestrelHttpServer/graphs/contribut...](https://github.com/aspnet/KestrelHttpServer/graphs/contributors)

~~~
ckluis
It looks like they are freezing techempower’s next round in a couple weeks.
And congrats on the awesome work!

------
pjmlp
Very interesting post.

I love that they could have the resources and time to explore using safer
languages for their use cases.

------
jtwebman
So wait was this all just serving Hello World?

~~~
profquail
That's basically an echo server, which is a standard networking test. Leaving
out the overhead of actually processing the payload means you're able to see
how well the networking stack (and the parts of your program which interface
with and manage it) can perform. Getting a clean number from a benchmark like
this also makes it easier to estimate how much performance overhead is induced
by the other parts of your program.

~~~
jtwebman
I think it is just a fake number. Everything has running code also what if the
VM you spin up is in a different part of the data center or next to another
heavy network app? To me this seems like a waste of time to look at because
depending on the logic it will not matter if the server can do twice as many
hello world requests when it has to wait 30ms for business logic anyways. I
want to see a real running system and the change out the web server as well as
the hosting vm and see who comes out ahead with real systems. This just shows
use this web server for making a CDN, nothing more.

