That's one of the issues HTTP/2 has. Because all the requests and responses are now multiplexed over a single TCP connection, losing a single packet is going to block everything. This makes HTTP/2 actually work on unreliable networks. This problem is also known as Head-Of-Line (HOL) blocking.
Google is aware of that issue, that's why they explored SCTP first and now QUIC as alternate layer 3 protocols to replace TCP in the future.
SCTP is problematic because middle boxes need to be made aware of it since it uses a different IP protocol. This is gonna require massive infrastructure changes and probably won't fly (like IPv6).
QUIC, which runs over UDP, is easier to adopt. However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable.
TCP is still the safest bet. We need to figure out a better (and magical!) way somehow...
[quickly becoming standard comment for me]: IPv6 is in widespread use. IPv6 is not "the future". It's the present. I have services in production where >30% of users access the service over v6. Use is growing. IPv6-only mobile operators are also now a thing.
The idea that middle boxes need to be aware of protocols is incredibly destructive to the end-to-end principal - the critical property of IP networks that enables a rapid pace of permissionless innovation. Middleboxes are evil incarnate.
Fortunately, academic research as well as operational experience has shown that TCP and UDP provide the primitives required to build any protocol. QUIC, which you mentioned, is an example of this. Google has made the explicit - and probably wise - decision to keep protocol state encrypted. While there would be some nice optimizations possible if routers could be aware of things like 'this session is closed now', the risk of middleboxes making it so that the protocol can't continue to evolve is too great. The solution is to specifically prevent middleboxes from having the knowledge needed meddle in an 'intelligent' way.
The last statistics I saw from Google were that 95% of networks don't cause problems for QUIC (i.e. QUIC connections are as fast or faster than HTTP/2). The main issue I'm aware of is not so actually de-prioritization but rather rate limiting (it's often implemented as a quick - and very very dirty - DDoS mitigation technique). Google's stated solution is to fallback to HTTP/2 based on measurements of user experience on a per-AS basis.
Networks which try to be "intelligent" rather than just provide adequate bandwidth will tend towards providing a crappy user experience. Solutions which optimize for well behaved adequately provisioned networks and have an acceptable-though-degraded fallback seems like a reasonable way to promote progress.
SCTP has an RFC-defined UDP encapsulation which will deal with many evil networks.
> IPv6 is in widespread use. IPv6 is not "the future". It's the present. I have services in production where >30% of users access the service over v6. Use is growing. IPv6-only mobile operators are also now a thing.
That's... great, and I'm glad there's progress being made somewhere, but I don't think it's fair to say it's "the present". It's on its way.
My ISP has rolled out FttH across a wide area but still only offers IPv6 on business buildouts. Even if they offered IPv6 to general customers, they still have a ton of home gateway devices out there that don't support any IPv6 at all. The other ISPs don't offer it or have any external roadmap either. I don't think this situation is all that unusual or unique - it's just not something that the people paying the bills are clamouring for and it's a long road from where we are to significant penetration.
Oh, and EC2 still doesn't support IPv6 on anything but their load balancers. What portion of the market and traffic do you think they make up?
Until you can run an IPv6 service without dual-stack, it's not in the present in the same way you're talking about. We are still very much in the transition period. Yes, it's positive to see IPv6 growth, but the fact remains that any new service today must provide both IPv4 and IPv6 end-points and that is likely to continue for years.
SCTP uses IP protocol number 132. TCP is 6 and UDP is 17.
Many middle boxes, especially cheap home routers, are not aware of IP protocols other than TCP/UDP (and maybe GRE/ESP if you're lukcy). Their NAT will not work with SCTP.
That really assumes residential ISPs out there decide to fully embrace the idea behind IPv6 instead versus doing something asinine like handing out a single /128 address.
Fortunately, mine hands me a /64 so I'm not in that situation. :)
But its taken us 20 years to get to this point and even with the IPv4 exhaustion it's still relatively slow to be picked up on by providers in Europe or North America (except for mobile, now).
Not to say I wouldn't want SCTP to succeed but IPv6 doesn't exactly spring to mind as a model for the successful adoption of a new internet protocol.
> However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable.
What makes you think this? I play a UDP-based FPS game where "ping" times are critically important and it seems like latencies across the board have continued to drop over the years.
Latencies are ~2x the speed of light in most case. Not too shabby.
My home ISP throttles UDP traffic to about 5Mbps while TCP reaches 80+Mbps. Packet loss during peak hours can be as high as 60% for UDP but a manageable single digit percent for TCP.
My workplace has anti-BitTorrent mechanism deployed and any meaningful amount of UDP outbound traffic is dropped.
Couldn't agree more. When I was implementing adaptive routing on NuevoCloud that was one of the problems I worked on. Ended up de-multiplexing http2, once the requests hit the edge node.
So there's still one connection between the edge and the browser (can't do anything about that), but once it gets to the edge node, each request is sent over a prenegotiated connection to an edge node near the server.. then it drops down to a more reasonable number when sending the requests to the customer's server. So if a packet is dropped, it only affects that one file/connection.
Don't think any other CDNs are doing this yet.. but I think it'll become more common as they invest more resources into their http2 implementations.
This works for your use case which is great, but in my experience, you get gains with full http2 between edge and origin that you miss out on if you demux or fall back to http 1.1. We saw slightly better median latencies and much better >90th percentile latencies with full http2 from edge to origin, versus when we did http2 to edge users and http 1.1 to origin.
At Facebook, we had clients talk http2 to edge nodes near the users, which then edge terminated TLS and opened up the request to figure out which upstream was best to forward the request to. We kept http2 conn pools to each upstream from a given edge location. We also had tunable amounts of domain sharding so that the client could open N http2 conns at once to a given edge hostname (cdn[1-4].facebook.com vs just api.facebook.com or www.facebook.com). This helped with head of line blocking, assuming that your ISP wasn't hopelessly over-congested. In that case, no amount of sharding will help you, and sometimes it even works against you depending on what tcp middleware your mobile ISP is using.
We did different TCP settings for edge<->origin conns than edge<->user which let us balance between latency and throughput for origin traffic as needed, and we did some interesting work tuning edge tcp settings differently for different user networks too. Things like determining the best congestion window for a given ISP's users, how large to set the send buffers, etc.
Admittedly, this work was much easier since we had good http and tcp instrumentation, and we had high quality links between edge and origin. I just wanted people reading your comment to understand the tradeoffs. We saw a measurable difference in perf with a full http2 stack that made up for the extra complexity of running http2 top to bottom.
Your experience with http 1.1 to origin matches mine. I'm not suggesting using http 1.1 from edge > origin. Most CDNs are currently doing that, and it doesn't perform well.
I should be more specific in my description: We are using http2 everywhere.
There's a single http2 connection between the client and edge (the customer could setup sharding, but there's no way for us to force it).
That edge node has a pool of http2 connections to other PoPs, and maintains QoS stats for each PoP. It'll pick the fastest path to send the request. The pool is large enough to service all of the requests that are being handled at any one time. So in effect, if the client sends 10 requests to the edge node, each of those will be sent edge<->edge over 10 http2 connections. (The connections in the pool are kept active, so there's no tls resumption delay either.)
Once the edge node near the origin has received it, there's a smaller http2 pool to the origin server.
So at the origin, it's handling a handful of requests in parallel.. even if there's only a single client making all of the requests.
I think that's a bit clearer.. I've been happy with the performance we're getting from this.. but plan to test sctp and quic later this year for edge<->edge.
Ah cool, that's similar to what we did. I think http2 gets a bad rap, we saw a lot more positives than negatives from it. You should do a writeup after you test sctp and quic, I'd be pretty interested in reading it and I bet others would too.
My understanding is that http/2 completely fixed HOL blocking .If any streams has any issues ,server or client can reset that stream independently [1]. Isn't that the case ?
There's http1's head of line blocking.. and tcp's head of line blocking. http2 fixed the former.. the latter applies to any tcp connection--including http 1.x, http2, etc (anything that uses tcp).
In other words.. he's saying, because HTTP 2 is multiplexed, tcp's head of line blocking becomes a larger problem. Which is true.
I wonder why SCTP isn't more widely-supported. It seems like a lot of Internet traffic is message-based rather than stream-based and would benefit from a more appropriate protocol.
Seems to me that the basic Internet protocols got more or less frozen in time in the 90s, not because they had reached perfection, but just because it became too difficult to support new ones. With ubiquitous firewalls and NAT devices, anything that didn't fit into the existing TCP/UDP system would fail to work with so much of the Internet that people just wouldn't bother with it.
It really is unfortunate that we ended up with UDP and TCP as the only choices. There are so many more things you might want outside of "unreliable packets" and "automatically recovered streams."
The list of firewall-punchable level 4 protocols is frozen in time to just TCP and UDP, and it's sad at first glance, but it's not as bad as we think:
* Most ISPs don't mangle UDP, and if you care enough about perf to be using UDP in the first place, you can afford to figure out who mangles UDP and fall back to TCP for those networks. It's a very small number of networks.
* Any TCP/SCTP/etc inspired protocol you can think of can be implemented on top of UDP. Its header is only eight bytes long and it adds minimal overhead. UDP socket semantics are simple enough that you can treat them as a slightly smarter IP socket and build your protocol as if you were running directly on top of IPv6.
* There's a huge advantage to being able to implement your protocol in userland since you get fine grained control over congestion and connection logic. There's a huge disadvantage too, since it's actually hard to implement that stuff yourself. But for the people who are large enough to build their own protocols (FB, Google, etc), it's worth it.
* You can even encapsulate SCTP inside UDP to punch through middleware firewalls if you want to. I don't know of anyone doing that for real in the wild, but there's a RFC for it (RFC 6951).
+1 for your userland comment. I think that's a wildly under-appreciated element of UDP encapsulation.
The fine grained control over congestion and congestion logic is cool, sure. But more important in my mind is the fact that there's a cross-platform API available and you can rapidly make changes to how the protocol works without needing to require a specific kernel version or a specific operating system.
There's a whole lot of really cool frameworks for SDN to allow people to work at lower levels, but that's the problem - a standard API that everyone can expect to be widely available hasn't really emerged yet.
At long last, many of the 'hacks' for dealing with the ugly world of NAT etc have become standardized.
E.g. "ICE" provides a negotiation protocol to make a couple arbitrary machines able to talk to each other directly, including the NAT hole punching part of things. In the limited % of cases where there's no known way to punch a hole through NATs, an automatic fallback (using a relay server) is provided as a part of the protocol.
It's not elegant, at least we don't have everybody re-inventing the wheel. And, good results when ICE is used can become part of the criteria now used to judge the quality of NATs and other evil middlebox devices.
Genuine question, other than NAT punchthrough what does UDP not include that you would want? It seems like unreliable packets are the foundation for anything else you'd want to build into a higher level protocol.
Just to clarify, UDP is a great primitive, it's about as close to how the underlying hardware works as you can reasonably get for a userland-accesible protocol, and anything that could work over the Internet in general can be built on top of UDP. I'd just like things to be a bit more "batteries included" as the Python folks would say. Consider, for example, that even TCP is unnecessary, as you could implement it on top of UDP, but we all benefit from having it built in.
Anyway, I'd love to be able to pick and chose among:
1. Packets versus streams.
2. Reliable versus unreliable.
3. Immediate delivery, or delivery in the same order as data was sent.
Right now, we have UDP, which is packets, unreliable, immediate, and TCP, which is streams, reliable, delivery in order of transmission. But all eight possible combinations make sense (albeit some more than others), and it's unfortunate that we have to reimplement stuff that already exists if we don't fit within the two choices we have.
And if we're going beyond what TCP provides, I'd really like to see:
4. Choice of ACK versus NACK based loss notification.
#1,2,3,4 are all pretty straightforward to build. I've done that more than a few times. There's a bunch of parameters that tend to increase complexity(how many ordered streams do you need, additional data associated with an endpoint, etc) that makes me think a one-size fits-all would be hard to do.
#5 should really be done at the framing layer, otherwise your frame delimiters can also be corrupted. It's also very tied to physical layer and probably better off not being configured in platform agnostic software.
No opinion on #6, I've not deal with it much AFAIK.
I agree they're largely straightforward, but there would be a huge advantage in having one standardized version of each one rather than a million different buggy implementations.
I also missed one, which is hard to get right: rate limiting. Especially if you want to coexist with TCP rather than drown it out (or be drowned out by it).
I've actually been looking into (and started using) SCTP. It's pretty good, except some places seem to have it turned off. In particular, Windows has little support.
I'm glad to see they came to the sensible conclusion : "Multiplexing on top of TCP in overall fails to deliver the advantages it is assumed to provide."
It depends entirely on the use-case of course. Back in the J2ME days we multiplexed multiple connections over a single TCP stream because certain shitty phones coughnokiacough would pop up a permission dialog for every. single. connection. On startup our app would need to connect to several services and the user would be spammed with permission dialogs. Multiplexing the TCP streams solved that issue.
But to reach that conclusion they had to make assumptions, and yet their conclusion has washed away these assumptions so as to state a solid fact. Not very trustworthy IMO.
I don't disagree with the conclusion but from my understanding it was made entirely on a theoretical basis. I would be interested in seeing an implementation of multiplexing benchmarked against separate connections.
There are also (low-traffic) use cases where overrun of receivers is unexpected so you don't need an application-level ACK mechanism, just notification of overrun so you can handle it, e.g. by breaking up and restarting the application associated with the overflowing stream.
> I'm glad to see they came to the sensible conclusion : "Multiplexing on top of TCP in overall fails to deliver the advantages it is assumed to provide."
So basically the only thing the HTTP/2 crowd could come up with as a real advantage over HTTP/1.1 (discounting ofcourse that HTTP/1.1 supports pipelining), which is a massive source of complexity, which will also be a massive source of bugs, does not provide the benefits it claimed it would.
Can we just call off HTTP/2 yet? It was pushed by Google to cover their needs and agenda, without any respect for the protocol's history. It had a bunch of riders introduced with misleading language with regard to privacy violating semantic changes (For instance HTTP/2 cannot do regular HTTP, only HTTPS. You can no longer deploy private apps on your LAN without registering with centralized internet registries such as DNS and CAs. Etc etc).
Basically HTTP/2 was Google's attempt at making it easier for them to keep their huge amount of tracking cookies on every single HTTP(S) request across the internet, without it causing too much of an impact on users.
You know what? If I have 200KBs of Google-cookies tracking me, I want those cookies to impact performance. I want to know something is up.
Can't we just say "HTTP/2 considered harmful" at this point?
I'll be disabling it in all my browsers which support proper user-managed configuration. Needless to say, that excludes Chrome.
What on earth is this comment even about. First of all, HTTP/2 has a lot of advantage over HTTP/1.1, like server push. Saying it doesn't provide the benefits it claimed is wrong. It uses a single TCP connection, which for websites that require a lot of them (hint: lots of people nowadays do. You've got the webapp trend to thank for that) actually IMPROVES things. TCP overhead is not negligeable. And then there's the header compression algorithm that's included in HTTP2 that further helps getting size down.
HTTPS-only is NOT a protocol limitation. It was pushed by BOTH Google AND Mozilla, in the name of security, like all the new features coming to the web (need webrtc ? Https. Want the camera ? https. Want https ? https). I'm not actually too sure I like that trend either, it feels like shoving candies down my throat. I like candies. But not like that.
However, saying this is a Big Google Conspiracy is ridiculous. I mean what the hell does cookie have to do with anything.
PS: On a separate note, could we get 10.0.0.X, 192.168.X.X, 127.0.0.1 and the file:// protocol counted as "Secure" please ? It's a pain to have to create a self-signed certs just to develop stuff.
The point is, it's also outside the reach of the public internet, so you can't get "proper" HTTPS anyway (unless you get a custom root CA). I fail to see the danger added in making this ip "Secure". Suddenly, a MITM-ing actor could modify the responses to do some webrtc and ask the user for his geoloc. I don't think it's that much more dangerous.
And let's not mention the fact you can get a free HTTPS with let's encrypt, MITM an existing 10.0.0.X connection and serve a 301 to your https-enabled domain name.
My use-case is rather simple : I host a few things at home, and I've had to install a root CA on every device just because I can't do webrtc otherwise.
I could get a letsencrypt certificate for a domain name, but honestly that sucks.
So.. first you jump to the conclusion that because this guy asserts TCP multiplexing is not favorable for his use case, you decide that all TCP multiplexing is useless and does not perform so HTTP/2 must be worthless.
Then you complain that you can't use HTTP/2 for some potentially useful cases.
Then you proceed to accuse Google of pushing HTTP/2 for improving the performance of their websites, effectively conceding that HTTP/2 does improve performance of multi-resource webpages.
Then you argue that it's actually a good thing when a multi-resource webpage is slow.
I don't think so. On the sites I worked on, adding SPDY gave a nice double-digit& load time benefit.
HTTP/2 does not require HTTPS. But all browsers decided this was a good thing, so browser require HTTP to enable HTTP/2. And you do not need to get a centralized CA to give you certs; just use your own.
HTTP/2 makes parsing way faster, which can save significant resources. It'll also cut down an ambiguity which might prevent some security issues.
I think it really depends on the use case. My current work project is multiplexing on one port with a separate port for critical control commands. It seems like a good design, but maybe I'll make another writeup like this in a year when I realize it was a terrible decision.
Google is aware of that issue, that's why they explored SCTP first and now QUIC as alternate layer 3 protocols to replace TCP in the future.