Hacker News new | comments | ask | show | jobs | submit login
HTTP-over-QUIC will officially become HTTP/3 (haxx.se)
498 points by payne 71 days ago | hide | past | web | favorite | 187 comments



We were doing testing several years ago with a custom UDP based protocol to overcome some of the TCP issues, and ran into a perplexing problem.

To meet the data rate, we were sending datagrams that were 64KB in size (there was no system call at this time to send multiple datagrams in one go).

The datagrams were being fragmented into MTU sized fragments by the OS and reassembled on the remote end. However, with the data rate and the fact that there is only a 16 bit IP ID in the IP header, the fragments would often get missassembled on the receiving end and passed up to the UDP layer. Usually, the UDP checksum would detect the problem and throw the datagram out. However, since the checksum is weak and we were sending a lot of data, it would sometimes pass the datagram up to the application layer.

So, we would get corrupted data on the receiving end once in a while. It took quite a bit of time to figure this out, of course.

The solution was to implement dynamic MTU discovery along with the protocol so we could avoid the fragmentation.

The custom solution is still being used 20 years later!


Sounds like a bad fragmentation implementation. At 64KB datagrams you have to send over 4GB of data before you need to recycle an ID

Even if your packet ids are assigned poorly, on the receiving side Linux has a heuristic that takes in to account reordering to minimize the chance of reassembling bad packets. Basically if packets with the same frag id are too far apart then the whole datagram is chucked:

https://marc.info/?l=linux-netdev&m=113267833031122&w=2


Fragmentation by the IP layer is very unreliable in today's internet. I've seen plenty of consumer routers that throw away the whole datagram if a fragment arrives out-of-order, and who knows what kind of hell corporate middleboxes and other similar systems do to them. Pretty much every protocol these days goes to great lengths to avoid upper level fragmentation.


IPv6 helps in that it mandates a MTU of at least 1280 bytes. In addition, in IPv6 only source nodes can break a packet into fragments - intermediary nodes such as firewalls and routers cannot. Sending back a ICMPv6 Packet Too Big for packets larger than the MTU is a must, which helps with path MTU discovery.


Ok, so there have been literally decades spent in managing and optimising TCP to behave in various network conditions, from modem lines to multi-gigabit fiber, in all kinds of congestion conditions, lossy-ness, and other hairy edge cases of real physical networks.

So, what's the point of ditching all that and reinventing HTTP over a thin wrapper (UDP) over (lossy) IP? Has there been a huge theoretical breakthrough which simply can't be applied to TCP?


QUIC does things TCP will never do. Firstly QUIC is a combination of SSL + TCP. This allows it put the SSL handshake into the TCP handshake - thus reducing the overhead of starting an SSL connection. It also leverages SSL session reuse to avoid the TCP 3 package handshake completely. As others have mentioned, QUIC is optimised for something that HTTP/2 does often that TCP does not support very well - multiple child streams over the one connection.

In general, it is true that TCP is highly optimised, but it is only optimised for one use case: a long lived single flow over a low error rate connection to a single destination. That was the predominate type of connection back in the FTP and early web days. But is hasn't been that way for a while now. HTTP is short lived, video goes to multi destinations, HTTP/2 and multi media in general has multiple flows. Wireless only appears to be low error rate because the physical layer had to cover up it's shortcomings otherwise TCP's performance became abysmal.

TCP hasn't remained king of the mountain because its fantastic protocol that can't be improved. On the contrary - looks to be the simplest thing it's inventors could come up with. It even lacked round trip time measurement - which is something I suspect that would have killed it if an extension hadn't saved it. TCP has remained king of the mountain purely because it appears to be near impossible to replace. Now that Google has done it, I expect we will see a lot more experimentation in this area.


Nice review. Just a theoretical question: Wasn't IPsec supposed to do the encryption right in the IP layer? I wonder how this compares with the handshake roundtrip time needed (for IPsec, for TCP).


Well, the problem is complex. For security you need integrity, authenticity and anonimity/secrecy. Crypto of course is the way we implement all that, via keys and other kind of information theoretic tokens that somehow build upon a pre-shared secret. You need an anchor, a mutual trust foundation to build your secure channel upon.

If you want to put security on/into IP, you suddenly need the whole key management thing, and if you want it to be automatic, you need to somehow cobble together the assignment, provisioning, allocation, publication, revocation of keys with IP addresses (or hostnames, domain names, and you just put DNS into IP).

Eventually it means either you need manual key management (thus IPsec becames a hard to manage old rusty and poorly supported concept from an old era) or you just descended into the recursive pit of all hells of circular dependencies. Sure, it can be done, we can put keys into DNS, and we can use that, but then we're back to QUIC or DTLS basically.


The issue is that IP is generally handeled by the OS. This means that IPsec also needs to be handled by the OS.

The thing is, how you set up (mutual) authentication is very much an application level thing. Do you want certs, an SSH key, passwords, hardware tokens, SIM based authentication, or something else? These aren't things the OS can decide, and they aren't so simple we could tell the OS what we want with a few flags.


Doesn't IPsec suffer from legacy cruft and outdated implementations?


> Wireless only appears to be low error rate because the physical layer had to cover up it's shortcomings otherwise TCP's performance became abysmal.

It then seems that optimized protocols like QUIC could benefit from disabling this lower level error correction.

Is this at all possible? Do the wireless physical protocols allow for such a thing? Do the operating systems offer it as a flag?


The Wi-Fi extender I'm using allows disabling some option related to that. TCP becomes absolutely unusable.


Genuine question: why not SCTP then?


I wondered the same - if we are going to have QUIC over UDP, why not SCTP over UDP (https://tools.ietf.org/html/rfc6951)?

I did a quick read on the subject and main argument for QUIC seems to be zero roundtrip session establishment - using TLS session resumption. SCTP uses a 4-way handshake.

Also found this comment from 2015: https://news.ycombinator.com/item?id=9639824, and this ietf draft: https://tools.ietf.org/html/draft-joseph-quic-comparison-qui...


Looking at this comment, it may be to do with ISPs refusing to route it properly: https://news.ycombinator.com/item?id=18429404


TCP is a general stream oriented protocol that must cover a variety of use cases and situations. That's why there are so many OS knobs to tweak it if you have a special use.

Since HTTP is the use case here, it is possible to create a protocol that is optimized for it, whereas TCP cannot be optimized specifically for HTTP.

Source: I've implemented a reliable, stream oriented protocol over UDP which has been in use for many years.


UDP is the new IP. You can't practically have your users install an upgraded OS kernel with TCP rate control improvements, you can't fix queuing in their NAT gateway, and you can't get SCTP working through their ISP. So, you do it all in UDP. (Your servers can handle the streams in kernel or even accelerated by the NIC chip, or in a userspace IP stack e.g. DPDK.)


OS modifications are not strictly required for new L3 protocols, though some new APIs would be nice to handle it more ergonomically. You can receive/send any IP packets with standard socket APIs.

ref: https://docs.microsoft.com/en-us/windows/desktop/WinSock/tcp... https://linux.die.net/man/7/raw


Biggest problem with multiplexing over TCP is mixing latency sensitive flows (like getting HTML to render) with high throughput flows (like downloading a YouTube video).

You can try and diddle with chunk size and message prioritization in your application to reduce HOL blocking of low latency messages behind chunks of high throughput data, but your OS will buffer everything in one big queue anyway (because TCP is a single stream and flow control applies to that entire stream as a whole).

How do you fix that? Well you can reduce socket buffer sizes on your TCP connection, and that will fix latency... but small buffer sizes mean your total throughput is limited by round trip time (because of the way the TCP receive windows and positive acknowledgment work). If you're working over a LAN, things will be fine, but over the Internet you're hosed.

Of course, the solution really is to have separate long-lived TCP connections for different kinds of flows, so you can set buffer sizes appropriately... but that's impossible for your browser (which is responsible for opening connections). It has no way to know whether a GET request triggered by an <img> tag is going to yield a 100 byte response or a 100MB response.


Does HTTP2 server push help solve this? Is there a way to send headers for things you’re about to push before pushing them, then letting the browser switch some subset of your resource loads over to another TCP thread without interrupting things?


It doesn't, and there is not.


TCP has some real issues, MTU discovery is either slow or not enabled or not very smart on a lot of clients. Google, for some reason, doesn't want to fix tcp in Android, instead they went with implementing tcp inside of tcp (http/2), and since that ran into the obvious problems, they've realized that tcp over UDP is a better choice. As far as I can tell, they still haven't enabled TCP MTU discovery in Android though. Or build apis to allow applications to influence packetization and congestion control -- which is what is probably needed. The java apis for TCP are fairly inflexible too, to set most socket options you have to call into c code, and hunt around to get the FD.


h2 is not tcp on tcp.

h2 failed because it's mux everything into 1 single tcp connection, and it only takes 1 failed tcp to block everything in line.

h2 really need to do demux. But h2 actually can technically do M:N connections, but browsers do only 1:N.

Also h2 is a failed promise. Most sites deliver web assets and page and js in different host, also ws/wss in a different host. Thus renders h2 advantages complete useless.


> “h2 failed”

Just curious, what do you mean? Quickly Googling this, it looks like 1/3 websites use HTTP2 today, which is darn good for a relatively new protocol.

https://w3techs.com/technologies/details/ce-http2/all/all


By failed I mean h2 did not fulfill its design goal and technical potential.

Most sites use h2 by just wrapping http/1.0 behind some h2-enabled CDN or upgrade to latest nginx and call it done.

What about h2 push? Priority? WS inside h2? Nope. No one use them obviously.

I mean gRPC utilized more h2 features than most h2 browsers/CDN do.


I mean, it’s just the beginning right? You’re making it sound like it’s the end of the road for HTTP2.


Jetty/CometD has decent support. You can use HTTP2 as a transport for push notifications. You can also specify additional resources like scripts, css, or images to be sent on the first request.

I am less worried about what most sites do and more interested in the features offered by implementations. A huge number of sites don't even use https.


The idea is good, but static assets and dynamic content (APIs, pages, etc) have different scaling models. I don't think mix them in a single transport layer have much real world usage.


That doesn't sound like failure to me. Newer generations of standards and protocols have features that won't be used by everyone. The goal is not maximum feature adoption, the goal is to reduce latency. There's a tradeoff between implementation complexity and benefit that changes as you incorporate additional h2 features into your web site. At some point you decide the additional features do not provide enough benefit to support the additional implementation complexity. This is normal and expected, and I'm surprised that someone would interpret this as a failure of h2.

Again, the goal of h2 is not to maximize usage of features in the h2 protocol. So judging h2 by that metric does not make any sense. I could just as easily complain that HTTP is a failure because few sites use more than just GET/POST, or because many of the headers defined in the HTTP standard are not commonly used.


I used push with the help of Caddy. Pretty easy.

I'm curious to hear what you mean by WS inside h2, not sure how that's different. WS is only http for the first part i.e. the upgrade request.


Can't always correlate usage share with performance, especially on things that are extremely hyped.


Googlebot never adopted HTTP2, only SPDY


How is multiplexed streams with individual windows on each substream not tcp on tcp?

Sure, there's some other stuff in there, server sent goaway in particular is a great way to indicate the server intentionally closed the connection, but the primary objective is to get around congestion control -- which could have been better solved by just adjusting congestion control to pool by destination IP in clients Google could control (at least Android) and on servers Google could control (at least their servers)


TCP has major issues in low-latency environments. It has too many timers and they are too coarse for 50us RTTs and 100gbps bandwidth. QUIC as a transport for gRPC can solve a lot of this. What we know for sure today is that Linux distributions, out of the box, are garbage for TCP in datacenters. You can tune some of it away, if you happen to know where the knobs are, but some things like microsecond packet time stamps you have to patch. Switching to a UDP transport means you don’t have to argue with Linux kernel maintainers in order to make progress.


First actually useful and technical answer I've seen in this thread.

But... which use cases of HTTP need us timers? It's not like gmail or other bloated web apps will load faster on my phone because of QUIC?


The timers are in TCP not HTTP. TCP only ACKs every other packet. If the stream pauses after an odd number of packets, the sender has to just wait for the receiver to perform a delayed ACK, for which there is a timer on both sides. The sender infers that the last packet has been lost if the ACK doesn't arrive after a while. The sender has a thing called the retransmission timeout (RTO) and the receiver's delayed ACK timeout (ATO) sets a lower bound on the RTO. Modern TCP stacks have a thing called the tail loss probe that just sends a retransmit early (before the RTO) if the stream has stopped at an odd boundary. This is a bit of a hack. All of these things affect client-observed latency since obviously the client can't proceed until it gets the last packet. On mobile this is pretty important because mobile networks have massive packet loss.

Anyway, that's mobile. In the datacenter all of these timers are way too high, by several orders of magnitude. 1ms is an outrageously long RTO in a datacenter fabric but 200ms is the minimum RTO allowed by the TCP RFCs.


One issue is head of line blocking in TCP when multiplexing requests over a single connection. That problem can't be solved without something like QUIC.


To be clear, this became a bigger issue because HTTP/2 moved to “1 TCP connection per origin with all requests multiplexed on it”. There has always been head of line blocking in tcp, however H2 compounded it since it now impacts all requests to the origin instead of just a few requests in one of four or one of six tcp connections to origin. H2 effectively moved the problem to TCP (really just made underlyubg problem more impactful). Quic is an attempt to solve the TCP problem


So HTTP/1.1 worked fine for around 2+ decades.

Google(tm) HTTP/2.0 was unasked for, tried to reinvent TCP, found buggy and fundamentally broken after less than a year.

So now Google is pushing HTTP/3 to solve the bugs they made with HTTP/2, by reinventing even more of the IP-stack. What could possibly go wrong?

Can someone please tell Google that protocols are not Chrome-releases?

They need to work and stay working and supported for decades. You can’t ship a new revision every 6 month.

Edit: Correct HTTP/1.1 longdetivity.


That's really not what happened. Spdy (HTTP/2) and quic were in development simultaneously at Google before the standardization process began in 2012. HTTP/2 solves a bunch of performance challenges around running web applications over HTTP (a protocol designed for simple, static documents). Part of this was adding pipelining and multiplexing to solve the HTTP/1.1 problem of head-of-line blocking. (A problem which previously people solved by opening a bunch of connections and even using different subdomains so they could bypass browser limits). And to be clear, you can still do that in HTTP/2, it's just less necessary.

However there's only so much you can at the application level. Some problems need to be fixed at the transport level, and there's where QUIC comes in. It's not fixing problems in HTTP/2 (at least not compared to HTTP/1), it's making things even better for high performance web applications.


A problem which previously people solved by opening a bunch of connections and even using different subdomains so they could bypass browser limits

I suppose that's one way of putting it.

Currently, the home page at www.google.com wants to do twenty-five requests in order to show me an interface consisting of a text input and two buttons. Google decided to fundamentally redesign HTTP to accommodate their own inability to impose any kind of discipline on their programmers. I don't see this as a good thing.


> Google decided to fundamentally redesign HTTP to accommodate their own inability to impose any kind of discipline on their programmers

1) That is a ridiculous statement with absolutely no evidence to back it up.

2) The number of requests that Googles homepage does in order to load has _nothing at all_ to do with whether or not solving head of line blocking problems for the rest of the web is a good thing.


> That is a ridiculous statement with absolutely no evidence to back it up.

Press Ctrl+U on any Google page.

> The number of requests that Googles homepage does in order to load has _nothing at all_ to do with whether or not solving head of line blocking problems for the rest of the web is a good thing.

No, but it's evidence you claim doesn't exist.


Google's home page doing 25 requests does not prove that Google spent the last 6 years and millions of dollars developing new protocols because they unable to reduce the number of requests.


Not the requests per se, but the HTML source. But you're right, it says nothing about the reasons for developing new protocols, also not anything about inability, it could also be unwillingness.

It takes large team to produce a pages so bad as the Google landing page, blogger etc. For me, just about anything Google I looked at is to a well crafted website what elevator music played with a 1000 man orchestra is to someone with a guitar singing a really good song. The scale and the available resources makes the result worse to me, not better.


The number of requests that Googles homepage does in order to load has _nothing at all_ to do with

It has a lot to do with the desire to work around limits on the number of in-flight connections allowed to a single host, which is one of the primary things HTTP/2 is designed for.


Google has spent at least 6 years (and counting) working on SPDY / HTTP2 / QUIC on design, experimentation, implementation, and specification tasks. This is low-level, complex work that requires a combination of experience and talent. This is expensive work - and there was never any guarantee it would pan out. I think it's unreasonable to assert, without evidence, that they did all that just because parts of their organization are incompetent and they refuse to fix it or that the problems they are trying to address have some sort of trivial solution.

It's also pretty unreasonable to point at www.google.com and say that it makes too many requests and it could be trivially fixed to use fewer. But, let's assume that the problem you identify actually is the case: Google is a huge, cash rich organization - if they can't figure out how to enact the change you suggest, how could any other organization? And if HTTP/2 and QUIC (HTTP3) solve this problem for Google, and since they are making it an open standard, also for other organizations, why is that a bad thing?


> That is a ridiculous statement with absolutely no evidence to back it up.

I’d hate to bring it up but here goes: SPDY was designed specifically to load the Google homepage quickly on typical American DSL MTU configuration, even with all that JavaScript tracking enabled.

Turned on it’s head: SPDY was designed to make google’s wide tracking of end-users impact page-load times as little impact as possible, to avoid it being a cause of end-user dissatisfaction.

SPDY was extremely tailored to a very niche need which first and foremost serves Google.


The thread here is about HTTP3 - ie, HTTP2 over QUIC. HTTP3 has features that are going to benefit high latency, high loss connections a lot. Things such as speeding TLS negotiation, and only having to do it for one connection; better handling of packet loss; forward error correction (if it works).

I see you don't cite any evidence that SPDY was miopically designed for American DSL connections, but regardless of that, HTTP3 is going to benefit a wife range of connection types.


3 decades? 1.1 was released in 1997.

Networks change as the world evolves and supports more people and applications, so new protocols are expected. HTTP/2 was great progress at the application layer, and QUIC is now about improving the transport layer.

Can you clearly list the bugs you claim HTTP/2 has?


> 3 decades? 1.1 was released in 1997.

Fair enough. Corrected.

> Can you clearly list the bugs you claim HTTP/2 has?

They seemed fairly well outlined in the post I was replying to, no?


The only issue raised in the post you replied to was that TCP head-of-line blocking can be a bigger issue in HTTP/2. Claiming this makes the protocol "fundamentally broken" is a bit of a stretch.

Plus, this wasn't "found after less than a year". It was a known drawback from the very beginning, when the protocol was still SPDY, and deemed an acceptable tradeoff.


That is not a bug with HTTP/2. It is designed to be more efficient by using a single TCP connection, thereby magnifying any efficiency issues that TCP had all along. These issues are not only about HTTP but include the massively increased amount of devices, network types, security features, and general throughput over the years.

Perhaps a new version of TCP could've been made, and that's a valid concern, but it's a slow moving standard that takes decades to alter. QUIC evolved faster on UDP so that's what we have. What strategy do you suggest for improving TCP faster than the current process?


I agree with what you have to say, but just want to respond to last second paragraph:

QUIC basically is a new version of TCP. Yeah, it's based on UDP, but, that is just an implementation detail. If it weren't for various poorly behaved middleboxes, QUIC could just sit on top of IP like TCP does. But, it can't - so we hold our noses and put it on top of UDP and we end up with something that actually works.


> Can someone please tell Google that protocols are not Chrome-releases?

This is why browser marketshare matters. One player holding such a dominant position gives them according control over how web standards evolve.



> Google(tm) HTTP/2.0 was unasked for, tried to reinvent TCP, found buggy and fundamentally broken after less than a year.

How vile of them to force a purely optional protocol on us.


> purely optional

And then one day Google declares HTTP/1.1 deprecated, lowers the score of sites without HTTP/3, and removes support for HTTP/1.1 from Chrome in a week. You're personally free to not use it, but I'm not sure that everybody else will while you'll be the one responsible for supporting it on your website.


They might lower your search ranking, but dropping support won't happen as it would force people to ditch the browser. New things that don't support http2 are still being made.


But moving the solution away from TCP into something new (in this case userland UDP-based QUIC) would make sense if (and probably only if) the overhead of QUIC connections + request bookkeeping is lower in memory and CPU usage (ideally both) than just establishing multiple TCP connections.

Right?


TCP flow control is generally applied per-connection. So, a web browser that creates 6 connections to load a website is going to get a greater share of available bandwidth than an application that just uses one. And what will get greater bandwidth still is if the website owner can trick the browser into opening 12 or 18 or more connections by sharding their site over multiple domains.

"Just open more connections" has issues. QUIC making it less necessary to do that has value even if it didn't save memory or CPU usage.


The need for maintaining flow control still exists, both in QUIC and TCP, it doesn't just vanish.

Also, big(ish) routers are aware of TCP and can adapt as the load changes. They literally won't know what to do with a huge increase in UDP traffic.

I'd make an educated guess that mass deployment of QUIC will prove to be pretty awful until it's hacked to look more or less exactly like TCP with multiple connections, just with more overhead.


Afaik, the bigger the router, the more likely it is to just route ip packets. I'll make the educated guess that deployment of QUIC will be just fine as routers already know how to handle UDP and IP just fine.


> They literally won't know what to do with a huge increase in UDP traffic.

Randomly drop UDP packets.


My understanding is multiplexing over an existing connection also provides time savings as new data does not need the full handshake and ramp up a completely new connection would.


> There has always been head of line blocking in tcp, however H2 compounded it since it now impacts all requests to the origin instead of just a few requests in one of four or one of six tcp connections to origin.

I feel like you're conflating two very different things into "head of line blockings".

Head of line blocking in HTTP/1.1 was that HTTP/1.1 cannot transmit responses out of order, so a slow response "blocks" any responses behind it. HTTP/2 solves this by being multiplexed (effectively, it tags the requests & responses s.t. you can match out of order responses back to the appropriate request). This isn't necessarily that the network can't handle it; a request that might be easy for the server to handle (e.g., a static asset) can get caught behind one that is not (e.g., one that entails a slow DB query).

The other "head of line blocking" that I think you're getting at is just network loss or saturation. If the network can't deliver packets, I don't see how having 6 connections is going to help: you're going to have to wait for six different state machines to determine that the link is saturated, back off, etc.

Regardless, my understanding is that QUIC / HTTP/3 is still a multiplexed protocol; I would think it would behave very similarly. Nonetheless, Google's announcement of it backs up your claim[1]:

> Like HTTP/2, QUIC multiplexes multiple streams into one connection, so that a connection can serve several HTTP requests simultaneously. But HTTP/2 uses TCP as its transport, so all of its streams can be blocked when a single TCP packet is lost—a problem called head-of-line blocking. QUIC is different: Loss of a UDP packet within a QUIC connection only affects the streams contained within that packet. In other words, QUIC won’t let a problem with one request slow the others down, even on an unreliable connection.

But what here makes QUIC think that those other requests have any hope where the first didn't? To me, this approach seems like you're just spamming the link with packets and hoping for the best; those packets could just end up dropped like the other ones were. What makes QUIC think this approach will have any benefit? (It could, perhaps, if the link is just going to drop one packet and we can get right back to sending. I'm not sure that still conveys any benefit over TCP. Not saying QUIC is wrong, just that I don't see a decent reason to buy into it.)

(This whole thing makes me think there are two problems here: there's the actual network connection between two endpoints, and what its state is, how lossy it is, its PMTU, how much data we should allow in-flight between those two endpoints, etc. But then there is also logical streams, such as HTTP request/response pairs. It almost seems like to me any data about the link, such as PMTU/lossiness/window, should be shared between all TCP connections, and the individual logical streams should be otherwise cheap/easy to make.)

[1]: https://cloudplatform.googleblog.com/2018/06/Introducing-QUI...


> But what here makes QUIC think that those other requests have any hope where the first didn't? To me, this approach seems like you're just spamming the link with packets and hoping for the best; those packets could just end up dropped like the other ones were. What makes QUIC think this approach will have any benefit? (It could, perhaps, if the link is just going to drop one packet and we can get right back to sending. I'm not sure that still conveys any benefit over TCP.

There are a lot of ways to lose just a single packet. One way is a router is overloaded and so it throws some packets away. Another is some brief interference on a wireless connection.

I think the phrase "spamming the link" is incorrect. QUIC does flow control to determine how much data to send. Once it's decided that the link can take a certain amount of data, it will send it. So, say it sends packets 1, 2 and 3. But, let's say that packet one is lost but 2 and 3 get through. With QUIC, the receiver can potentially do something with packets 2 and 3 with TCP, the receiver couldn't. Regardless, QUIC will respond to the packet loss by updating its flow control information - I haven't read anything that suggestions that QUIC responds by "spamming the link".

> This whole thing makes me think there are two problems here: there's the actual network connection between two endpoints, and what its state is, how lossy it is, its PMTU, how much data we should allow in-flight between those two endpoints, etc. But then there is also logical streams, such as HTTP request/response pairs. It almost seems like to me any data about the link, such as PMTU/lossiness/window, should be shared between all TCP connections, and the individual logical streams should be otherwise cheap/easy to make.

You've basically described what QUIC is in this paragraph


> You've basically described what QUIC is in this paragraph

You would need kernel support here, though, would you not? (And you would need a generic protocol. I have admittedly not studied QUIC thoroughly yet, but my impression was that it was strictly HTTP / not generic.) Without kernel support and/or being generic, I don't see how it's really different from HTTP/2 w/ a single TCP connection.


QUIC is a general protocol, it's just that the only significant customer for the fancy features today is HTTP. Speaking QUIC means accepting a non-trivial connection refusal rate (for HTTP you fall back to HTTP/2 or 1.1) because some corporates block it, so if you don't want the extra features you probably wouldn't want QUIC.

This document is about doing HTTP on QUIC, so it's renaming itself HTTP/3, the main QUIC document is as you've described it 'generic'


> I feel like you're conflating two very different things into "head of line blockings".

I wasn't as clear as I wanted to be. Yes HTTP/1.1 has head-of-line blocking on a single HTTP Connection (read TCP connection). Attempts to solve HTTP's head-of-line blocking like HTTP Pipelining didn't receive widespread adoption for many reasons, primarily middle ware boxes screwing things up.

H2 achieves logical parallelism by multiplexing requests/responses over a single connection. H1 solved it by allowing 2 (then 4, then 6-8) parallel TCP connections to a single origin.

While you might think that having 6 parallel TCP connections makes you 6x more like to encounter network loss, saturation, and dropped packets, that doesn't seem to be the case.

The best paper to date on this is "HTTP/2 Performance over Cellular Networks" [1]. In rigorous testing, they found that in slower/congested networks, H2 is actually worse than H1, sometimes by a lot. If 1 TCP connection in H1 stalls, no big deal, you have 5 others to that host. If 1 TCP connection in H2 stalls, you are stalled.

[1] https://www.akamai.com/kr/ko/multimedia/documents/technical-...


Fixing bugs and experimenting with new protocols will be easier in user space than waiting for operating systems and routers to get fixed and their users to deploy them. Look at all the problems TLS 1.3 has faced with busted routers and enterprise MITM boxes.


Udp is preferred for content that can be lossy. The overhead of tcp for streamable content, especially over WiFi, is too high. I ran into this issue by trying to have a simple video server through nginx on a DigitalOcean droplet. Wouldn't work well over WiFi at range, but YouTube would work easy on chrome.


I think that has a lot more to do with YouTube having an a very optimized video player, and using ABR streams. That works fine over WiFi in general. If you are doing MP4 pseudo-streaming, thats really a completely different beast.

You don't really want to drop data for streaming video either, sure, in some cases it might not make a huge difference, but if you drop i-frames, you are going to have quality issues.


Problem is on wifi that the whole syn/ack pattern gets lost and has to retry. I could have probably optimized this by having a range, but afaik if any in the range fail it has to still re-send that packet and get an ack.

I'm not sure if HTTP/3 will have a way to do selective re-sending, but that is how very versatile lossy video streams do it. I-frames can get re-sent while normal frames can be lost.


At first I wrote a lengthy rant about not supporting QUIC as an OSI layer 4 protocol on its own merits and develop HTTP to sit on top of either TCP or QUIC. But I'm hoping that the people pushing this along are only creating a homunculus guinea-pig, and that once this gets adopted widely, will push to turn QUIC out into a real independent low-level protocol like it should have been.

Bundling HTTP and QUIC into one protocol is a bad idea because (1) it increases complexity, (2) it creates application-specific QUIC implementations, which will become a problem later if QUIC is adopted by anything else, not to mention the security holes/feature gaps in different implementations, etc, (3) it reduces the likelihood of the operating system supporting QUIC as a first class L4(OSI) protocol, which will further doom any network application that uses any method of IPC other than an HTTP request.

Honestly, the way modern network applications operate is embarrassing. People in the future are going to look back and wonder what was wrong with us.


But why would you ever want to use anything else than HTTP??

/s


I hope those future people think about backwards compatibility.


For anyone interested in understanding QUIC, here's a post I wrote a few years ago explaining the tech, the challenges involved and the intended fixes: https://ma.ttias.be/googles-quic-protocol-moving-web-tcp-udp...


SCTP should be the preferred protocol here. SCTP is

1. An internet standard 2. Already has kernel support 3. Was not developed by one company

And, most importantly,

4. IT'S ALREADY IMPLEMENTED IN EVERY MAJOR BROWSER!


SCTP is a good protocol itself, but there are lots of NAT devices/firewalls/other strange middleboxes that do not play nice with SCTP. These devices only recognize TCP and UDP, and they tend to drop everything they do not know.

This is not something very easy to fix - not as easy as fixing Cisco devices that hijack 1.1.1.1, or upgrade enterprise-grade MITM firewalls that don't know and block TLS 1.3.


Can't you just encapsulate SCTP in UDP? Isn't that the reason we have UDP in the first place? If a middlebox will pass QUIC, it should stand to reason that it'll pass SCTP over UDP, since they're both oddball UDP protocols.


Luckily, SCTP, like QUIC, has a standardized UDP encapsulation.


If you do SCTP in user space with UDP encapsulation, most of its benefits disappear (and you have to do that - also because of Windows).

It's standard only on paper. There is only one significant user space implementation (usrsctp). It's used both by Chrome and Firefox. I don't think it has much use outside of that. And, I don't think other browsers implement data channels (which require SCTP).

Browser implementations will probably never have to interact with kernel implementations (which are not really used outside of telecoms). There is really no reason to make them talk the same protocol. It's likely better to use two different protocols tuned to those specific uses:

https://tools.ietf.org/html/draft-joseph-quic-comparison-qui...

They will probably replace SCTP with QUIC also in WebRTC:

https://w3c.github.io/webrtc-quic/


You only need to do UDP encapsulation on the client side (and then again, you can easily fallback to the kernel acceleration on supported platforms). On the server side, SCTP has a BSD sockets API already, and with some creative ethernet bridging, you can convince the kernel to unencapsulate your SCTP packets (doing this right now in a project for WebRTC, so I know it's possible). This is a one-time investment that can be made open-source and can be done in userspace. Of course, a kernel patch to encapsulate SCTP in UDP shouldn't be that difficult either.

There's two sides to networking -- client and server. Your arguments may make sense for the client, but they do not make sense for the server, IMO.

> There is only one significant user space implementation (usrsctp)

That's right, but there are others (one was posted a while back) independent implementations, and the kernel implementations (BSD, Linux, likely others) are indeed different.


> There is only one significant user space implementation

Only one significant opensource implementation. Behind the scenes, SCTP is used to connect most telephone calls in the world.


The "IMPLEMENTED IN EVERY MAJOR BROWSER" refers to SCTP-over-UDP (as used by WebRTC).


You are forgetting to explain why the technology is preferable, apart from expressing that the technology is well-supported.

I know little about SCPT. Why is it preferable?


SCTP is a flexible transport protocol with a crazy amount of features. It is message-oriented and each message on a connection belongs to a stream (of which there can be many). Each stream is optionally ordered, but this is managed independently for each stream in a connection. My understanding is that you can tune each stream separstely from unreliable, unordered to fully reliable and ordered as your application needs it.

This looks to me like a strict superset of what QUIC is intended to solve. And it is already supported in all major operating systems.


More frustratingly, I can imagine a future where SCTP encapsulated in UDP, serving for what QUIC is going to end up doing, would have caused adoption of kernel-mode SCTP drivers and the addition of SCTP support in the damnable "middle boxes" driving this whole mess to begin with.


> 4. IT'S ALREADY IMPLEMENTED IN EVERY MAJOR BROWSER!

Google is not interested in what is implemented in every major browser. Google is interested in making sure that Chrome is the only browser of any significance on the internet and it behaves in a way that Google wants it to behave ( such as forcefully signing into Google services ).

That's why we got HTTP/2 ( which seems to be viewed as a flop at Google ) which is why we are getting HTTP/3.

Follow. The. Money.


What's wrong with SCTP?


I know-- right? Especially in light of UDP encapsulation of SCTP being "a thing"[1]. Instead, we're stuck with this hack.

[1] - https://tools.ietf.org/html/rfc6951


most system administrators are only aware of tcp and udp(that quic uses). its really hurts protocol adoption if it does not work for 3rd parties blocking/not handling it on th e network gear.


SCTP packets would need to be allowed through the networks. And networks don't bother if nobody is using it. So it's a chicken-and-egg problem.

But google is big enough to push through that problem, IMHO, as long as browsers fail back to HTTP/2 or HTTP/1.1, the widespread acceptance of SCTP would be a boon to e.g. VoIP and game developers. But alas it's too late. QUIC has been in development for six years now.


> But alas it's too late. QUIC has been in development for six years now.

So what ? We standardize on a hack because Deep Pocket ?

Not even mentioning competitive advantage for big cloud companies ...


> But google is big enough to push through that problem

Not really. There are hundreds of millions of home routers deployed that all do NAT. And they all don't support NAT for SCTP. And many do NAT with hardware support, so it's probably not even fixable with a firmware upgrade, which isn't even an option for a ton of unsupported devices anyway.

So, I think encapsulating in UDP is the only realistic option if you want to gain any adoption any time soon.

Also, SCTP has the same problem that TCP has in that the network can look inside the protocol, and thus you would get protocol ossification. While Google does this for selfish reasons, I think it is a really good idea to establish a protocol that is completely opaque to telcos and should ultimately benefit the public. Telcos really don't want to be dumb pipes, and they tend to abuse any power they get, as they have demonstrated time and time again, and the only way to force this issue is by simply making it impossible for them to see or manipulate anything at all. So, while we may have to live with the UDP encapsulation forever, and as stupid as that is, this at least ensures that anyone in the future can trivially invent and deploy new protocols, as it is trivial to masquerade anything at all as QUIC. The adoption of QUIC for the web has the potential to get all ISPs to fix things so that QUIC actually works reasonably reliably over their network. And the fact that as far as the network is concerned it's just UDP packets filled with random data ensures that as long as your new protocol is UDP packets filled with random data, that will work as well, even if you use completely different mechanisms for framing or flow control or multiplexing or whatever.


It has taken 20 years to get IPv6 adoption to where it is now. This takes amazing dedication and is a much more fundamental change. Why can't SCTP adoption be a similar long term project? A home router probably has a life span of less than a decade. So it would be realistic to get a majority adoption of SCTP within approximately 15 years if there were a bit of a push in that direction. QUIC has been in the making for 6 years now? SCTP was standardized in 2000. So we could be 6 years into this 15 year project by now instead. And that is not comsidering the time it will take to finish QUIC, build compatible implementations and deploy them.


> It has taken 20 years to get IPv6 adoption to where it is now. This takes amazing dedication and is a much more fundamental change.

It doesn't take any dedication at all, it only takes address exhaustion. Which is precisely why it took so long.

> Why can't SCTP adoption be a similar long term project?

Because there is zero incentive for Telcos.

> QUIC has been in the making for 6 years now?

And QUIC (the Google "prototype") has probably been successfully deployed to more devices than IPv6 by now?


hmm Facebook run their entire internal network on IPv6


Why are you mentioning this?


Why do people keep bringing this up. Here is how you encapsulate any network protocol in UDP.

    void encaps_udp(void *out_buf, void *old_pkt, size_t old_pkt_sz) {

      struct udphdr udp;
      udp.dport = UDP_DEST_PORT;
      udp.sport = UDP_SRC_PORT;
      memcpy(out_buf, &udp, sizeof(udp));
      memcpy(out_buf + sizeof(udp), old_pkt, old_pkt_sz);

    }
(Accounting for memory overflow in the real world of course).

Of course, this encapsulation for SCTP is even standardized in case it wasn't obvious that this is how you encapsulate protocols.


For one, it's not, because PMTUD and stuff.

More importantly: You are missing the point. Great, you have encapsulated your protocol in UDP. Now, what do you notice? Your ISP is throttling your protocol because for some braindead reason their traffic management has categorized it as some sort of unimportant protocol, probably P2P or something. Or they just throttle UDP in general, because who uses UDP besides DNS and VoIP? So, high bandwidth UDP usage is obviously a DoS, and we don't want that! If you are unlucky, maybe they even shut off your server because it is obviously part of a botnet?

OK, so enough people complain to their ISPs that SCTP over UDP doesn't work. What happens? Exactly, ISPs start putting in rules that "recognize SCTP over UDP". So next time you want to try out a new protocol or extend SCTP somehow, you run into the exact same problem.

The difficult part isn't stuffing bytes into UDP packets, the difficult part is making sure it actually works reliably and fast over the public internet.


Everything you wrote about SCTP over UDP applies equally to QUIC


No, it doesn't.

1. While getting telcos to fix their network so that QUIC works optimally might also need some work, it should be easier than for unencapsulated SCTP for the simple reason that (a) there is a relevant proportion of the network that already does work fine with it, so it is easier to blame the bad providers when things don't work so well with some provider that throttles UDP, say, and (b) in the case of simple UDP throttling, QUIC is in competition with VoIP and DNS, which could lead to degradation of those other services, which should increase the pressure to fix things. And in any case, you at least don't need to touch all CPE, only middle boxes within the infrastructure.

2. Probably more improtantly, SCTP is a plain text protocol, so getting it to work over telcos' networks does not help you at all the next time around when you want to evolve protocols. Once the work is done for QUIC, you won't have to do it ever again, because anything new that you could come up with is easily made indistinguishable from QUIC as far as the telco is concerned.


> SCTP is a plain text protocol,

There's an old phrase: better to keep your mouth shut and be assumed a fool, than to open it and remove all doubt.

SCTP is not a plain text protocol. I'm not sure how you managed to convince yourself of this one.

Here is the bit-by-bit format of the SCTP header, in case you were ever confused again:

https://tools.ietf.org/html/rfc4960#section-3.1

In the above post, I was talking about SCTP over UDP, which is an IETF standard already. You have moved the goal posts by talking about unencapsulated SCTP. We all agree that this is a tough problem. However, QUIC is also encapsulated over UDP, and when encapsulated, has the same advantages / disadvantages as SCTP over UDP.

SCTP over UDP over DTLS already constitutes a good portion of web traffic, since WebRTC (and hence technologies like hangouts, facebook video messaging, etc) is based on it.


> SCTP is not a plain text protocol. I'm not sure how you managed to convince yourself of this one.

> Here is the bit-by-bit format of the SCTP header, in case you were ever confused again:

> https://tools.ietf.org/html/rfc4960#section-3.1

I can not find there a single hint that anything is encrypted, and I would have been very surprised if I had. The security considerations do even explicitly acknowledge that it's up to the application to encrypt the payload if it wants to.

> In the above post, I was talking about SCTP over UDP, which is an IETF standard already. You have moved the goal posts by talking about unencapsulated SCTP.

No, I simply covered that case as well because SCTP on IP was mentioned as a supposedly viable alternative in this discussion as well, which supposedly would be preferable over QUIC because of lower overhead (which is true if it works, of course).

> However, QUIC is also encapsulated over UDP, and when encapsulated, has the same advantages / disadvantages as SCTP over UDP.

No, it then has the advantage of being nearly completely encrypted, so we don't get any protocol ossification.

> SCTP over UDP over DTLS already constitutes a good portion of web traffic, since WebRTC (and hence technologies like hangouts, facebook video messaging, etc) is based on it.

Well, yeah, but then what's the advantage over QUIC? It's also a complete flow control and encryption stack in user space if you want to run it over DTLS! I mean, I don't mind SCTP, but it's not like the kernel implementation is of any help when you want to run it over DTLS ...


> I can not find there a single hint that anything is encrypted, and I would have been very surprised if I had. The security considerations do even explicitly acknowledge that it's up to the application to encrypt the payload if it wants to.

Plain text is typically used as the opposite of 'binary'. You are intending to say 'unencrypted' or 'clear text', not 'plain text'. SCTP is not encrypted by default. It is a transport protocol and it is intended that

> Well, yeah, but then what's the advantage over QUIC? It's also a complete flow control and encryption stack in user space if you want to run it over DTLS! I mean, I don't mind SCTP, but it's not like the kernel implementation is of any help when you want to run it over DTLS ...

the advantage is that DTLS over SCTP is an internet standard that was created organically by a variety of motivated individuals and organizations. DTLS over SCTP does not require kernel encryption, and is implementable in userspace. This is similar to how TLS is implemented in userspace but uses the kernel TCP drivers. SCTP has been a standard for almost two decades and DTLS over SCTP for a similarly long time. There is no reason to replace something that works, with something that is exactly the same, except for the fact it's invented by Google.

You claim QUIC is encrypted. It is not. QUIC is a transport protocol and transport protocols need their headers to be inspectable to provide good routing, etc. The payloads are encrypted, like any other transport protocol.

The point is that there is no technical advantage to QUIC over DTLS over SCTP over UDP. Thus, we should reject the new standard in favor of the existing one.

This is similar to how, if someone made a POSIX-like specification that fulfilled all the same requirements as POSIX, we shouldn't simply adopt because it's newer. If the new technology offers no advantage over the old, then the new technology should be rejected as offering nothing new.


> Plain text is typically used as the opposite of 'binary'. You are intending to say 'unencrypted' or 'clear text', not 'plain text'. SCTP is not encrypted by default. It is a transport protocol and it is intended that

https://en.wikipedia.org/wiki/Plaintext

Also, context. None of what I wrote makes any sense under the assumption that I was talking about a 'character string protocol'.

> the advantage is that DTLS over SCTP is an internet standard that was created organically by a variety of motivated individuals and organizations. DTLS over SCTP does not require kernel encryption, and is implementable in userspace. This is similar to how TLS is implemented in userspace but uses the kernel TCP drivers.

Which is true, but obviously does not apply to SCTP over DTLS, which is what you were talking about in your previous post.

> There is no reason to replace something that works, with something that is exactly the same, except for the fact it's invented by Google.

Well, true. Which is why it is relevant that QUIC is not exactly the same, as I have explained a dozen times now.

> You claim QUIC is encrypted. It is not. QUIC is a transport protocol and transport protocols need their headers to be inspectable to provide good routing, etc. The payloads are encrypted, like any other transport protocol.

Routing happens at the IP layer, so, no, a transport protocol does not need to be inspectable. And in fact it is an explicit design goal of QUIC to minimize what is inspectable. Which is why it is essentially uninspectable.

All that is inspectable is a connection ID and a packet sequence number. The sequence number doesn't really tell you anything as it is not even a sequence number of payload segments, but only of transport packets (so retransmits get new sequence numbers). The connection ID tells you which packets form a connection ... but then there is essentially one connection between two endpoints, so nothing to see there either that isn't obvious from the network layer anyway.

But none of the flow control machinery is inspectable by the network, and even the unencrypted header fields are still authenticated so as to prevent any meddling by middle boxes. Also, you know, you can read all of this in the specification, maybe that'd be better than removing more doubt?

https://tools.ietf.org/html/draft-tsvwg-quic-protocol-02#sec...


Many middle boxes do flow control and such based on deeper packet fields.

I did mention sctp over dtls, but not as a replacement for quic. Only to say that it is a transport protocol that already is being used


> Many middle boxes do flow control and such based on deeper packet fields.

Which is exactly why we need QUIC (or something like it), because they shouldn't, it's causing more problems than it helps.


SCTP can be encapsulated within UDP, has kernel support, and is already implemented in most major web browsers, because of WebRTC, for which it is the standards mandated protoclo.


It needs industry's adoption. It's probably the better long term solution.

But why bother when you can hack a wrapper around UDP and pretend you're long term planning for the "greater good" ?


Probably the "it's not userland" syndrome. Apparently, moving things to the userland is the new black.


QUIC is technically excellent. I have a game-theoretic concern with putting it in userspace.

I worry that everyone will have an incentive to "cheat" at congestion control, leading to a tragedy-of-the-commons situation of ever-more-aggressive flows, followed by eventual congestion collapse.

This effect didn't happen with in the TCP world, since most applications don't run with enough privileges to speak TCP without the kernel's involvement. But now that we're moving to a model in which applications do their own congestion control, an arms race seems inevitable.


Apps have been free to abuse UDP all this time, but it's not been a real issue. Even data-hungry applications like video-streaming have gone with TCP almost every time.

Also, couldn't the OS still throttle back greedy applications? No reason it couldn't detect the heavy stream of UDP traffic.


> Apps have been free to abuse UDP all this time, but it's not been a real issue. Even data-hungry applications like video-streaming have gone with TCP almost every time.

In the past Bittorrent and the creation of the µTP prooved ( http://blog.bittorrent.com/2009/11/13/testing-%C2%B5tp-is-%C... ) that unmanaged protocol abuse in userspace can lead to big degradation in QOS yes. Mainly due to "unfair" congestion policy between protocols.


> Apps have been free to abuse UDP all this time, but it's not been a real issue. Even data-hungry applications like video-streaming have gone with TCP almost every time.

I suspect that's more down to NAT than anything else


I worry that it'll become easier (and this more common) in an all QUIC world. It's always been possible, but few people have had the expertise and motive.

The OS can throttle sends on a single machine, but it can't arbitrate across machines or do anything about incoming flows.


> The OS can throttle sends on a single machine, but it can't arbitrate across machines or do anything about incoming flows.

Isn't that the case with TCP's congestion control as well? If the server's kernel isn't backing off properly, how can the client's TCP stack mitigate that?


People running services can and do play games with TCP send-side congestion control: see https://news.ycombinator.com/item?id=1942442

So far, these hacks have been limited by the need to work with stock receive-side TCP stacks. Once you move to UDP, all bets are off.


I'm having a bit of trouble understanding your point. Perhaps you could give a specific example of what you mean by "ever-more-aggressive flows?"


Say I ship an app and control its backend servers. I control QUIC on both ends. What's stopping my using a non-standard QUIC that starts faster and that backs off less aggressively in response to congestion signals like delay and loss? If I do that and you don't, then on a congested link, my app will end up using more than its fair share of bandwidth.


What is stopping you from doing this with TCP?


It is addressed above:

> This effect didn't happen with in the TCP world, since most applications don't run with enough privileges to speak TCP without the kernel's involvement.

The argument is that while kernel developers wouldn't start this supposed "arms race", but application developers would.

I think we would have seen this already with things like (m)uTP if it was an issue.


The GP was talking about controlling both ends of the connection, I assumed that implied having root-level access to both ends. A scenario where you control both ends but don't have full TCP/IP stack access on at least one side seems pretty academical.

(But even then, just open lots of TCP connections if you want to have your unfair share of bandwidth)


In HTTPS, nobody. Else, proxies ?


Clients' kernels?


So, what does this mean for HTTP and HTTP2? Will they always be around, and servers will just "gracefully degrade" and opt for the fastest/newest protocol?

Or will HTTP/1 and 2 slowly decline over time?

Genuine question, not snark.


We cannot know the distant future. In the medium term lots of corporates block QUIC already, and so far as I know nobody at all has even deprecated HTTP/1 let alone ceased to support out.


An obvious, and probably stupid question to ask: Does UDP NAT traversal work well enough nowadays with minimal user intervention?

Remember that UDP itself is stateless. So you either need some explicit port mapping via UPnP (which seems like a nightmare because HTTP connections are so common), or some really smart router that understands QUIC.


Is there a recommended guide for those without background to get up to speed on QUIC and it’s pros and cons?


HTTP/3 is HTTP/2 with minor updates and running on QUIC instead of TCP.

QUIC is basically rebuilding the stateful connection abilities of TCP on top of UDP with optimizations.

Multiplexed streams with packets that can arrive out of order, removing the head of line blocking issue with TCP. There is also forward error-correction to reconstruct lost packets instead of retransmitting. Better packet sizing and congestion algorithms to reduce round-trips. There are also updates to include all the TLS 1.3 stuff for 0-RTT setup.

All of this reduces latency and makes connections more reliable, especially with changing networks on mobile devices. Not many downsides other than that TCP is far more open and optimized than UDP so it may take a while to effectively get this protocol out there.


> especially with changing networks like mobile

In particular QUIC contains everything necessary to continue connections after the client's IP address (any one side really, but the other side has to find it) changes, which is helpful when devices change between networks without needing a Mobile IP-style bouncer.


Doesn't Multipath TCP already exist as a solution for this?


That's very inefficient as it is essentially creating backup TCP connections to get around the fact that TCP might not behave well in certain situations.

On mobile, it is meant to use an entirely separate network backbone (wifi vs LTE) so doesn't offer much benefits when you're limited to a single type.


Indeed. I’m not sure why adoption is so poor on that.


It's only available on apple iOS and maybe Mac OS out of the box. Android is probably not going to support it; if Android does support it, it will be a long time before there's critical mass and early versions may claim to support it, but not work well, and it may be hard to selectively enable it, and we know many devices won't get the updates to fix it.

It's expensive to support on servers because (to my knowledge) there's not a good way to ensure related sub flows arrive at the same nic rx queue on the same server in real world load balancing situations, which is critical for performance.

Nobody else is doing it (other than Apple for Siri), so it's not clear what the benefits are.


It's weird that Android wouldn't support it, it was developed for linux first.


In practice I’ve seen poor handling in Chrome on desktop when the router changes WAN IP. Chrome reports ERR_NETWORK_CHANGED rather than do something sensible — this problem doesn’t exist in HTTP/1.x. I assume it should just be reconnecting — or the server should be using the connection ID field properly.


Except no mobile devices use this yet :(


My Pixel 2 running the Hangouts app transparently switches between home WiFi, LTE and xfinity WiFi (Comcast's attempt to make every one of their customers into a WiFi hotspot).

The Hangout connection doesn't drop. It may go fuzzy/staticy before the WiFi drops in favor of LTE but that's all.


Plenty of mobile devices use QUIC. The OS only needs to support UDP (which I believe all do) and applications that use QUIC will work.


Very interesting. It sounds like the people behind it have learned a lot about networking in the current era. Is there any work toward making this a peer protocol to TCP and UDP?

Obviously that doesn't solve any immediate problems, in that I'm sure it would take a decade to get something like that in widespread use. But if the theory is that TCP isn't so great given what we know now, I'm wondering if it's worth making those capabilities available to other protocols.


How will negotiation work? If my work say blocks UDP 80 outbound will I have to wait for a timeout every single time I try to go to a site?


Chrome's implementation solves this by racing a TCP connection with the QUIC connection.


Ah now that's clever.


So the server now sends a TCP ACK to SYN that is never followed up on? That is, the actual server software will wake up from a listen(), go through the motions of setting up a session for nothing for 50% of the time with this hack? Nice waste of other people's resources right there!


Well, presumably, the client with close the TCP connection once QUIC is established, and cache the result of the race for some period of time.


Okay, I had to do some extra digging. And I was a bit off in my original comment: accept returns a socket only when a connection is fully established, that is, after the ACK from the client. Until then, the application is oblivious to the connection attempt. So sending SYNs by themselves is it too hard on the servers.


Apple supports this with iOS, albeit with multiple TCP connections over both wifi and cellular for user experience concerns.

https://support.apple.com/en-us/HT201373


The bigger boss will see his favourite website does not work over HTTP/1 and needs to open UDP 80 to access.


QUIC is a re-implementation of SCTP, an internet standard protocol. Google thinks it's better because they wrote it. Everyone else rolls their eyes.


That's just BS. (IETF) QUIC was developed from scratch by an IETF working group. Google submitted its (Google) QUIC version as input, however the IETF version is not directly based on it.


From wikipedia [1]:

> HTTP/3 (originally named QUIC) is an experimental transport layer[1] network protocol designed by Jim Roskind at Google,[2] initially implemented in 2012,[3] and announced publicly in 2013 as experimentation broadened

and then

> In June 2015, an Internet Draft of a specification for QUIC was submitted to the IETF for standardization

Unless you count 'developed independently by Google and then three years later submitted to IETF for standardization' as 'developed from scratch by an IETF working group', I fail to see how your assertion is backed by actual events.

For the record, SCTP has been standardized since 2000, and it was actually developed by an IETF working group.

[1] https://en.wikipedia.org/wiki/QUIC


I didn't look into it much at the time, but there was an article here referencing QUIC the other week that had a lot of comments. I imagine there's probably some good resources in the comments, which should be easily found from a search.

Sorry, I would find it myself but I'm on mobile right now...



An interesting and somewhat related article called “How Unreliable is UDP?”: https://www.openmymind.net/How-Unreliable-Is-UDP/.

Discussion from four years ago: https://news.ycombinator.com/item?id=8465956.


What happened to QUIC as a non-HTTP-specific layer that any protocol could use? Even the QUIC Wikipedia page has been changed to a HTTP/3 page.


The vendor history of network and transport protocols can be seen as invisible to end user market warfare. Those who are able to push their network based ideas to better address the way they want to do things (and shape the market) win.

Way back this was IPX/SPX vs IP/(TCP|UDP). We know who won that and what happened to the loser.

There is another aspect to this: agnostic network and 'good enough' conceptions have been historically adopted by engineering organizations to remove vendor disposition from the end user landscape and protect the end user. On this forum a lot of these historical protections are seen as passe or unimportant for reasons best not scrutinized too closely.

IMO, the idea embodied in encapsulation of application protocol traffic in stateless transport to avoid a vendors technical issues with existing application stream orientation|transport in the non-vendor real world is one of the ugliest ideas I've yet encountered.

There are a lot of good ideas floating around in the comments here that avoid vendor lock in for applications stream based transport that already exist.


Talking with a friend, we believe QUIC itself could be reconceptualized into multiple layers without changing the wire format:

1. Connection management 1.1. Initial handshake allows arbitrary layer above initial handshake in same packet as body 1.2. Subsequent packets have connection identifier.

2. Encryption. The SSL bits.

3. Multiplexing and Congestion control.

So in essence the trick of QUIC is not collapsing layers, but splitting TCP into connection management and congestion control and moving SSL in the middle.

So why wasn't this proposed? I hope not because people are too "practical" to care about layering and abstraction anymore!


I've always felt the lack of congestion control made UDP like the Wild West. I think the endpoints (server and client) have a lot of opportunity to really screw things up for everything in between.


Say I decide to implement HTTP/3 in a language that doesn't have a library for it yet.

Is there any testing toolkit that will tell me "bzzt! you messed up header X" or any other error I might make?


TBH i dont know much in this area but nobody has mentioned CoAP which is basically simple http over udp for the IOT world. Have code a couple of use cases in c# for both server and client. As it is both blinding fast, and has option for reliabe messaging I wonder if CoAP was influenced by QUIC or other way round. Just curious. Seems modern. apps that are micro service driven eminently suited to using CoAP for its lower overhead and speed... And as such HTTP/3 makes a whole load of sense. #justsaying


Garh.

So we are to believe that the people who thought that multiplexing a single TCP connection over mobile was a good idea, that moving lockstock and barrel to UDP will solve our problems?

Look, its very simple: HTTP is now a file server protocol with a chatty control channel smashed in. However instead of optimising for that we have this.

It basically seems like a massive ego trip: "we can't admit we were wrong about the multiplexing thing, lets just bash out a reliable UDP protocol"

Look, the vast majority of file transfer protocols are TCP for a reason. Loosing chunks of files is annoying, and making a custom protocol over UDP that is both fast start reliable and fast, is actually quite difficult.

Yes, UDP has the advantage that you don't have a connection, and you can just fire stuff at the destination port and it'll magically arrive. however, anyone can do that. How does HTTP3 handle noise? how does it handle spoofing?

Basically, H2 was a regression because it was designed by people who didn't appear to understand real world networks, this seems like a doubling down.


It seems like you're just saying things without having actually researched how QUIC works. QUIC is the next iteration of TCP. That it lives on top of UDP (& might ship that way) is an implementation detail due to the realities of how the Internet is built, but it's a reliable in-order transport (+ guaranteed encryption, loss of packets on a stream doesn't block other unrelated streams, etc).


> without having actually researched how QUIC works

I have evaluated QUIC, as reliable stream protocols on UDP are something I have a professional interest in.

> QUIC is the next iteration of TCP

Its not designed for that.

> That it lives on top of UDP (& might ship that way) is an implementation detail due to the realities of how the Internet is built

no, its fundamental. You can't just swap out TCP with UDP and have done with it. If we ignore the Datagram vs stream aspect, we are left with implementing our own flow control, packet loss and authentication.

Which is where my comment comes in, with a TCP socket, if someone tries to inject spoofed packets, (assuming we in a non-LAN environment) those packets will be rejected long before they reach us. In UDP no such guarantees exist.

Thats not considering flow control, which is a whole 'nother issue. Its always a trade-off between efficiency, speed and reliability.


QUIC is an encrypted protocol. If an adversary injects forged packets the AEAD tag won't match in decryption and those packets will be discarded.

I would suggest an evaluation in which you completely omitted to notice the central feature of the protocol doesn't reflect well on your abilities, regardless of whether you claim it's a "professional interest".


encrypting the whole protocol intrinsically protects against denial of service, buffer overflows and other common mistakes how exactly?

My point is this, the client is protected from a _lot_ of noise. QUIC changes this.


Er, no. What happened is you have no idea what you're talking about.


so your saying that in TCP, a connection orientated protocol, at each hop the router/firewall/kernel is _not_ doing basic validation to see if that connection is valid, and not coming from a spurious source?

The client does all this when you open socket.connect()?

no, no, you're not and no, it doesn't, which is part of my point. Lots more noise will be forwarded to the client directly, after all, stateful connection management is impossible. With more noise, comes a bigger attack surface.


> Its not designed for that.

Yes, yes it is. I did 20% on Quic, I've talked with the engineers on it, I've read a lot of the paper, I've written some code. It's 100% intended to replace TCP as much as SCTP & other next-gen stream-oriented protocols are. It's written on top of UDP because they're dealing with the realities of public routing on the internet.

> no, its fundamental. You can't just swap out TCP with UDP and have done with it. If we ignore the Datagram vs stream aspect, we are left with implementing our own flow control, packet loss and authentication.

This again leads me to question how much you've actually researched Quic. It does all that & more. Now it is easier to write a Quic client in user-space that misbehaves for flow control & distribute that (TCP would typically require root access). However, that's the reality of the Internet - middleboxes have ossified the ability to add new protocols. It's why SCTP & the like are non-starters even though IETF has tried to get them to work for so long.

> Which is where my comment comes in, with a TCP socket, if someone tries to inject spoofed packets, (assuming we in a non-LAN environment) those packets will be rejected long before they reach us. In UDP no such guarantees exist.

Are spoofed packets validated at every hop? It would seem like memory & CPU load at backbones would be insane to keep track of all the connections flowing across it. I'm also curious how that would be possible because a backend route switch (i.e. new BGP route) would cause all middleboxes to drop the connection. I'm not sure even ISPs try to reject TCP connections early.

As for the "protection" you lose (if there is any), you gain by your ability to hop physical networks without losing the connection (i.e. WiFi & cellular). This means there's all sorts of cool things you can do that simply weren't possible before; use both wifi & cellular at the same time to download the same file, seamless handoff of a video chat without requiring any ISP support, seamless detection & handoff of a failing internet route to a fallback connection (i.e. walking in the parking lot without having to manually switch off WiFi).

The connection itself does have a connection ID in the public header in the version negotiation packet which is then mirrored in the encrypted packet header to protect against spoofing. However, the connection ID may be omitted from the public header in future packets.

Anyway, I recommend reading the IETF docs on QUIC [1][2], if you have an open mind. If you just have the mentality of "UDP bad", then probably you will be wasting your own time reading it. There's lots of really smart networking experts who work on & design QUIC & lots of smart people in IETF looking over & providing feedback/criticism. I think it's safe to assume they've covered any obvious problems concerns.

[1] https://tools.ietf.org/html/draft-ietf-quic-transport-08#pag... [2] https://tools.ietf.org/id/draft-ietf-quic-manageability-01.h...


Wouldn't UDP be easier to spoof?


QUIC is encrypted, so no not really.


Amazing: I couldn't find a "is quick working?" kind of test website. Tried several wordings to no avail.


URL should be https://daniel.haxx.se/blog/2018/11/11/http-3/ , which is the permanent link to this post (not to the "http3" tag on the blog).


OK, changed from https://daniel.haxx.se/blog/tag/http3/. Thanks!


What will this do to Tor, which can't route UDP?


A better way to think about it is that Tor routes streams. Tor has the concept of "pluggable transports" which don't care what form the data takes over the network, as long as a stream can be reconstructed.

For example, Snowflake[1] which is WebRTC based, and meek[2] which chunks the stream into a series of HTTP requests/responses for domain fronting. TCP is still used between nodes.

[1] https://trac.torproject.org/projects/tor/wiki/doc/Snowflake

[2] https://trac.torproject.org/projects/tor/wiki/doc/meek


AFAIK the proposal for HTTP-over-QUIC was to race with a TCP connection. Similar to Happy Eyeballs for IPv4/IPv6 dual stacks (meaning we could end up with 4 connections attempts?).


Exit nodes would be the only part that needs to support it.


I have a problem with QUIC. As everyone implementing this knows, TCP and UDP headers have very similar structure and the difference mostly lies within different treatment of packets in these two. QUIC will increase the size of transport and this affects capped customers. Why is QUIC chosen despite this major flaw? :Not everyone uses gigabit fiber:


It looks like the opposite to me?

TCP minimum header is 20 bytes.[1] QUIC "short header" (after key negotiation) looks like it can go down to 2 bytes for non-roaming connections with a 1-byte packet number, if the server allows the omit_connection_id, and 10 bytes otherwise.[2] Add the UDP header (8 bytes[3]) and you're at 10-21 bytes. So if you both supply a connection ID and a 4-byte packet number you're just barely at the same transport overhead as TCP; otherwise it's less.

(The QUIC "long header" is longer but that's a fixed cost and shouldn't matter too much for overhead.)

Or am I missing something?

[1]: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#... [2]: https://tools.ietf.org/html/draft-ietf-quic-transport-08#sec... [3]: https://en.wikipedia.org/wiki/User_Datagram_Protocol#Packet_...


> QUIC will increase the size of transport and this affects capped customers.

Will it? The TCP header is 20 bytes, the UDP header is 8 bytes. And in as far as packets are bigger, will it actually cause more traffic? For example, you have muliplexing of streams over one connection without head of line blocking, while with HTTP over TCP you need multiple TCP connections and thus multiple TLS connections for that, for example. Also, QUIC allows you to change UDP endpoints without the need to establish a new connection. I wouldn't be so sure it needs more bandwidth.

Now, arguably, building directly on IP would be better, but that is unfortunately an option that ISPs and other manufacturers and operators of middleboxes have destroyed, so I think you can't really blame QUIC for that. There really is no realistic way to evolve beyond the limitations of TCP unless you build on UDP. If QUIC succeeds and if we manage to move to IPv6 and get rid of NAT, though, I guess there might be a chance to one day see "QUIC on IP"--as far as the protocol is concerned, it should be trivial to adopt it for direct IP transport.


Because it's not obviously a flaw, and since it's not obvious you'll have to supply arguments if you want people to believe it's a major flaw.

QUIC is supposed to send fewer packets for the same payload, at least that's what I've heard. I don't follow it closely. Assuming that's true, if you want people to believe that QUIC packets are bigger and that the increased size more than compensates for their reduced number, you'll have to supply arguments. Packet traces for a few typical connections would be good arguments. Assuming it's false, you'll need much the same kind of argument.

(FYI I've written ten RFCs and like to think I've learned the RFC processes by now.)


I think the correct comparison is QUIC vs TCP+TLS.


And vs SCTP.

A deep dive impartial comparison at different usecase/workload would be great.


I don't have hard measurements, but maybe fewer round trips still results in a net decrease in transfer.

Also clients don't have to use QUIC.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: