QUIC is a really nice protocol as well, I find. It basically gives you an end-to-end encrypted & authenticated channel over which you can transport multiple streams in parallel, as well as datagrams. A lost packet in a given stream won't block other streams, and the overhead is quite low. Both ends of the connection can open streams as well, so it's really easy to build bidirectional communication over QUIC. You can also do things like building a VPN tunnel using the datagram mechanism, there are protocols like MASQUE that aim to standardize this. Apple is using a custom MASQUE implementation for their private relay, for example.
HTTP/3 is a protocol on top of QUIC that adds a few more really interesting things, like qpack header compression. If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values. I found that quite confusing when testing connections as I thought "It's impossible that I only get 2 bytes, I sent a long header string..." until I found out about this.
I dabbled with QUIC a few years ago and I couldn't agree more. It was pleasant to work with, and because it's UDP based, suddenly you can do NAT hole punching more easily.
Funny that you mentioned a VPN, because I made a little experimental project back then to hole-punch between two behind-the-NAT machines and deliver traffic between them over QUIC. I was able to make my own L2 and L3 bridges across the WAN, or just port forward from one natted endpoint to an endpoint behind a different NAT.
At one point I used it to L2-bridge my company's network (10.x) to my home network (192.168.x), and I was able to ping my home server from the bridging host, even though it was different networks, because it was essentially just connecting a cable between the networks. It was quite fun.
I only tested one hop, e.g. A(10.x)---->B(192.x). All I had to do there was to adapt the routing tables: On A, route 192.x traffic to the "natter" tap/tun interface (I always forget which is L2), and on B, route traffic to 10.x accordingly. That's all.
For it to be routable in the entire network, you'd need to obviously mess with a lot more :-D
Well it's abandoned and experimental, and there are better ways to hole punch than what I did, e.g. using STUN and TURN. But yeah it could replace one ngrok use case, though I think ngrok does not do L2/3 bridges.
Also: I think technically this was my first Go program (https://github.com/binwiederhier/re), but that was so tiny that it doesn't really count. ;-)
I believe that qpack(like hpack) still has some sharp edges. As in...low entropy headers are still vulnerable to leakage via a CRIME[0] style attack on the HTTP/3 header compression.
In practice, high entropy headers aren't vulnerable, as an attacker has to match the entire header:value line in order to see a difference in the compression ratio[1].
> And there are bit flags in the encoder instructions that carry header values that signal they should never be inserted into a dynamic table.
My understanding is that these header values have to be known to the Qpack implementation beforehand. As a result, it isn't possible for an HTTP/3 client to signal to Qpack to treat a particular custom header as a never-indexed literal.
I’ve been trying to wrap my mind around whether (and how much) QUIC would be better than TCP for video streams that rely on order-sensitive delivery of frames, especially for frames that have to be split into multiple packets (and losing one packet or receiving it out of order would lose the entire frame and any dependent frames). We used to use UDP for mpegts packets but found TCP with a reset when buffers are backlogged after switching to h264 to be a much better option over lossy WAN uplinks, (scenario is even worse when you have b or p frames present).
The problem would be a lot easier if there were a feedback loop to the compressor where you can dynamically reduce quality/bandwidth as the connection quality deteriorates but currently using the stock raspivid (or v4l2) interface makes that a bit difficult unless you’re willing to explicitly stop and start the encoding all over again, which breaks the stream anyway.
QUIC handles the packet ordering and retransmission of lost packets before it is handed to the upper layer of the application. However, if you are just sending one channel of video packets through the pipe, it probably won’t buy you much more. Where QUIC additionally excels is being able to send and/or receive multiple streams of data across the “single” connection where each is effectively independent and does not suffer head of line blocking across all streams.
> stop and start the encoding all over again, which breaks the stream anyway.
It's a common thing I wish encoders could do - if I try to compress a frame and the result is too large to fit in my packet/window, I wish I could 'undo' and retry compression of the same frame with different options.
Sadly all hardware video compressors mutate internal state when something is compressed, so there is no way to undo.
Since it lives on top of UDP, I believe all you need is SOCK_DGRAM, right? The rest of QUIC can be in a userspace library ergonomically designed for your programming language e.g. https://github.com/quinn-rs/quinn - and can interoperate with others who have made different choices.
> Since it lives on top of UDP, I believe all you need is SOCK_DGRAM, right? The rest of QUIC can be in a userspace library ergonomically designed for your programming language e.g. (...)
I think that OP's point is that there's no standard equivalent of the BSD sockets API for writing programs that communicate over QUIC, which refers to the userspace library you've referred to.
A random project hosted in GitHub is not the same as a standard API.
Did they not answer that question? It uses the BSD sockets API with SOCK_DGRAM?
Right, that random project is not a standard API - its built using a standard API. You wouldn't expect BSD sockets to have HTTP built in... so you can find third-party random projects for HTTP implmented with BSD sockets just like you can find QUIC implmented with BSD sockets.
QUIC is roughly TCP-equivalent not HTTP-equivalent, and we do have a BSD sockets API for TCP. You might be thinking of HTTP/3 rather than QUIC; HTTP/3 actually is HTTP-equivalent.
You can turn the OP's question around. Every modern OS kernel provides an efficient, shared TCP stack. It isn't normal to implement TCP separately in each application or as a userspace library, although this is done occasionally. Yet we currently expect QUIC to be implemented separately in each application, and the mechanisms which are in the OS kernel for TCP are implemented in the applications for QUIC.
So why don't we implement TCP separately in each application, the way it's done with QUIC?
Although there were some advantages to this while the protocol was experimental and being stabilised, and for compatibility when running new applications on older OSes, arguably QUIC should be moved into the OS kernel to sit alongide TCP now that it's stable. The benefit of having Chrome, Firefox et al stabilise HTTP/3 and QUIC were good, but that potentially changes when the protocol is stable but there are thousands of applications, each with their own QUIC implementation doing congestion control differently, scheduling etc, and no cooperation with each other the way the OS kernel does with TCP streams from concurrent applications. Currently we are trending towards a mix of good and poor QUIC implementations on the network (in terms of things like congestion control and packet flow timing), rather than a few good ones as happens with TCP because modern kernels all have good quality implementations of TCP.
> QUIC is roughly TCP-equivalent not HTTP-equivalent, and we do have a BSD sockets API for TCP. You might be thinking of HTTP/3 rather than QUIC; HTTP/3 actually is HTTP-equivalent.
No, I understand QUIC is a transport and HTTP/3 is the next HTTP protocol that runs over QUIC. I was saying QUIC can be userspace just like HTTP is userspace over kernel TCP API. We haven't moved HTTP handling into the kernel so what makes QUIC special?
I think it is just too early to expect every operating system to have a standard API for this. We didn't have TCP api's built-in originally either.
> What makes TCP special that we put it in the kernel? A lot more of those answers apply to QUIC than to HTTP.
I mean, that seems like GP's point. There's no profound reason TCP should be implemented in the kernel, besides trying to fit every form of IO into Unix's "everything is a file descriptor" philosophy.
If QUIC works in userspace, that's fine. If it ain't broke, don't fix it.
Huh, you think so? I'd be a bit surprised if that was the intent, because TCP obviously is in the kernel, so if their point blatantly disagrees with preexisting decisions in every OS then that's something that needs to be addressed before we can talk about QUIC.
> I was saying QUIC can be userspace just like (...)
I think you're too hung up on "can" when that's way besides OP's point. The point is that providing access to fundamental features through a standard API is of critical importance.
If QUIC is already massively adopted them there is no reason whatsoever to not provide a standard API.
If QUIC was indeed developed to support changes then there is even fewer arguments to not provide a standard API.
Isn't the point of QUIC to offer high performance and flexibility at the same time? For these requirements, a one-size-fits-all API is rarely the way to go, so individual user-space implementations make sense. Compare this to file IO: For, many programs, the open/read/write/close FD API is sufficient, but if you require more throughput or control, its better to use a lower-level kernel interface and implement the missing functionality in user-space, tailored to your particular needs.
It occurs to me that QUIC could benefit from a single kernel-level coordinator that can be plugged for cooperation - for instance, a dynamic bandwidth-throttling implementation a la https://tripmode.ch/ for slower connections where the coordinator can look at pre-encryption QUIC headers, not just the underlying (encrypted) UDP packets. So perhaps I was hasty to say that you just need SOCK_DGRAM after all!
There are dozen of libs to do it right now but I expect ultimately distro folks will want to consolidate it all to avoid redundant dependencies, so we'll probably end up with openssl/gnutls/libressl doing it eventually and most apps using that.
Note there were talks to add it in-kernel like kTLS, but since the certificate handling is difficult that part will be offloaded to userspace as it is with kTLS -- you won't ever get an interface like connect() and forget about it.
"QUIC improves performance of connection-oriented web applications that are currently using TCP.[QUIC] is designed to obsolete TCP at the transport layer for many applications, thus earning the protocol the occasional nickname "TCP/2"."
Also taken from the wiki page:
"QUIC aims to be nearly equivalent to a TCP connection but with much-reduced latency."
Here is the only relevant bit regarding TLS:
"As most HTTP connections will demand TLS, QUIC makes the exchange of setup keys and supported protocols part of the initial handshake process. When a client opens a connection, the response packet includes the data needed for future packets to use encryption. This eliminates the need to set up the TCP connection and then negotiate the security protocol via additional packets."
[1] https://en.m.wikipedia.org/wiki/QUIC
The RFCs for QUIC (RFC 9000 and RFC 9001) mandate encryption.
Some random stackoverflow answer[1] claims there are implementations that ignore this and allow "QUIC without encryption", but I'd argue that it's not QUIC anymore -- in my opinion it'd be harmful to implement in the kernel.
Per the RFC, QUIC is literally defined as a transport protocol... Literally the first sentence of the overview is:
"QUIC is a secure general-purpose transport protocol."
More importantly, QUIC literally bakes in TLS into how it works which is a far cry away from replacing it.
"The QUIC handshake combines negotiation of cryptographic and transport parameters. QUIC integrates the TLS handshake [TLS13], although using a customized framing for protecting packets."
> So why don't we implement TCP separately in each application, the way it's done with QUIC?
Because library/dependency management sucked at the time, and OSes competed for developers by trying to offer more libraries; on the other side we were less worried about EEE because cross-platform codebases were rare and usually involved extensive per-OS code with #ifdefs or the like.
I don't think we would or should put TCP in the kernel if it was being made today.
> So why don't we implement TCP separately in each application, the way it's done with QUIC?
Because TCP also does multiplexing, which must be handled in some central coordinating service. QUIC doesn't suffer from that since it offloads that to UDP.
You wouldn't use a system TLS implementation either (well, technically SChannel/NetworkTransport exist, but you're vastly better off ignoring them).
That's going to happen regardless, that's just the nature of Ethernet, or even just existing in reality. TCP has to deal with the same thing. Either way the solution is the same: acks and retransmissions.
> Did they not answer that question? It uses the BSD sockets API with SOCK_DGRAM?
No, that does not answer the question, nor is it a valid answer to the question. Being able to send UDP datagrams is obviously not the same as establishing a QUIC connection. You're missing the whole point of the importance of having a standard API to establish network connections.
> Right, that random project is not a standard API - its built using a standard API.
Again, that's irrelevant and misses the whole point. Having access to a standard API to establish QUIC connections is as fundamental as having access to a standard API to establish a TCP connection.
> so you can find third-party random projects for HTTP (...)
The whole point of specifying standard APIs is to not have to search for and rely on random third-party projects to handle a fundamental aspect of your infrastructure.
Not having a system API is the entire point of QUIC. The only reason QUIC needs to exist is because the sockets API and the system TCP stacks are too ossified to be improved. If you move that boundary then QUIC will inevitably suffer from the same ossification that TCP displays today.
No, the reason QUIC exists is that TCP is ossified at the level of middle boxes on the internet. If it had been possible to modify TCP with just some changes in the Linux, BSD and Windows kernels, it would have been done.
"""
A middlebox is a computer networking device that transforms, inspects, filters, and manipulates traffic for purposes other than packet forwarding. Examples of middleboxes include firewalls, network address translators (NATs), load balancers, and deep packet inspection (DPI) devices.
"""
I don't know about that. Without middlebox problems we might have used SCTP as a basis and upgraded it. But it's so different from TCP that I doubt we would have done it as a modification of TCP.
The alternative is that either browsers will be the only users of QUIC - or that each application is required to bring its own QUIC implementation embedded into the binary.
If ossification was bad if every router and firewall has its own TCP stack, have fun in a world where every app has its own QUIC stack.
User-space apps have a lot more avenues for timely updates than middleboxes or kernel-space implementations do though, and developers have lots of experience with it. If middleboxes actually received timely updates and bugfixes, there would be no ossification in the first place, and a lot of other experiments would have panned out much better, much sooner than QUIC has (e.g. TCP Fast Open might not have been DOA.)
There's also a lot of work on interop testing for QUIC implementations; I think new implementations are strongly encouraged to join the effort: https://interop.seemann.io/
I am not seeing the problem with every participant linking in their own QUIC implementations. The problem of ossification is there is way too much policy hidden on the kernel side of the sockets API, and vanishingly few applications are actually trying to make contact with Mars, which is the use-case for which those policies are tuned.
There are a billion timers inside the kernel, not all of which can be changed. Some of them are #defined even.
In these days when machines are very large and always have several applications running, having an external network stack in the kernel violates the end-to-end principle. All of the policy about congestion, retrying, pacing, shaping, and flow control belong inside the application.
Can you point me to an example of a timer in the kernel that is not settable/tunable that should be? My experience in looking at such things suggests that most of the #defined bits are because RFCs define the protocol that way.
As for network stack per application: you're more than welcome to so so a myriad of ways - linux provides many different ways to pull raw IP, or raw ethernet into userspace (e.g. xdp, tun/tap devices, dpdk, and so on). It's not like you're being forced to use the kernel stack from lack of supported alternatives.
I wouldn't. I would write down the protocol in a good and extensible way, the first time. It's no good throwing something into the world with the assumption that you can fix the protocol later.
> The alternative is that either browsers will be the only users of QUIC - or that each application is required to bring its own QUIC implementation embedded into the binary.
This is already being done with event loops (libuv) and HTTP frameworks. I don't see why this would be a huge issue. It's also a boon for security and keeping software up-to-date because it's a lot easier to patch userspace apps than it is to roll out a new kernel patch across multiple kernels and force everyone to upgrade.
> because the sockets API and the system TCP stacks are too ossified to be improved
What part of the sockets API specifically do you think is ossified? Also, that doesn't seem to have kept the kernel devs from introducing new IO APIs like io_uring.
I think the point of QUIC is 'if the implementation other using is problematic, I can use my own. And no random middlebox will prevent me from doing so' instead of `everyone must bring their own QUIC implementation.`
There is a slight different here. It's the difference between 'the right to do' and 'the requirement to do'.
While at same time. You must use 'system tcp implementation' and you are not allowed to use custom one. Because even system allow it (maybe require root permission or something), the middlebox won't.
> Not having a system API is the entire point of QUIC. The only reason QUIC needs to exist is because the sockets API and the system TCP stacks are too ossified to be improved.
I don't think your take is correct.
The entire point if QUIC was that you could not change TCP without introducing breaking changes, not that there were system APIs for TCP.
Your point is also refuted by the fact that QUIC is built over UDP.
As far as I can tell there is no real impediment to provide a system API for QUIC.
> If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values
Reminds me of tokenization for LLMs. 48 dashes ("------------------------------------------------") is only a single token for GPT-3.5 / GPT-4 (they use the cl100k_base encoding). I suppose since that is used in Markdown. Also "professional illustration" is only two tokens despite being a long string. Whereas if you convert that to a language like Thai it is 17 tokens which sucks in some cases but I suppose tradeoffs had to be made
That makes sense -- if a packet is lost, and it affected just one asset, but you're on TCP, then everything has to wait till the packet is re-requested and resent.
HTTP2 allowed multiple streams over one TCP stream, but that kinda made HoL blocking worse, because in the same scenario HTTP 1.1 would have just opened multiple TCP streams. QUIC, as GP said, basically gives you a VPN connection to the server. Open 100 streams for reliable downloads and uploads, send datagrams for a Quake match, all over one UDP 'connection' that can also reconnect even if you change IPs.
There is an API for this on Linux. It's used to checkpoint the state of a TCP connection, for example to move a live TCP connection to another machine incase you want to do zero downtime hardware replacement while keeping all TCP sockets open.
I can't wait to start implementing RTP atop QUIC so we can stop having to deal with the highly stateful SIP stack and open a media connection the same way we open any other Application layer connection.
QUIC is junk.. You people all care about raw throughput but not about multiuser friendless. Selfish. Problem is, QUIC is UDP and so its hard to police/shape.
I really want to play some FPS game while someone is watching/browsing web.
Also, I really want my corpo VPN have a bit of priority over web, but no, now I cannot police it easly. TCP is good for best effort traffic, and thats where I classify web browsing, downloading, VoD streaming. UDP is good for gaming, voice/video conferencing, VPNs (because they are encapsulate stuff you put another layer somewhere else).
I feel like that’s an ungenerous characterization. First, QUIC should contain some minimal connection info unencrypted that can be middleware to do some basic traffic shaping. It’s also intentionally very careful to avoid showing too much to avoid “smart” middleware that permanently ossifies the standard as has happened to TCP.
Finally, traffic shaping on a single machine is pretty easy and most routers will prefer TCP traffic to UDP.
Finally, the correct response to overwhelm is to drop packets. This is true for TCP and UDP to trigger congestion control. Middleware has gotten way too clever by half and we have bufferbloat. To drop packets you don’t need knowledge of streams - just that you have a non-skewed distribution you apply to dropping the packets so that proportionally all traffic overwhelming you from a source gets equally likely to be dropped. This ironically improves performance and latency because well behaving protocols like TCP and QUIC will throttle back their connections and UDP protocols without throttling will just deal with elevated error rates.
So what? You dropping packets, and they are still coming, eating BW and buckets.
Because traditionally UDP did not have any flow control, you just treat is as kinda CBR traffic and so you just want to leave it queues as fast as it can.
If there was a lot of TCP traffic around, you just drop packets there and vioala, congestion kick sin and you have more room for importand UDP traffic.
Now, if you start to drop UDP packets your UX drops.. packet loss in FPS games is
terrible, even worse than a bit of jitter. Thank you.
But your complaint is about QUIC, not generic UDP. QUIC implements TCP-like flow control on top of UDP, designed to play well with TCP congestion control.
QUIC does play well with others. It's just implemented in the userspace QUIC library instead of the network stack.
I really don’t follow your complaint. QUIC (and other similar UDP protocols like uTP used for BitTorrent) implement congestion control. If packets get dropped, the sender starts backing off which makes you a “fair” player on the public internet.
As for gaming, that remains an unsolved problem, but QUIC being UDP based isn’t any different than TCP. It’s not like middleware boxes are trying to detect specific UDP applications and data flows to prioritize protecting gaming traffic from drops, which I think is what you’re asking for.
Now, I wish ToS/QoS were more broadly usable for traffic prioritization.
It sounds like you're using UDP vs. TCP as a proxy for ToS/QoS. At a minimum, you're still going to have a bad time with TCP streams getting encapsulated in UDP WireGuard VPN connections.
Is your complaint fundamentally that it's harder to tell the difference between games/voip and browser activity if you can't just sort TCP versus UDP?
That's true, but it's not that big of a deal and definitely doesn't make QUIC "junk". Looking at the port will do 90% of the job, and from what I can tell it's easy to look at a few bytes of a new UDP stream to see if it's QUIC.
The quick test is that the first packet is generally going to start with hex C[any nibble]00000001 and be exactly 1200 bytes long, ending with a bunch of 0s.
A better test is to see if the full header makes sense, extract the initial encryption key, and check if the rest of the packet matches the signature.
This is correct, you can recognize Initial packets easily/reliably, and they contain the connection IDs in plaintext so that a stateful packet filter/shaper can recognize individual data flows.
Note that packet numbers (and everything else other than port/IP) are encrypted, so you can't do more than drop or delay the packets. But blocking/shaping/QoS is perfectly doable.
I want to be clear that I was talking about checking the encryption of the very first packet, which isn't secret yet.
Once keys are already established I don't see any particularly reliable test for a single packet, but as you say the connection ids are accessible so if they're in the right place in both directions then that looks good for discovering a QUIC flow.
Somewhat the point. The issue we've had is that multiple ISPs and gov have been "policing" TCP in unsavory ways. Security and QoS are just fundamentally at odds with each other.
QoS is still possible with QUIC: initial connection IDs are in plaintext, and while you can't modify packets (or even see the packet numbers) you can drop or (ugh) delay them.
On the level of entire networks serving multiple end consumers and businesses, I really hope that ISPs get bigger pipes instead of trying to shape traffic based on type. I'm fine with making traffic shaping on a local network a little harder, if it ends up biting those who oppose net neutrality (or want to use middleboxes to screw up traffic in various technical-debt-fuelled ways).
Upvoted because I think you bring up some interesting challenges, but you might consider a softer tone in the future. (Calling the OP "selfish" goes against site guidelines, and generally doesn't make people open to what you're saying.)
That selfish was NOT the the OP. It was for general audiency who prefer all the bandwidth for themselfs. We know how most people behave. They do NOT really care what others are doing. For years, I was proud of my QoS because my entire home could utilize my (not so fast Internet) and I always could do gaming, because everything was QoS correctly. Nothing fancy, just separating TCP vs UDP and futher, doing some tuning between TCP bulk vs interactive traffic. Same went to UDP, some separation for gaming/voip/interactive vs VPN (bulk). HTB is pretty decent for this.
I don’t think they’re super valid concerns though - QUIC isn’t just dumb UDP that transmits as fast as it can, it has congestion control, pacing etc. built in that’s pretty similar to certain TCP algorithms, just it’s happening at the application layer instead of the kernel/socket layer handling it. In the design of these algorithms, fairness is a pretty key design criteria.
If anything, potentially QUIC lets people try better congestion control algorithms without having to rebuild their kernels which could make the web better if anything…
HTTP/3 is a protocol on top of QUIC that adds a few more really interesting things, like qpack header compression. If you e.g. send a "Content-type: text/html" header it will compress to 2 bytes as the protocol has a Huffman table with the most commonly used header values. I found that quite confusing when testing connections as I thought "It's impossible that I only get 2 bytes, I sent a long header string..." until I found out about this.