Thanks to Marten Seemann for maintaining the quic-go library we use. (I still haven't heard whether Go will add HTTP/3 to the standard library.)
Caddy 2.6 should be the first stable release of a general-purpose server to support and enable standardized HTTP/3 by default. HTTP versions can be toggled on or off. (Meaning you can serve only HTTP/3 exclusively if you're hard-core.)
PS. Caddy 2.6 will be our biggest release since 2.0. My draft release notes are about 23 KB. We're looking at huge performance improvements and powerful new features like events, virtual file systems, HTTP 103 Early Hints, and a lot of other enhancements I'm excited to show off on behalf of our collaborators!
I have recently gave Caddy a shot, and immediately loved it!.
I am a bit worried about the performance deficit compared to apache or Nginx, but none of my servers really have reached the scale where it actually matters.
Its unfortunate that there is no dynamic brotli support, but oh well, a trade off I'm willing to make.
One of things that aren't clear to me are Caddy plugins. How big of an affect do they have on peformance? Are plugins a core part of Caddy's usage? Or are they just exist to cover some edge cases?
I'm thinking of using the caddy-filter and caddy-crowdsec-bouncer plugins.
Caddy's performance is competitive with nginx, but there are too many dimensions to discuss here. I guarantee that for most people, your web server will not be your bottleneck.
Dynamic brotli is expensive, as brotli compression is inefficient. There are no optimized Go implementations of this yet. But Caddy can serve precompressed brotli files without any separate plugins.
What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin. (All plugins are compiled in natively.)
> What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin.
Oh I didn't know that, thanks for the awesome project!
Having plugins installed has no performance incursion at all (because they don't run unless configured), but using them will, of course affect performance. But it depends on the plugin and what it does.
You have to do your own benchmarking to see what effect they have to performance.
Everyone has different needs and configs, so it's not possible to give authoritative benchmarks. Just try it out and see.
Regarding brotli, there is a plugin you can try if you'd like https://github.com/ueffel/caddy-brotli, but brotli's on-the-fly performance is not that good. It works well if you pre-compress you assets though, and you can serve those with the file_server directive's precompressed option.
I've just moved over to Caddy and had a really good experience in doing so (thank you to you and all the other Caddy contributors!). One thing that stuck out to me during setup is that it seemed surprising to see gzip encoding not enabled by default. Wouldn't it make sense for a server that enables https by out of the box with zero configuration and soon supports http3 to also gzip by default?
When I first announced Caddy in 2015 and the server got busy, it started downloading everything as .gz files. (facepalm) (good ol' days)
But that's just a fun story, not actually the reason we don't enable it by default...
The tricky thing about enabling features by default is that turning them off is awkward in config. And while we like to have "magic" in Caddy, we don't like too much magic. Plus, gzip performance in Go is less good (but more memory-safe) than it is in C and assembly. Klauspost's flate implementation is very fast and that's the one we use. But even with a super-fast implementation, gzip requires memory and CPU that busy servers may be in short supply of.
So it's opt-in for now. We can always change it later; going the other way is harder.
(Caddy also supports serving pre-compressed files, if enabled! And starting with Caddy 2.6, those can be teleported at the speed of electricity to HTTP clients using sendfile. HTTPS also sees faster file serving in 2.6 due to optimized copying.)
Thanks for the response! I'd imagined reasoning along those lines. I'd perhaps expect that people anticipating such high load that gzipping output is problematic would be investing enough time in server setup to review and tweak these options. And for everyone else, compression by default is a good starting point. Moving from Nginx (HTTP2 off by default) to Caddy (HTTP2 on by default) you'd already need to review your setup if running a highly trafficked site to ensure any reverse proxied servers can handle the additional request load due to multiplexing.
But it's obviously a trade-off and there's lots to weigh up...
It is an interesting decision point, and I get your reasoning. But, I have noticed there are a good number of websites that could be better for their users (especially mobile users), because someone just took the defaults.
Not high priority, but if Caddy would offer a report of things that aren't configured but perhaps should be...that could help. Things like Cache-control/Expires/Etag headers, Gzip/Deflate, Caching, and so on.
It was making the browser download files with the .gz extension, instead of serving the content in a way that the browser would display. TL;DR it was broken at the time. But it was a very long time ago.
In addition to the reasons that mholt mentioned in response to your question, enabling HTTP/GZIP compression could possibly be less secure for some web server configurations due to things like the BREACH attack. See https://en.wikipedia.org/wiki/HTTP_compression#Security_impl... and https://en.wikipedia.org/wiki/BREACH for more info. I might be wrong but I don't think that current web serving protocols mitigate an attack like this. It might be better to default to safe settings that don't use HTTP/GZIP compression even if it might slow things down for the time being.
Eh, it's not just current web serving protocols. Any protocol where:
- An application uses compression
- An attacker is able to supply chosen data to it
- The application compresses the attacker's data and static secret data together
- The attacker is able to monitor the size of the compressed data
- This can be repeated by the attacker a number of times
will be vulnerable to having its secret data stolen by techniques like BREACH. If you want your secret data to stay secret, don't compress it with attacker chosen plaintext where the resulting size could be monitored.
I'm not super familiar with Traefik, but if you're talking about using container labels for config, no, Caddy's config is given as a JSON document through a REST API. The CLI can let you use config files if you prefer. Or the Caddyfile is a more human-friendly format that Caddy supports by default. You can actually use any config format you want if there's a config adapter for it: https://caddyserver.com/docs/config-adapters
Planning on trying Caddy 2.6 one of these day to see how much improvement it can make over other web servers, maybe CentmindMod would be interested in it. We all are aware local vs real world benchmark are differ.
But also, please don't take benchmarks like this as gospel. Run your own benchmarks, on your own hardware, with your own config. Someone else's config may not be ideal for your usecase.
I've been using Caddy in a 3 node HA cluster sharing an anycast bgp address for about 18 months now and it's been fantastic. Certs "just work" across the cluster once consul is wired up. I recently added IPv6 which also "just works."
greenpau/caddy-security is fantastic and "just works" for OIDC sso.
mholt, thanks for recently adding the ability to bind to multiple specific IP addresses by default, this help me conserve precious public IPv4 addresses.
They're essentially just saying they run multiple instances of Caddy, and those instances are reached by clients via Anycast. This isn't anything specific to Caddy, that's networking-level stuff, in front of Caddy.
TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. Most anycast setups I have seen have been geographic which means unless you move you won't get routed to the wrong endpoint.
*http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.
You are right, but it's easier than you think. At my workplace we have anycast without a ton of "tie breakers" so each client may have several backend servers that it could reach. But the router itself does do the IP based hashing when there are two valid routes to the same prefix. So yes, there is something doing hashing, but it's not anything special, it's just the usual "equal-cost multipath" behavior on the router.
> TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance.
It's called BGP protocol, as OP mentioned. This is basically how cloudflare works, you have the same IP for dozens of datacenters around the world. If routing changes and suddenly you are routed to different POP then your TCP session is broken but in most cases this is rare. You could "fix" this by routing TCP packets to POP B when detecting on POP A that session origin was POP B but this would incur additional latency for the rest of the session etc. But this has nothing to do with Caddy, you could use any other software same way using anycast IP.
BGP just makes sure your packets get to the right location. If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.
> BGP just makes sure your packets get to the right location.
I think I wasn't clear enough. Yes, BGP will route the packet to the nearest POP for that IP if it was announced in different locations, that's how "anycast IP" works, thanks to how BGP works. OP asked:
> or something in front of Caddy
So I've described what is it and how it works in simple terms.
> If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.
If you are talking only about BGP then of course you are right but I was not talking about solving this using only BGP. You can tunnel the packet to your other POP which was origin, as I mentioned in my answer.
So if you have a session established after 3 way handshake in POP B
client <-> POP B
and suddenly you are routed via POP A then just tunel all packets for that session:
client <-> POP A <-> POP B
or you could use MPTCP to solve this if supported etc.
... and now we're talking about contraptions you'd have to put in front of Caddy to make this work, which was what the commenter above was interested in hearing about. I don't care either way, except that the implication that BGP4 solves this problem set me off.
Obviously, in reality, people mostly rely on geography to deal with this: you're just unlikely to get randomly rerouted from a Singapore rack to a Sydney rack, and so the issue doesn't come up. But if you care about it --- as the previous commenter does --- you want a real answer.
> But if you care about it --- as the previous commenter does --- you want a real answer.
Yeah, I wasn't clear enough in my first comment, my last comment was more detailed so I guess it should match "a real answer" now.
> Obviously, in reality, people mostly rely on geography to deal with this: you're just unlikely to get randomly rerouted from a Singapore rack to a Sydney rack, and so the issue doesn't come up.
Yeah, most of the time it doesn't come up, anecdotally it's also more specific to some regions, in Europe for example I've seen this to occur more often (between POPs in EU) than in other regions.
This is all in my basement but I tried to build it like I would a rack in a datacenter the stack is:
1: EdgeRouter4 connected to ISP. This operates as a typical router, nothing special other than some static routes to item 2.
2: Pair of VyOS routers running in VMs. I think of them as top of rack L3 switches in a datacenter. Their primary purpose is for ECMP [1].
They also use VRRP so the can be upgraded without downtime.
3: Three node caddy cluster I mentioned. Running plain old Debian. anycast-healthchecker configures the anycast IP on each VM's loopback interface. Bird advertises the address to the two VyOS routers via BGP.
4: Caddy binds to each address.
All of the hosts in this network are configured with a default route to the VRRP address on the VyOS VM's. That's pretty much it.
I've recently started using Calico in k8s on some separate VM's, and they work the exact same way, advertising Services of type LoadBalancer to the VyOS routers which does ECMP across the nodes. I learned quite a bit configuring anycast-healthchecker with Caddy, then comparing it to how Calico works.
Thanks for a great piece of software that I use every day in production and just works.
At first I was scared of how stupid simple it is. It feels like web servers are supposed to have giant config files with a hundred mysterious knobs to twiddle. Now I always default to Caddy, and have yet to find an instance where it didn't fit my needs. Congrats.
This is my experience as well. I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt. Figured it out with Caddy within minutes. Really happy I found this software.
That said, while the capability is there, things like the DNS-01 validation need additional work and Caddy comes with more sane defaults out of the box and arguably a way easier config format.
It would also be really cool if Nginx came with its own implementation of ACME certificate provisioning out of the box, following the trend of other servers, but certbot is also perfectly passable.
Either way, it seems like most of the "popular" web servers are viable for most use cases nowadays and I can say that Caddy definitely belongs in that list!
My only complaint with it is this half-way json vs standard config file situation in which it finds itself. The documents push you to the json system, but most of the available documents are in the config system.
There's also some handy features that are only available in the JSON format. e.g. `strip_path_suffix` and `http_redirect` for the `rewrite` handler (https://github.com/caddyserver/caddy/issues/2011#issuecommen...), which would be a more ergonomic way to set a rule to strip trailing slashes than handling via regex.
While it is true that some features are only possible via JSON config, the ones you mentioned are available in the Caddyfile. See the uri directive which lets you strip a suffix, and redir which performs Location redirects. Also, the issue you linked to is from an early beta of Caddy v2, and the discussion and config there no longer applies.
It can also do regex replacements on the path portion of the URI. And yes, these are internal rewrites since uri is a directive that wires up the rewrite handler.
Note that the file_server will automatically enforce canonical URIs, including redirecting to add or remove the trailing slash according to whether the resource is a file or a directory.
To redirect many unspecified paths, your regex is probably the best way to do it for now. Feel free to open an issue to propose an alternative!
Thanks! I'd read all that documentation and came to the same conclusion. I'm using disable_canonical_uris as I prefer avoiding trailing slashes even for directories (as I see it, page_name/index.html layout is an implementation detail, and I'd like Caddy to expose it as /page_name).
I'll follow up with issues tomorrow (I think an issue for updating the common caddyfile patterns bit on the website also makes sense - I suspect a blanket rule for removing trailing slashes is more likely to be a common pattern people seek a solution for than removing a trailing slash on explicitly enumerated paths).
The problem is, in the browser, relative URLs (like on CSS/JS assets and such) won't be relative to your index.html if you remove the trailing slash, and they'll instead be relative to the parent directory. That can mess things up.
Of course if you're using absolute paths in your HTML then this isn't a problem, but it does become a problem if you want to move to serving this same site from a subpath, because then your absolute paths won't be constrained to this subpath.
JSON is the "machine readable" config language for Caddy, but a great majority of users are using the Caddyfile. What are you confused about specifically?
Weirdly enough, among the reasons to switch from nginx to Caddy I never find the lack of observability nginx and its almost useless Prometheus exporter have.
I see several posts discussing under which circumstances or use cases one might outperform the other but they never seem to care about having decent metrics.
I might overrate the importance of this, who knows...
It's not really possible to compare the performance of webservers apples to apples. Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time.
> It's not really possible to compare the performance of webservers apples to apples.
I'd say that it's possible, just not always easy, due to how many different configurations there can be for any given workload - the same servers might be used to achieve the approximately same configuration in different ways.
However, if you took a common enough use case or workload, such as reverse proxy with SSL and gzip compression for $FOO resource types, serving files for $BAR application and proxying $BAZ API, then it should definitely be possible. You would just need some meaningful real-world test instead of a purely synthetic benchmark, otherwise you'll probably test something slightly different than what your server will be doing in practice.
The other option is just to settle on some bit of common enough functionality and try to increase your sample size as much as possible.
For example, some attempts have been made by OpenBenchmarking and at least give a vague idea of the performance of some servers:
I'm mentioning this because while not everybody has the time to reproduce a given setup in any number of technologies, even similar enough tests in a controlled environment can produce meaningful information, at least to let you infer what orders of magnitude you're working with. For something concerning programming languages themselves and their frameworks, one just needs to look at what TechEmpower is trying to do: https://www.techempower.com/benchmarks/#section=data-r21
> Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time.
This is well said, though.
In general, I'm inclined to agree: most of the time, the performance of most web servers can be described as "good enough".
The exception to this might be using your application servers (e.g. Tomcat) as a web server and running into situations where serving static assets (that might be baked into your application) would slow down because of API calls being slower and processing them digging into comparatively more conservative HTTP thread limits for the whole thing. Then again, personally I'd argue that you should have one of the popular web servers in front of your applications (and typically serving static assets) to act as an ingress in most cases, but I've seen some interesting things over the years.
Regarding that last point, that's often done with the X-Accel-Redirect header; backend can decide which file to serve, responds with the path to that file as that header, then the webserver can serve that file if that response header is there. Caddy can do this btw, with response handlers in the reverse_proxy directive. This pattern can be used to serve uploaded files stored in a non-public location, after the backend authenticated the request etc.
> This pattern can be used to serve uploaded files stored in a non-public location, after the backend authenticated the request etc.
That's a perfectly valid use case!
Packaging a front end application (e.g. React/Angular/Vue) inside of a .jar file and making Tomcat or something else serve it, as a part of a larger Java application, though? Perhaps less of an optimal solution than just using Apache/Nginx/Caddy for it, outside of really wanting to be able to deploy everything as a single package.
> I understand why it’s hard to convince the rabid nginx greybeards that it isn’t “the shit” anymore.
I'm not sure about that. Nginx still feels like one of the more popular options and is generally a good fit for many of the problems that it tries to solve, at least in the deployments that I've seen it used in (reverse proxy, serving static assets, connected to PHP-FPM sometimes).
It also is the de factor Kubernetes ingress controller and other options like Traefik still haven't quite taken off in that space, at least to that degree. For example, the K3s distro has Traefik as the default ingress, but it's not quite as well documented (e.g. setting up your own custom wildcard SSL certificate).
Cannot really comment on the paid version, though. I can understand why it would be a thing, but it's also a bit concerning, as it usually is with open core and similar products.
I generally recommend Caddy over Nginx, but Nginx does still have certain advantages:
- Nginx supports OpenSSL commands that enable features like TLS record padding.
- Performance: better latency and scalability to more connections. Not everyone uses a CDN for static/cached content
- Kernel-accelerated TLS offload on Linux and FreeBSD
- Many existing modules provide unique functionality. The many modules for live video streaming and image processing are good examples.
- An ecosystem of patches with features like HPACK static dictionaries, dynamic TLS record sizing, etce
> …has terrible language integration.
Generally, "language integration" isn't really a use-case for vanilla Nginx; it's a use-case for Nginx Unit, an Nginx language-specific module, or OpenResty. I personally prefer the reverse-proxy route since it lets me use whatever language I want regardless of server support: Go, Rust, Python, C, etc.
If none of these are that important then I absolutely would not recommend Nginx; Caddy would be the better tool.
> People aren't writing internet scale software in lua for a reason.
I'd include Itch.io, much of Taobao, and some of the most popular API gateways (including Kong) in the category of "Internet-scale software written by 'people'".
- Performance: Requires nuanced tuning. Caddy performs competitively well for real world usage
- kTLS: Sacrifices memory safety.
- Existing modules: How do they perform compared to natively-compiled code? Caddy modules can do all that nginx modules can do, and more, but are natively compiled. I ran experiments with Caddy+Starlark that performed 2x as fast as Nginx+Lua.
I recommend users who link against OpenSSL to enable padding to multiples of at least 1024 bytes if they want to impede traffic analysis. The Nginx devs aren't interested in implementing random record padding or supporting the feature in BoringSSL/LibreSSL, unfortunately.
Can Caddy leverage either form of padding? If so, I might need to give it another look!
And regarding modules: most are written in C and dynamically loaded as shared objects or statically linked during compile-time. A bunch are listed at https://www.nginx.com/resources/wiki/modules/. The ones for live streaming and VODs are the hardest to replace, IMO. IPScrub was my favorite but I haven't used it for a few years.
Personally, I think live streaming and ffmpeg-based encoding are specialized enough to warrant a specialized server (like a custom Nginx build) and are a bit out of scope for a general-purpose user-friendly server like Caddy.
I'm not sure, I'd have to see what the crypto/tls package does.
I would push back against the notion that something like that is "out of scope" for a "general-purpose user-friendly server". Caddy is far from user friendly if you utilize its low-level JSON configuration API, and at its core, Caddy is an extensible server platform. Even its HTTP app is a plugin, and it can be extended to do frankly anything if you want it to. Streaming video is a use case that I know several people use it for already.
Redbean is a webserver. Go is not a webserver. Redbean cannot be "faster than Go". That's like saying that a car is faster than aluminum. Do you have benchmarks of Redbean versus Caddy that you can point to? I certainly haven't been able to find any.
I also haven't been able to find any in-depth independent benchmarks of Redbean. About the only thing I could find was someone struggling to reproduce the benchmark results.[0]
Personally, I've dealt with more than enough memory safety vulnerabilities caused by no programmer being good enough to write safe and correct C (or C++). Those vulnerabilities are easily prevented by choosing just about any other language people are likely to know about... so, I have no desire to deploy anything like Redbean when alternatives exist. It's cool to see the hacks they've been able to do with Cosmopolitan to make cross platform binaries, but hacks like that are not what I want for anything other than educational purposes, at least not until they've been more battle tested.
"If you want Rust-like promises then redbean can be compiled with ASAN memory safety. Cosmopolitan Libc has the only open source implementation of the Address Sanitizer runtime that's intended for production use. It causes redbean worker processes to crash and log a report should a bug like a buffer overrun, use-after-free, or stack smash occur. This has a marginal impact on performance. It can be useful in environments where interruptions in a service are more desirable than risking the system being compromised."
ASAN by itself cannot make those guarantees, from what I understand. Cosmopolitan would also have to make sure never to reuse virtual memory addresses for a start, and there’s actually even more to it than that. So, no, I’m not impressed by arbitrary claims like that. It’s such a bold claim. If it were true, Apple and Google would be adopting this ASAN implementation ASAP since they have tons of C and C++ code that would benefit from better safety at marginal performance cost.
In the mean time, if Redbean and Cosmopolitan want Rust-like promises of memory safety… they need to write it in Rust.
Uhm you realize Google invented ASAN? It's how they hunt for memory bugs in software like Chrome. If you love Rust, then we're actively working to port Rust to Cosmopolitan. https://github.com/ahgamut/rust-ape-example Rust in practice has `unsafe` code which needs something like ASAN to make it safer. In order to do that, you need a C library that has first-class support for ASAN where it isn't just an afterthought. I believe that in the future, Cosmopolitan Libc is going to help Rust be even more safe going forward.
I did specify “this ASAN implementation” with emphasis on the “this”, since it was presented higher in the thread as having marginal performance impact and excellent, rust-like safety. Google’s ASAN is not used on all production instances, last I heard, due to the large performance penalty. All sorts of memory safety issues slip past ASAN right now, especially when it is only used outside of production.
If ASAN were a silver bullet, C memory safety vulnerabilities would be a distant memory. They’re not. ASAN helps, but it isn’t enough.
lucaslorentz/caddy-docker-proxy works like Traefik, in that Container metadata labels are added to the reverse proxy configuration which is reloaded upon container events, which you can listen to when you subscribe to a Docker/Podman_v3 socket (which is unfortunately not read only)
So, with Caddy or Traefik, a container label can enable HTTP/3 (QUIC (UDP port 1704)) for just that container.
Note on both projects, if you care about security, you should split the generator/controller container out of the main webserver container, so the docker socket is not used in container that is directly exposed.
> servers/experimental_http3: Enable experimental HTTP/3 support. Note that HTTP/3 is not a finished standard and has extremely limited client support. This field is not subject to compatibility promises
TIL caddy has Prometheus metrics support (in addition to automatic LetsEncrypt X.509 Cert renewals)
Yeah those docs are for v2.5. The experimental_http3 option is removed in v2.6 (which is currently only in beta, stable release coming soon). To configure protocols and disable HTTP/3, you'll now use the "protocols" option (inside of the "servers" global option), and it would look like "protocols h1 h2" (the default is "h1 h2 h3").
This is a little-known awesome project! If you like how Traefik is configured with Docker labels but want to use Caddy, well now you can have your cake and eat it, too.
I tried caddy for my home setup. Replacing nginx and traefik. I am wondering if I should ditch them in production too. I am hesitant, cause I don't see caddy as "business" software: nginx has a learning curve and configuration to master, where caddy just does the job, sane defaults and not much to "work".
It just doesn't feel like work using caddy...
I have been using HAproxy and it has been very performant but the lack of documentation for its APIs may be the reason I will start playing with Caddy.
The documentation is extensive but I was unable to find tutorial that could help me dynamically add/remove backend servers using the dataplane API. Also being able to dynamically refresh certificate. These are pretty much the only reasons for why I am considering Caddy.
If you find the documentation insufficient or unclear, then this can be considered a documentation bug that I recommend filing here: https://github.com/haproxy/haproxy/issues
I'm curious about the implementation, and haven't looked at the source of quic-go yet: Does it use a single UDP socket to handle datagrams for all QUIC connections, does it use connected UDP sockets per connection, or does it use multiple UDP sockets, where each handle a certain set of connections - and where an external load balancer is required to redirect to the sockets?
Unfortunately there's no best answer for this: Using a single socket will allow for connection migration, but it will end up being a bottleneck in terms of scalability since it will serialize access on a lot of kernel and driver datastructures (just a single transmit/receive queue). Connected sockets avoid that, but don't allow for address migration. And doing external load balancing gets far more complex than just starting a binary - even the most simple solution requires running XDP code.
I think I'm stumbling over terminology here, but UDP is connectionless, right? I think the QUIC protocol re-implements connections on top of it, but I've never heard of "connected UDP".
A socket pool does seem to make sense from a performance standpoint!
UDP is connectionless. Connected UDP sockets are a OS construct, and don't influence anything that happens on the wire. If you "connect" a UDP socket, you don't have to use sendto and recvfrom anymore, and all packets received and sent (via send/recv APIs) are targetting the same peer. It's kind of just sets up some routes in the kernel, but doesn't make anything stateful or reliable at the networking layer.
Marten is in the process of making a bunch of optimizations [0], as are we within Caddy, and 2.6 includes some of those. It will only get better with time.
Most users do not operate at nearly the volume required to feel the impact, including enterprises.
Would be interested in repeating your experiments and knowing your real world use case and seeing how similar they are.
Yeah I agree this is a edge case, 99,99% of Caddy users don't push so much data ;)
But Caddy can't be used as a big files download server for instance, unless sticking with HTTP/1.1 (that and sendfile + kTLS not been supported last time I checked?) (at least with a single http instance).
Another issue for HTTP/3 is https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-B...
Current Linux in WSL for instance has this issue.
This will probably resolves over time as more and more Linux distros will take HTTP/3 requirements into account.
Caddy 2.6 has some optimizations for serving large files, including sendfile, but other optimizations too. No ktls yet, but maybe someday. It's just... kinda gnarly.
with BPS you mean throughput on a single request? On an unconstrained connection (like loopback)? In this case you are mostly measuring CPU efficiency, since for a real world deployment the network connection would likely be the bottleneck. In that real world setup then loss of CPU efficiency becomes a scaling challenge, which actually makes it less RPS.
Btw: A good recommendation for anyone doing benchmarking is to both measure throughput for a single connection, but also throughput for the whole server when using lots of connections (e.g. 100 connections per core, and each connection doing multiple concurrent requests). The latter will point out further scaling issues.
In terms of full system efficiency HTTP/2 vs HTTP/1 (with TLS) is actually not that bad for good implementations - might have 10-50% more cost but in the end the performance critical parts (doing syscalls for TCP sockets) are the same for both. QUIC can be much worse. Well optimized setups (which e.g. use optimized UDP APIs with GSO or even XDP) somewhere between 50-100% more expensive than TCP+TLS setups. And simpler implementations can be 5x as expensive, and might not even scale beyond a single CPU core.
- httpx is mostly requests-compatible, supports client-side caching, and HTTP/1.1 & HTTP/2, and here's the issue for HTTP/3 support:
https://github.com/encode/httpx/issues/275
Since address validation is about blocking senders which forge their IP address, I think the number of connection attempts which where the client doesn't eventually validate its address should factor into the decision to enable this feature. This should rarely happen for legitimate clients (e.g. connection loss/cancellation during the first roundtrip) but always happen for IP address forgers.
Or perhaps simply use the number of half-open/embryonic connections as the metric.
Good point. Our current metric (number of active requests) is a bit naive, but mostly does the job essentially for free. If your proposed metric doesn't cost too much in memory, it might be a good fit.
How does such an attacker influence your active requests metric? Is it incremented when the initial UDP packet arrives (even though the http request itself hasn't arrived yet) and decremented when it times out?
No, currently we increment it when the HTTP request is parsed. (It counts HTTP requests, not packets/connections.) This was a quick and easy way to count; but I am very open to improving this! Feel free to submit a proposal or a PR.
In that case, I don't think your metric works at all, since legitimate users will increase it, but attackers won't. During an attack it will either remain the same, or even go down as fewer legitimate users manage to connect.
This attack is the QUIC equivalent of a SYN flood, which results in half-open connections, because the attacker is unable to complete the connection by responding to message the server sends. RequireAddressValidation corresponds enabling syn cookies.
Unfortunately I don't use caddy myself, and don't have time to get involved deeper in improving this.
I'm really glad to hear this is an option. But will browsers accessing the website accept the self-signed certs when setting up the TLS connection required by QUIC? I don't know how they behave. I'm genuinely asking.
Yes they do, as long as you installed Caddy's root CA cert in the browser's trust store. Caddy will attempt to install it automatically with https://github.com/smallstep/truststore if possible (usually requires root) but if it fails you can try again with "sudo caddy trust". You might be using a client or trust store that isn't supported though, in which case you'll need to install the root cert manually.
What if I just want to generate and use a self signed cert without any CA involvement? I do that with HTTP all the time and it works for the public web even if the browsers like to scaremonger about it.
Does a QUIC connection work without a CA (corporate) root cert being referenced?
To be clear, Caddy itself manages its own internal CA. It's the same thing as self signing except you have a root cert you can trust, to trust every cert Caddy generates; that's instead of having individual self signed certificates. This makes it significantly easier if you have many subdomains for each of your services, etc. It has nothing to do with corporate CAs, no external entities.
QUIC/HTTP3 use the same TLS stack as H1/H2, same things are trusted. So there's no difference there.
(Fun-fact, "self-signed" certs also have "CA involvement" -- it's just that the leaf is also the signing authority, which is generally bad practice. Caddy does this properly with a separate signing root -- that stays offline and has long validity -- as Francis explained.)
Thanks to Marten Seemann for maintaining the quic-go library we use. (I still haven't heard whether Go will add HTTP/3 to the standard library.)
Caddy 2.6 should be the first stable release of a general-purpose server to support and enable standardized HTTP/3 by default. HTTP versions can be toggled on or off. (Meaning you can serve only HTTP/3 exclusively if you're hard-core.)
PS. Caddy 2.6 will be our biggest release since 2.0. My draft release notes are about 23 KB. We're looking at huge performance improvements and powerful new features like events, virtual file systems, HTTP 103 Early Hints, and a lot of other enhancements I'm excited to show off on behalf of our collaborators!