Hacker News new | past | comments | ask | show | jobs | submit | jchw's comments login

Heard about this recently. Extremely fun and interesting. Would really like to try it, not sure if I should try it on a GameCube or Wii.

I believe non-x86 versions of Windows NT came with MS-DOS emulation in the form of SoftPC, at least in NT4. If anyone happens to have an appropriate copy of SoftWindows to try, that sounds like potentially even more fun.


DEC made fx!32 to allow x86 emulation on alpha only. They didn’t tackle ppc or mips.

https://en.wikipedia.org/wiki/FX!32



> The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you’ll get an error. You can fix that yourself or you can feed the error back into the LLM and watch it correct itself.

Interestingly though, this only works if there is an error. There are cases where you will not get an error; consider a loosely typed programming language like JS or Python, or simply any programming language when some of the API interface is unstructured, like using stringly-typed information (e.g. Go struct tags.) In some cases, this will just silently do nothing. In other cases, it might blow up at runtime, but that does still require you to hit the code path to trigger it, and maybe you don't have 100% test coverage.

So I'd argue hallucinations are not always safe, either. The scariest thing about LLMs in my mind is just the fact that they have completely different failure modes from humans, making it much harder to reason about exactly how "competent" they are: even humans are extremely difficult to compare with regards to competency, but when you throw in the alien behavior of LLMs, there's just no sense of it.

And btw, it is not true that feeding an error into an LLM will always result in it correcting the error. I've been using LLMs experimentally and even trying to guide it towards solving problems I know how to solve, sometimes it simply can't, and will just make a bigger and bigger mess. Due to the way LLMs confidently pretend to know the exact answer ahead of time, presumably due to the way they're trained, they will confidently do things that would make more sense to try and then undo when they don't work, like trying to mess with the linker order or add dependencies to a target to fix undefined reference errors (which are actually caused by e.g. ABI issues.) I still think LLMs are a useful programming tool, but we could use a bit more reality. If LLMs were as good as people sometimes imply, I'd expect an explosion in quality software to show up. (There are exceptions of course. I believe the first versions of Stirling PDF were GPT-generated so long ago.) I mean, machine-generated illustrations have flooded the Internet despite their shortcomings, but programming with AI assistance remains tricky and not yet the force multiplier it is often made out to be. I do not believe AI-assisted coding has hit its Stable Diffusion moment, if you will.

Now whether it will or not, is another story. Seems like the odds aren't that bad, but I do question if the architectures we have today are really the ones that'll take us there. Either way, if it happens, I'll see you all at the unemployment line.


Somewhat relatedly, I've always appreciated Go adding some runtime checks to detect race conditions, like concurrent map writes. It's not as good as having safe concurrency from the ground up, but on the other hand it's a lot better than not having detection, as everyone makes mistakes from time to time, and imperfect as it may be, the detection usually does catch it quickly. Especially nice since it is on in your production builds; a big obstacle with a lot of debugging tools is they're hard to get them where you need them...

Most of the other "forks" (e.g. Librewolf) are just patches on top of vanilla Firefox sources, so it's really not a whole lot to scrutinize by hand. I've skimmed at least most of the patch files personally just out of curiosity. In my distro of choice, NixOS, the sources are built by Hydra or my local machine, so I'm not trusting that their binaries match the source either.

That makes it a bit easier to trust, but it does run into the issue that it stops working if Mozilla hits a certain level of untrustworthiness.


I'm personally hoping for Ladybird and Servo to eventually grow into usable browsers. Ladybird is making faster progress on actually delivering a browser to end users, but if Servo gains foothold in the embeddable use case it should help them gain more stakeholders with an interest in keeping development moving forward.

I wish to some day have as much optimism about anything as you have about Mozilla. How you can see the ToS change and this diff and conclude that the outrage is astroturfing is difficult for me to grasp. Both this FAQ entry and the ToS are specifically about the Firefox browser, the wording was unambiguous...

They literally went and deleted a paragraph that said Firefox would "never" sell your personal data. If they needed to clarify a technicality, they wouldn't need to delete that.


They replaced that paragraph with this:

> Mozilla doesn’t sell data about you (in the way that most people think about “selling data“), and we don’t buy data about you. Since we strive for transparency, and the LEGAL definition of “sale of data“ is extremely broad in some places, we’ve had to step back from making the definitive statements you know and love. ...

There are valid complaints you could have about this change (for example, I wish they were more specific about the potential legal issues), but calling this selling your soul is unironically bad faith trolling.


And now without cutting it conveniently before the fun bit:

> Mozilla doesn’t sell data about you (in the way that most people think about “selling data“), and we don’t buy data about you. Since we strive for transparency, and the LEGAL definition of “sale of data“ is extremely broad in some places, we’ve had to step back from making the definitive statements you know and love. We still put a lot of work into making sure that the data that we share with our partners (which we need to do to make Firefox commercially viable) is stripped of any identifying information, or shared only in the aggregate, or is put through our privacy preserving technologies (like OHTTP).

You appear to have cut off the part where they say that actually yeah they have to stop saying they don't sell your data because they are selling your data.


No, actually, they don't say that. They very clearly say that they don't (and don't believe most people will) consider what they are doing "selling your data", but that it may legally considered selling your data in some countries.

For example, Firefox runs ads using your language and city/country (on the default new tab page) - but no other data. I think the vast majority people would fine with the privacy implications of that, but this may be legally considered selling your data.


> Mozilla doesn’t sell data about you (in the way that most people think about “selling data“)

> calling this selling your soul is unironically bad faith trolling.

It's not. One of their biggest selling (!) points is that they are privacy focused so when they make these changes, it is extra alarming. It's not like, e.g. Google, saying the same thing (which would be equally shocking but for opposite reasons.


The paragraph I'm referring to in that diff does not appear to be replaced directly by anything, it just got removed. They did, however, add that non-answer paragraph separately, apparently hours ago.

Reading between the lines, it's pretty obvious. They're making steps in a certain direction. Enshittification doesn't usually happen entirely overnight, but you don't have to extrapolate a whole lot to see the blatantly obvious eventuality this is all pointing to. This is well beyond typical levels of brazen for a first step.

Realistically, Google removing "Don't be evil" also didn't really mean anything directly, either... but that doesn't mean it doesn't tell you anything.


You missed off the important part:

"the data that we share with our partners (which we need to do to make Firefox commercially viable) is stripped of any identifying information, or shared only in the aggregate"


It's mildly amusing that AO3 is considered more palatable than booru websites. Both AO3 and many of the boorus have content that would be hard for most folks to stomach. I guess AO3 has the advantage of being mostly text, (which seems to be generally less visceral to people,) but it is also nearly completely uncensored, since that's kind of the point of it.

Yes, overall I wouldn't describe either of them as particularly palatable to a general audience, haha. Your hunch is the same as mine -- text has much less potential for instant shock. On AO3 you have to, sort of, go out of your way (by reading, or searching for specific tags, etc.) to hit the unsavory stuff.

Even if you did find the unsavoury stuff on Ao3 it might take a while to get to it in the story whereas in an image you could be blasted with the sin of more than a thousand words all at once

Personally, I'd like to see more HTTP/2 support. I think HTTP/2's duplex streams would be useful, just like SSE. In theory, WebSockets do cover the same ground, and there's also a way to use WebSockets over HTTP/2 although I'm not 100% sure how that works. HTTP/2 though, elegantly handles all of it, and although it's a bit complicated compared to HTTP/1.1, it's actually simpler than WebSockets, at least in some ways, and follows the usual conventions for CORS/etc.

The problem? Well, browsers don't have a JS API for bidirectional HTTP/2 streaming, and many don't see the point, like this article expresses. NGINX doesn't support end-to-end HTTP/2. Feels like a bit of a shame, as the streaming aspect of HTTP/2 is a more natural evolution of the HTTP/1 request/response cycle versus things like WebSockets and WebRTC data channels. Oh well.


Duplex streams are not really a HTTP/2-only feature. You can do the same bidirectional streaming with HTTP/1.1 too. The flow is always: 1. The client sends a header set. 2. It can then start to stream data in the form of an unlimited-length byte-stream to the server. 3. The server starts to send a header set back to the client. 4. The server can then start to stream data in the form of an unlimited-length byte-stream to the client.

There is not even a fixed order between 2) and 3). The server can start sending headers or body data before the client sent any body byte.

What is correct is that a lot of servers and clients (including javascript in browsers) don't support this and make stricter assumptions regarding how HTTP requests are used - e.g. that the request bytes are fully sent before the response happens. I think ReadableStream/WritableStream APIs on browsers were supposed to change that, but I haven't followed the progress in the last few years.

NGINX falls into the same category. It's HTTP/2 support (and gRPC support) had been built with a very limited use-case in mind. That's also why various CDNs and service meshes use different kinds of HTTP proxies - so that various streaming workloads don't break in case way the protocol is used is not strictly request->response.


No browser I'm aware of is planning on allowing the request and response bodies to be streamed simultaneously for the same request using ReadableStream and WriteableStream. When using streaming request bodies, you have to set the request explicitly to half-duplex.

Anyways, yes, this is technically true, but the streaming semantics are not really that well-defined for HTTP/1.1, probably because it was simply never envisioned. The HTTP/1.1 request and response were viewed as unary entities and the fact that their contents were streamed was mostly an implementation detail. Most HTTP/1.1 software, not just browsers, ultimately treat the requests and responses of HTTP as different and distinct phases. For most uses of HTTP, this makes sense. e.g. for a form post, the entire request entity is going to need to be read before the status can possibly be known.

Even if we do allow bidirectional full-duplex streaming over HTTP/1.1, it will block an entire TCP connection for a given hostname, since HTTP/1.1 is not multiplexed. This is true even if the connection isn't particularly busy. Obviously, this is still an issue even with long-polling, but that's all the more reason why HTTP/2 is simply nicer.

NGINX may always be stuck in an old school HTTP/1 mindset, but modern software like Envoy shows a lot of promise for how architecting around HTTP/2 can work and bring advantages while remaining fully backwards compatible with HTTP/1 software.


> I think ReadableStream/WritableStream APIs on browsers were supposed to change that, but I haven't followed the progress in the last few years.

There has been a lot of pushback against supporting full-duplex streams[0].

[0]: https://github.com/whatwg/fetch/issues/1254


> a lot of servers and clients (including javascript in browsers) don't support this

To say nothing about the many http proxies in between.


HTTP2 works great on the LAN, or if you have really good network.

It starts to really perform badly when you have dropped packets. So any kind of medium quality wifi or 4/5g kneecaps performance.

It was always going to do this, and as webpages get bigger, the performance degradation increases.

HTTP2 fundamentally underperforms in the real world, and noticeably so on mobile. (My company enthusiastically rolled out http2 support when akamai enabled it.)

Personally I feel that websockets are a hack, and frankly HTTP 3 should have been split into three: a file access protocol, a arbitrary TCP like pipe and a metadata channel. But web people love hammering workarounds onto workarounds. so we are left with HTTP3


HTTP/2, in my experience, still works fine on decent connections, but the advantages definitely start to level out as the connection gets worse. HTTP/2 definitely has some inherent disadvantages over HTTP/1 in those regards. (Though it depends on how much you are constrained by bandwidth vs latency, to be sure.)

However, HTTP/3 solves that problem and performs very well on both poor quality and good quality networks.

Typically, I use HTTP/2 to refer to both HTTP/2 and HTTP/3 since they are basically the same protocol with different transports. Most people don't really need to care about the distinction, although I guess since it doesn't use TCP there are cases where someone may not be able to establish an HTTP/3 connection to a server. Still, I think the forward looking way to go is to try to push towards HTTP/3, then fall back to HTTP/2, and still support HTTP/1.1 indefinitely for simple and legacy clients. Some clients may get less than ideal performance, but you get the other benefits of HTTP/2 on as many devices as possible.


> HTTP 3 should have been split into three: a file access protocol, a arbitrary TCP like pipe and a metadata channel

HTTP3 is basically just HTTP2 on top of QUIC… so you already have the tcp-like pipe, it’s called QUIC. And there’s no reason to have a metadata channel when there are already arbitrary separate channels in QUIC itself.


You're right of course, it has virtual channels. I just think it would have been good to break with the old HTTP semantics and change to something reflecting modern usage.

It seems like the author is agreeing that HTTP/2 is great (or at least good) for browser -> web server communication, but not useful for the REST-style APIs that pervade modern app design. He makes a good case, but HTTP was never really a good choice for API transport _either_, it just took hold because it was ubiquitous.

What's the difference? Aren't they both request-response protocols?

It's not terrible, it just has a lot of fluff that you don't really need or want in an API call like chunked transfer encoding, request pipelining, redirect responses, mime encoding, content type negotiation, etc. Of course, you can just ignore all that stuff and either a) implement a stripped down, incomplete HTTP-ish protocol that has just the parts you need or b) use a full-blown HTTP implementation like nginx like most people do. The problem with (b) is when nginx suddenly starts behaving like an actual web server and you have to troubleshoot why it's now barfing on some UTF-16 encoding.

There are many other transport protocols for APIs, http basically took the lead in the early days because it made it through firewalls/proxies; not because it is better.

Yeah, it's a shame you can't take advantage of natural HTTP/2 streaming from the browser. There's the upcoming WebTransport API (https://developer.mozilla.org/en-US/docs/Web/API/WebTranspor...), but it could have been added earlier.

If you want to stream data inside a HTTP body (of any protocol), then the ReadableStream/WritableStream APIs would be the appropriate APIs (https://developer.mozilla.org/en-US/docs/Web/API/Streams_API) - however at least in the past they have not been fully standardized and implemented by browsers. Not sure what the latest state is.

WebTransport is a bit different - it offers raw QUIC streams that are running concurrently with the requests/streams that carry the HTTP/3 requests on shared underlying HTTP/3 connections and it also offers a datagram API.


I think the problem is that duplex communication on the web is rarely useful except in some special cases, and usually harder to scale as you have to keep state around and can't as easily rotate servers.

Some applications it is important but for most websites the benefits just dont outweigh the costs.


I thought http/2 was great for reducing latency for JS libraries like Turbo Links and Hotwire.

Which is why the Rails crowd want it.

Is that not the case?


H2 still suffers from head of line blocking on unstable connections (like mobile).

H3 is supposed to solve that.


Yep. Actually H1/H2/H3 do have the same problem (remember the good old days when everyone was trying to pipeline over H1?), except that H1 generally comes with multiple connections and H3 currently goes over QUIC and it's QUIC that addresses HoL by letting streams progress independently.

[flagged]


I have an nginx running on my VPS supporting my startup. Last time I had to touch it was about 4 years ago. Quality software

I really like Caddy, but these nginx performance comparisons are never really supported in benchmarks.

There have been numerous attempts to benchmark both (One example: https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-c... ) but the conclusion is almost always that they're fairly similar.

The big difference for simple applications is that Caddy is easier to set up, and nginx has a smaller memory footprint. Performance is similar between the two.


AFAIK both proxies are capable of serving at line rate for 10Gbps or more at millions of concurrent connections. I can't possibly see how performance would significantly differ if they're properly configured.

nginx's memory footprint is tiny for what it delivers. A common pattern I see for homelab and self-hosted stuff is a lightweight bastion VPS in a cloud somewhere proxying requests to more capabile on-premise hardware over a VPN link. Using a cheap < $5mo means 1GB or less of RAM, so you have to tightly watch what is running on that host.

To be fair 1GB is a lot, both caddy and nginx would feel pretty good with it I'd imagine.

1 GB should be way more than either should need. I run nginx, unbound, postfix, dovecot plus all the normal suff (ssh, systemd, etc) for a Linux system on a VPS w/ 500MB of RAM. Currently the system has ~270MB used. It actually has 1GB available due to a plan auto-upgrade but have never bothered as I just don't need it.

1GB would be for everything running on the server, not just the reverse proxy.

For small personal projects, you don't usually buy a $5/month VPS just to use as a dedicated reverse proxy.


1GB for a VPC that runs an HTTP load balancer/reverse proxy and a handful of IPsec or Wireguard tunnels back to the app servers (origin) is overkill. You could successfully run that in 512MB, and probably even 256MB. (That's the scenario described).

What needs to run on this that's a memory hog making 512MB too small? By my (very rough) calculations youd need 50-100MB for kernel + systemd + sshd + nginx base needs + tunnels home. That leaves the rest for per-request processing.

Each request starts needing enough RAM to parse the https headers into a request object, open a connection back to the origin, and buffer a little bit of traffic that comes in while that request is being processed/origin connection opens. After that you only need to maintian 2 connections plus some buffer space - Generously 50KB initially and 10KB ongoing. There's enough space for a thousand concurrent requests in the ram not used by the system. Proxying is fairly cheap - the app servers (at the origin) may need much much more, but that's not the point of the VPS being discussed.

Also worth noting that the cheap VPS is not a per-project cost - that is the reverse proxy that handles all HTTP traffic into your homelab.


Why would you use either when there is OpenBSD w/ carp + HAProxy?

There's lots of options out there. I mean, even IIS can do RP work.

Ultimately, I would prefer a PaaS solution over having to run a couple of servers.


You're going to need to show your homework for this to be a credible claim.

Is it just me or did anyone else completely miss Caddy for it's opening sentence?

>Caddy is a powerful, extensible platform to serve your sites, services, and apps, written in Go.

To me it reads that if your application is not written in Go, don't bother


The Go crowd, like the Rust crowd, likes to advertise the language their software is written in. I agree that that specific sentence is a bit ambiguous, though, as if it's some kind of middleware that hooks into Go applications.

It's not, it's just another standalone reverse proxy.


> The Go crowd, like the Rust crowd, likes to advertise the language their software is written in.

Probably because end users appreciate that usually that means a single binary + config file and off you go. No dependency hell, setting up third party repos, etc.


> Probably because end users appreciate that usually that means a single binary + config file and off you go. No dependency hell, setting up third party repos, etc.

Until you have to use some plugin (e.g. cloudflare to manage DNS for ACME checks), now it's exactly "dependency hell, setting up third party repos, etc."

I also fully expect to see a few crashes from unchecked `err` in pretty much any Go software. Also, nginx qualifies for `single binary + config`, it's just NGINX is for infra people and Caddy is for application developers.


Fortunately I don't think any of that applies to Rust ;-)

Actually, all of it applies to rust. The only stable ABI in Rust is C-ABI and IMO at that point it stops being rust. Even dynamically loading rustlib in rust application is unsafe and only expected to work when both compiled with the same version. In plugins context, it's the same as what Caddy making you do.

However, Rust Evangelical Strike Force successfully infiltrated WASM committee and when WASM Components stabilize, it can be used for plugins in some cases (see Zed and zellij). (Go can use them as well, rust is just the first (only?) to support preview-2 components model.


Yeah, I don't really do dynamic loading in my corner of Rust. And I can always target some MSRV, cargo package versions, and be happy with it. Definitely beats the dependency hell I've had to deal with elsewhere

Don't get me wrong, I love rust and use it almost every day. Doing `cargo run` in a project it handles everything is good. This gets lost once you start working in a plugin context. Because now you're not dealing in your neatly organized workplace, you're working across multiple workplaces from different people.

IIRC it's more than just MSRV or even matching version exactly. It also requires flags that were used to compile rustc match (there is an escape hatch tho).


When I see software written in Go, I know that it has a very sad plugin support story.

Terraform providers seem to work pretty well, but as far as I know, they're basically separate executables and the main process communicates with them using sockets.

Yes, works very well for terraform. You probably can see why it's not going to work for a webserver?

Why should a reverse proxy give a single shit about what your lang application is written in

It shouldn't, which is why I think wording there is strange. Nginx doesen't market itself as "platform to serve your sites, services, and apps, written in C". Reading the first sentence I don't even know what Caddy is, what does a platform mean in this context? Arriving on Nginx's site the first sentence visible to me is

>nginx ("engine x") is an HTTP web server, reverse proxy, content cache, load balancer, TCP/UDP proxy server, and mail proxy server.

Which is perfect


when it says 'written in Go', the subtext is - i'm fast, i'm new, i'm modern, go buddies love me please

the better one is 'written in rust', the subtext is - i'm fast, i'm new, i'm futurism, and i'm memory-safe, rust buddies love me please

--- cynicism end ---

i do think sometimes it's worth to note the underlying tech stack, for example, when a web server claims it's based on libev, i know it's non-blocking


Back when Caddy first came out over 10 years ago, the fact that it was written in Go was just simply more notable. For Go, it also at least tells you the software is in a memory-safe programming language. Now neither of those things is really all that notable, for new software.

I didn't even read the article, but I love the comments on the thread.

Yes. The implementation language of a system should not matter to people in the least. However, they are used as a form of prestige by developers and, sometimes, as a consumer warning label by practitioners.

"Ugh. This was written in <language-I-hate>."

"Ooo! This was written in <language-I-love>!"


There's certainly some aspect of that going on, but I think mainly it's just notable when you write something in a programming language that is relatively new.

Does it matter? In theory no, since you can write pretty much anything in pretty much any language. In practice... It's not quite that black and white. Some programming languages have better tooling than others; like, if a project is written in pure Go, it's going to be a shitload easier to cross compile than a C++ project in most cases. A memory-safe programming language like Go or Rust will tell you about the likely characteristics of the program: the bugs are not likely to be memory or stack corruption bugs since most of the code can't really do that. A GC'd language like Go or Java will tell you that the program will not be ideal for very low latency requirements, most likely. Some languages, like Python, are languages that many would consider easy to hack on, but on the other hand a program written in Python probably doesn't have the best performance characteristics, because CPython is not the fastest interpreter. The discipline that is encouraged by some software ecosystems will also play a role in the quality of software; let's be honest, everyone knows that you CAN write quality software in PHP, but the fact that it isn't easy certainly says something. There's nothing wrong with Erlang but you may need to learn about deploying BEAM in production before actually using Erlang software, since it has its own unique quirks.

And this is all predicated on the idea that nobody ever introduces a project as being "written in C." While it's definitely less common, you definitely do see projects that do this. Generally the programming language is more of a focus for projects that are earlier in their life and not as refined as finished products. I think one reason why it was less common in the past is because writing that something is written in C would just be weird. Of course it's written in C, why would anyone assume otherwise? It would be a lot more notable, at that point, if it wasn't.

I get why people look at this in a cynical way but I think the cynical outlook is only part of the story. In actuality, you do get some useful information sometimes out of knowing what language something is written in.


Pretty sure Ruby on Rails sites were the same way.

Python certainly was too, back in the day. It feels like it's roughly a "first 10 years of the language" thing, maybe stretched another 5 if there's an underdog aspect (like being interpreted.)

> ... and i'm memory-safe...

Go is memory safe..


For an open source product, it's fun to say "written in X language". It also advertises the project to developers who may be willing to contribute.

If you put "product made with Go", I'm not going to contribute as I don't know Go, though that wouldn't prevent me from using it should it fit my needs. But if you wrote your project in .NET, I may certainly be willing to contribute.


> elegantly

There is a distinct lack of elegance in the HTTP/2 protocol. It's exceptionally complex and it has plenty of holes in it. That it simply does a job does not earn it "elegant."


Honestly, I don't understand this critique. The actual protocol is pretty straight-forward for what it does. I'm not sure it can be much simpler given the inflexible requirements. I find it more elegant than HTTP/1.1.

Versus HTTP/1.1, some details are simplified by moving the request and status line parts into headers. The same HEADERS frame type can be used for both the headers and trailers on a given stream. The framing protocol itself doesn't really have a whole lot of cruft, and versus HTTP/1 it entirely eliminates the need for the dancing around with Content-Length, chunked Transfer-Encoding, and trailers.

In practice, a lot of the issues around HTTP/2 implementations really just seem to be caused by trying to shoehorn it into existing HTTP/1.1 frameworks, where the differences just don't mesh very well (e.g. Go has some ugly problems here) or just simply a lack of battle-testing due to trouble adopting it (which I personally think is mainly caused by the difficulty of configuring it. Most systems will only use HTTP/2 by default over TLS, after all, so in many cases end-to-end HTTP/2 wasn't being tested.)


From the perspective of a user, where it starts to seem inelegant is when grpc comes into the picture. You get grpc to function but then plain http traffic breaks and vice versa. It seems to be odd implementation details on specific load balancer products. When in theory, all of it should operate the same way, but it doesn’t.

grpc doesn't really do anything special on top of http/2. Load balancers that are aware of http/2 on both sides shouldn't have any trouble with either.

The problem that people run into load balancing grpc is that they try to use a layer 4 load balancer to balance layer 7 requests; that is, if there are 4 backends, the load balancer tells you the address of one of them, and then you wonder why the other 3 backends don't get 25% of the traffic. That's because grpc uses 1 TCP connection and it sends multiple requests over that connection ("channel"). If your load balancer tells you the addresses of all 4 servers, then you can open up 4 channels and load balance inside your application (this was always the preferred approach at google, with a control channel to gracefully drain certain backends, etc.). If your load balancer is aware of http/2 at the protocol level (layer 7), then you open up one channel to your load balancer, which already has one channel for each backend. When a request arrives, it inspects it and picks a backend and proxies the rest of the exchange.

Ordinary http/2 works like this, it's just that you can get away with a network load balancer because http clients open new connections more regularly (consider the lifetime of a browser page with the lifetime of a backend daemon). Each new connection is a load balancing opportunity for the naive layer 4 balancer. If you never make new connections, then it never has an opportunity to load balance.

grpc has plenty of complexity for "let applications do their own load balancing", including built-in load balancing algorithms and built-in service discovery and health discovery (xDS); http/2 doesn't have any of this. Whether these are actually part of grpc or just random add-ons to popular client libraries is somewhat up for debate, however.


The same arguments apply to WebSockets probably— yes the implementation is a little hairy, but if the end result is a clean abstraction that does a good job of hiding that complexity from the rest of the stack, then it's elegant.

Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.

Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.


> It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores

Their customers would be forced to give their business to Honest Achmed[1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=647959


Thank you, that was an awesome laugh this morning, that request was genius:

> 2. Sub CAs Operated by 3rd Parties

> Honest Achmed's uncles may invite some of their friends to issue certificates as well, in particular their cousins Refik and Abdi or "RA" as they're known. Honest Achmed's uncles assure us that their RA can be trusted, apart from that one time when they lent them the keys to the car, but that was a one-off that won't happen again.


Someone should start a real CA business, complete with all the proper audits and everything to get it into the browser trust stores… and call it “Honest Achmed’s Used Cars and Certificates” (have it buy some random used car dealership in the middle of nowhere so the name is not a lie)

Where’s a billionaire troll when you need one? (And a form of billionaire trolling that would upset a lot less people than Musk’s version of it.)

Would be even funnier if the billionaire’s name actually was Achmed


What has prevented this (a good legit CA co) from happening before now?

Where's the value?

Anyone can get a cert from lets-encrypt. No users of your website care how trustworthy your CA is. And CA's are too big to fail, so you barely need to worry about your CA being removed by the trust store. So you can only compete on price.

If we want to change this, we need a way for a certificate to be co-signed by multiple CA's. So that the certificate can be presented to the user, and they can figure out if any trusted CA of theirs has signed the certificate. This way, revoking trust in a CA becomes easier, because people should have multiple signatures on their certificate. That means, all of a sudden, that the quality of a CA actually matters.

Whilst it might seem this is already possible, it is not. Cross-signing is only a thing for intermediate certificates, and does not work. You can also have multiple certificates for the same key, but when starting a TLS session, you must present a single certificate. (This seems to have changed for TLS 1.3, so perhaps this is already possible?)


I think TLS 1.3 still requires the end entity (server/client) cert first. All the other certs can now be in any order and the verification is suppoaed to figure out a valid path.

In theory, you could make an intermediate CA and get cross signed certs from multiple CAs (hopefully with Name Constraints), your intermediate CA signs your server cert, and you include all your cross certs and the intermediate certs for those. And the client figures out which chains it can make and if it likes any of them.

But experience has shown, verification may find a chain with signatures that line up, but the CA is expired or revoked in the local trust store, and reject the verification, even though another chain could have been found with the provided certificates.

And, because of the limited information in tls handshakes from real world clients, it's difficult (maybe impossible) to know which CAs a client would accept, so that the server could present an appropriate chain.


The obvious solution is for TLS 1.4 to mandate that all certificate chains contain at least one expired certificate.

Time and money. Plus right now even if you bought the infra, the staff, paid for and passed the audits, and then waiting while Apple, Mozilla, Google and Oracle (at least) included your roots...Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again. You could purchase a root off an existing CA, subject to the trust stores approving it, and the boatload of cash you'd need to buy it (plus still having the staff and infra to operate it).

> Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again.

Interesting, I hadn’t heard that, is there anywhere I can read more about it?

> You could purchase a root off an existing CA,

As well as a sub-CA, I remember in theory you can have two independent CAs with cross-signing… but do browsers actually support that?


"Nothing public to point out to" is probably accurate but it is noted publicly here: https://learn.microsoft.com/en-us/security/trusted-root/new-...

Nothing public to point to, sorry.

Sub-CAs: Not really. Operational risk to the parent CA is huge, you'd be hard pressed to get any current public CA to sign an issuing CA to be operated externally. Cross-signing still works (though it is the stuff of nightmares in many cases) but again you have to have money and a CA willing to do it!


Entrust is/was doing it with ssl.com after they were detrusted.

No, SSLCorp are hosting and managing a CA with Entrust branding. Same as Sectigo are doing. Entrust aren't doing issuance, verification - they're straight reselling from white-labeled issuing CAs.

Let's Encrypt has been around for a decade and is doing pretty well for itself. The manual validation required for EV and OV certificates precludes it from being a passion project, unless managing a call center and a squad of document reviewers is your dream job.

Let's Encrypt doesn't issue EV and OV. Honest Achmed’s Used Cars and Certificates wouldn't have to either.

It wouldn't be fun if Honest Achmed’s Used Cars and Certificates didn't also issue EV and OV certificates. It wouldn't be on brand to miss out on entire segments of the certificate market.

I'm actually curious why that wasn't approved

To my understanding, the point of Honest Achmed was to demonstrate that it was possible to write a facially reasonable and compliant CA inclusion request that was also intuitively unacceptable. It successfully demonstrated that the inclusion policies at the time needed to be made more rigorous, which they were.

> It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.

Trust stores and Browsers specifically have more options than simply remove a CA. A more reasonable approach for distrust process of a CA this size would be to stop accepting any new certificates issued after a certain date. Then existing customers will have a heads up and even if asleep will find out the bad news during scheduled renewal rather than while the responsible employees are on vacation.


It's refreshing to see folks who're obviously used to hiding behind clouds of bullshit get skewered by people who both know enough to see through it and have the time and energy to follow every thread to completion.

The most recent digicert thread smells suspiciously similar to those that lead up to the Entrust debacle.


Is there any option to turn off trust for all certificates generated after a certain date? The ideal would be to let old certificates keep working, and not trust new ones.

It's been done previously, e.g. for Symantec:

https://security.googleblog.com/2017/09/chromes-plan-to-dist...


Yes, this has been done a few times in the past. It's unfortunate that "distrust after X date" and "distrust now" are the only penalties that currently exist but that's the reality of the current PKI world.

There are a lot of efforts to get name constraints widely supported, so a given CA can only issue for certain TLDs or even domains, but it's still not there. That could be a viable punishment for misbehaving CAs in the future, or even an intentionally chosen restriction for CAs that want less strict requirements. Government CAs could be a very useful application, if a CA cert could be name constrained to only issue for .gov.cctld at that point we could basically just let them do whatever they want without concern. Likewise for any small regional CAs, if they could be limited to only regional TLDs their blast radius is reduced which also could mean less strict requirements than those applied to CAs able to sign for .com.


It would also be really useful if I could add a "root" CA for my employer's private CA to sign certificates for say .internal or .example.com, but not allow them to sign certs for any arbitrary domain. There's decent support for constraints on intermediate certs, but then you have to trust the root won't ever be used to sign unconstrained intermediates.

It would also be useful for localhost. Create a private CA that is constrained to localhost and *.local, and you don't have to worry about it being used to MitM other sites if it gets compromised.


I 100% agree, and I also think it'd be a better solution than wildcard certs in most cases where they're used. I would love to be able to have a personal CA cert that was name constrained to mydomain.tld for the duration of my registration of that domain which I could then use to issue certs for subdomain.mydomain.tld for individual services instead of having to have one *.mydomain.tld which hypothetically might allow a single compromised service to be turned in to the ability to impersonate any of my services.

Obviously having a glut of new private CAs would cause scalability issues for CT logs that are already having issues keeping up with current uses, perhaps CT requirements could be reduced for such single-domain CAs in combination with limited lifetimes.


It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores.

How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things?

It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.


I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way.

I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.


> The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.

I'd imagine that they wouldn't do it to "take a stand" so much as "avoid the risk of getting their stuff broken in the short term" in this scenario, regardless of whichever party loses the blame game. See: the recent WordPress drama, which has turned customers away from both parties involved.


As I understand it, the way certificate authorities are removed from the trust store is progressive: they will announce a date after which new certificates from a given CA will no longer be trusted. I think this can be made even more progressive, by limiting the validity period of new certificates that will be trusted. DigiCert will have little recourse other than to let their customers know, and/or start providing certificates issued by another CA that follows the web PKI procedures and remains trusted. (They can still do that, of course, it's just that their own certificates issued by them directly won't be trusted anymore.)

On the flip side, for user impact, it will play out like this: Some bank or other important entity could possibly, for whatever reason, continue using a (presumably expired? unless DigiCert continues issuing anyways; note that most likely, they will not.) DigiCert certificate after the cut off date, which will lead to users receiving errors. Some of them will have HSTS setup, which will lead to an emergency situation where they have to issue a new certificate ASAP, as it will basically halt their business until they do. For places where there is no HSTS, users may be instructed to simply bypass the certificate warning temporarily, and support lines will be absolutely swamped until they actually fix the problem.

The WordPress situation is quite different. You don't have to use WordPress. Users don't even know what the Web PKI is to find an alternative to it, not that there is one or will be one.


Sure some users would keep the CA active out of fear, while everyone paying digicert would also begin to move out of fear.. What portion of the strange sort of sites that couldn't figure out how to get off but could figure out how to renew would bother with paying for a different error message?

The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.

...unless they notice the breakage, and you tell them to do so to stop it.


Ignoring the policy reasons that this won't happen, not all devices can be downgraded, so by then it's too late anyway. Best you can do is bypass a non-HSTS certificate warning.

That all assumes DigiCert continues issuing certificates they know won't be trusted. I assume most likely what will actually happen is they have to start selling certificates from another CA, but failing that they're probably just going to stop issuing certificates before the cutoff.


> mainly Google, Microsoft and Mozilla

They don't have free rein to do whatever they want. If they chose to remove them, they are going to have to be ready to defend themselves in court.


Why exactly do you think certificate authorities have a legal right to appear in trust stores?

This case literally involves the courts getting involved in what should appear in trust stores. DigiCert was ordered by the courts to not revoke certificates via a TRO.

If the browsers removed DigiCert, DigiCert could certainly sue based on the harm to their business. They could argue that the browser vendors were inconsistent with the application of their rules. They could argue that the browser vendors were unfairly competing by kicking them out.

Not saying they would win all these, but they would certainly fight it - the alternative would be the end of DigiCert, they aren't going to go down without a fight in whatever venue they can find.


> DigiCert was ordered by the courts to not revoke certificates via a TRO.

... and DigiCert did not try to appeal the TRO.


Is that really the claim, though? If someone chooses to punish DigiCert for compliance with the TRO then DigiCert might have grounds to file for declarative relief from the court, as the court could very well decide to help protect DigiCert from external consequences of their demand.

Perhaps, but only in the case that the delayed revocations were scoped to those certificates covered by the TRO, and not over 1000x more from subscribers who had nothing to do with the company who filed the TRO.

Well personally, I think that'd be a terrible position for them to take. Honoring the TRO is, as far as I can tell, not the issue that is being raised.

Both Google and Microsoft tend to act like they have enough money and lawyers to defend whatever behaviour they feel like engaging in.

Those companies are always ready to defend themselves in court, and this is a country with extremely strong free speech rights.

What country?

> All it will do is increase the lack of trust in centralised PKI in general.

“In general” is a broad statement, given that the overwhelming majority of people using the Web PKI have no idea that they’re doing so. DigiCert is not a legible part of the value chain for the average users; they won’t notice that the sites they use switch CAs. This strong disfavoritism towards vendors is arguably one of the Web PKI’s greatest strengths.


Many systems do not fetch updates from the Mozilla root store, but from their (possibly Debian-derived) stable distribution of it. Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage. When that happens, I can blindly trust they did the thing needed to keep the unaffected parts of that weird system working as intended.. and then still head to bugzilla and read about the background - to laugh at whoever triggered the mess.

> Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage.

Note that this is not a "both" but a "any of them". One can disagree with the other on this and still cause wide breakage.


If DigiCert were to lose browser trust (at this point, that's still a big if), it would happen the same way it happened with prior CAs, some of which were pretty big themselves (Symantec): all certificates issued after some date would not be trusted, yes. But all existing certificates would remain valid.

This gives certificate owners ample time to look for a different issuer and no certificate buyer would deliberately purchase a certificate from an issuer when they know some percentage of users will not trust that cert.

So for the end users, everything will keep working: the existing digicert certs stay valid and newly refreshed certs will be signed by a different authority. There is no need to turn off automatic updates over this.

Between Entrust and Symantec, we've already seen this happen to large well-known CAs and everything remained fine (not for the offenders, but, hey, that's the system working as intended)


I've had a pretty similar outlook and still kind of do, but I think I do understand the hype a little bit: I've found that Claude and Gemini 2 Pro (experimental) sometimes are able to do things that I genuinely don't expect them to be able to do. Of course, that was the case before to a lesser extent already, and I already know that that alone doesn't translate to useful necessarily.

So, I have been trying Gemini 2 Pro, mainly because I have free access to it for now, and I think it strikes a bit above being interesting and into the territory of being useful. It has the same failure mode issues that LLMs have always had, but honestly it has managed to generate code and answer questions that Google definitely was not helping with. When not dealing with hallucinations/knowledge gaps, the resulting code was shockingly decent, and it could generate hundreds of lines of code without an obvious error or bug at times, depending on what you asked. The main issues were occasionally missing an important detail or overly complicating some aspect. I found the quality of unit tests generated to be sub par, as it often made unit tests that strongly overlapped with each other and didn't necessarily add value (and rarely worked out-of-the-box anyways, come to think of it.)

When trying to use it for real-world tasks where I actually don't know the answers, I've had mixed results. On a couple occasions it helped me get to the right place when Google searches were going absolutely nowhere, so the value proposition is clearly somewhere. It was good at generating decent mundane code, bash scripts, CMake code, Bazel, etc. which to me looked decently written, though I am not confident enough to actually use its output yet. Once it suggested a non-existent linker flag to solve an issue, but surprisingly it actually did inadvertently suggest a solution to my problem that actually did work at the same time (it's a weird rabbit hole, but compiling with -D_GNU_SOURCE fixed an obscure linker error with a very old and non-standard build environment, helping me get my DeaDBeeF plugin building with their upstream apbuild-based system.)

But unfortunately, hallucination remains an issue, and the current workflow (even with Cursor) leaves a lot to be desired. I'd like to see systems that can dynamically grab context and use web searches, try compiling or running tests, and maybe even have other LLMs "review" the work and try to get to a better state. I'm sure all of that exists, but I'm not really a huge LLM person so I haven't kept up with it. Personally, with the state frontier models are in, though, I'd like to try this sort of system if it does exist. I'd just like to see what the state of the art is capable of.

Even that aside, though, I can see this being useful especially since Google Search is increasingly unusable.

I do worry, though. If these technologies get better, it's probably going to make a lot of engineers struggle to develop deep problem-solving skills, since you will need them a lot less to get started. Learning to RTFM, dig into code and generally do research is valuable stuff. Having a bot you can use as an infinite lazyweb may not be the greatest thing.


Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: