Just one thought because I spent some time thinking about this type of problem before deciding to do it a different way in my own work:
If you use an orthographic projection and just actually draw 3-D boxes, this problem becomes trivial, including the intertwined boxes problem, because the GPU will take care of it at a per-pixel level with the Z-buffer.
Of course this also means you're suddenly working in 3-D, which complicates other things. But you can still use 2-D textures by picking the right texture coordinates for the vertices on the box.
These days it usually makes a lot of sense to be doing 2-D stuff using a 3-D engine/library anyway, if only for access to hardware accelerated post-processing and hardware accelerated layering/compositing, in which case using actual boxes on a tilted camera and using "no projection" for your HUD/UI works pretty well.
"Use a 3D engine with Z-buffering" does seem like the right solution to these problems today. However, even if you don't, taking inspiration from one seems like another way to solve this class of problems, including the case of the three mutually overlapping boxes in the last example: independently decide the "front" box to draw for each screen coordinate. For each box, record which boxes appear "in front" of it, non-recursively. (So, A > B, B > C, C > A.) When rendering, only consider boxes that would render at the given screen coordinate in the absence of occlusion. (So, in the given example of mutual overlap, any given screen coordinate only needs to consider two boxes.) Then, render the front box at that screen coordinate.
If you have any translucency (water, plants/trees, objects), then render the frontmost opaque box followed by other translucent boxes in front of it, in back-to-front order.
(That approach would completely break in the presence of concave shapes, which could allow a single box to appear both in front of and behind another at the same screen coordinate.)
Depending on your shapes (e.g. boxes), you wouldn't need to do that for every screen coordinate; you could make the same decision for entire regions, bounded by the screen-projected coordinates of the vertexes.
However, once you reached any significant degree of complexity, you'd probably still be better off implementing a full 3D engine in software if you didn't have 3D hardware. Z-buffering isn't particularly complex to implement. (More complexity comes in when trying to pre-cull some geometry without rendering it at all, to avoid rendering over the same screen pixel many times.)
Thanks for expanding on my musings! Two thoughts to add:
1. Pretty much every 3-D graphics engine worth using supports importing and batch-rendering meshes that are actually made up of multiple sub-meshes. So you can fix the circular occlusion problem and the concave object problem with pretty much no performance loss by just subdividing those offending shapes during the art phase. This simplifies rendering because the engine doesn't need to do any dynamic mesh subdivision.
2. Pretty much every 3-D graphics engine worth using supports depth sorting for translucency, so you effectively get that for free too.
I should have clarified that "3-D" by itself doesn't necessarily get rid of your problem. However, there's very little reason to build your own graphics engine anyway, and modern 3-D graphics engines abstract away the occlusion problem for free.
I suppose I had a third point, a response to your last paragraph: it seems hardly worth it to basically implement your own software rasterizer with a RAM based Z-buffer if getting the dedicated hardware to do it is just a matter of pulling in a 3-D graphics engine. And doing it on dedicated hardware is going to be much faster. Granted you have to learn how to use the 3-D graphics engine and you'll have to do a bunch of work through shaders, but if you're doing graphics stuff, this is staple, bread and butter knowledge. Kind of like how web frontend developers would be expected to be able to write CSS and to be fluent in at least one JS framework.
> 1. Pretty much every 3-D graphics engine worth using supports importing and batch-rendering meshes that are actually made up of multiple sub-meshes. So you can fix the circular occlusion problem and the concave object problem with pretty much no performance loss by just subdividing those offending shapes during the art phase. This simplifies rendering because the engine doesn't need to do any dynamic mesh subdivision.
True, but this problem also just goes away if you represent objects as meshes of triangles, because individual triangles aren't concave.
> it seems hardly worth it to basically implement your own software rasterizer with a RAM based Z-buffer if getting the dedicated hardware to do it is just a matter of pulling in a 3-D graphics engine.
Agreed completely, if you have that hardware available. I could imagine constrained-computing scenarios (either as a challenge, e.g. "standalone minecraft clone in 4k of code", or a deeply embedded environment) where you wouldn't have a 3D engine available and you might want to implement a simplified rendering engine for a highly constrained rendering problem.
But yes, for almost every scenario, you want to just use the GPU.
I'm going into pedantic territory, but since we're on the topic of drawing order for translucent meshes... might as well.
You don't actually avoid the problem entirely when you're using just triangles because it's still possible to construct a "A overlaps B overlaps C overlaps A" cycle of occlusion using three triangles. It's more or less the same corner case as a cyclic occlusion problem with convex pole-shaped meshes.
For this reason it seems like graphics engines tend to just batch-render entire sub-meshes at a time because rendering entire meshes presents little additional complexity over rendering individual polygons (and is more performant). It's a leaky abstraction but the leaks are pretty rare.
A z-buffer solution will not help as soon as you involve transparency. Which I'd assume is the case in 99℅ of applications. Unless your application/game is all boxes :)
No, transparency is actually quite rare in games, for pretty much this exact reason, and outside of games is rare to have more than one or two layers of transparency.
The typical method is to draw all opaque objects first with zbuffering enabled. Then sort all the remaining trasnparent objects by depth and draw them from back to front
I didn't discuss transparency because transparency handling comes more or less free with a graphics engine.
That's besides the point that partial transparency is actually intentionally used very sparingly in games because it's computationally expensive.
In the same sense that you almost always start a web app front-end with some JS framework, you almost always start a 3-D game with some graphics engine. Graphics engines generally take care of the depth sorting for alpha blending for you.
If you're building your own graphics engine, then I'd mostly just question whether you really need to. Sure you might make something leaner, but the benefits will be marginal at best.
In other words, yes, you're right, transparency complicates things, but it's a well-understood problem and the solution is more or less free with the package when you pick up a graphics engine.
Of course, when you really dig into the bowels of the problem, you'll notice that most graphics engines don't gracefully handle the circular overlap problem. In practice, it's more efficient and lower cost to just cut the object up at asset creation time than to dynamically split things at render time. That means you're just avoiding the problem, but it's a rare enough corner case that it's just cheaper to work around the corner case.
Boolean transparency can be performed in any order, so you can use non-boxes straightforwardly. It's only semi-transparent textures or alpha-blended objects that need more help if they're to be blended correctly relative to each other (and most situations will involve particle & sizzle effects, which don't need true correctness anyway).
Just one thought because I spent some time thinking about this type of problem before deciding to do it a different way in my own work:
If you use an orthographic projection and just actually draw 3-D boxes, this problem becomes trivial, including the intertwined boxes problem, because the GPU will take care of it at a per-pixel level with the Z-buffer.
Of course this also means you're suddenly working in 3-D, which complicates other things. But you can still use 2-D textures by picking the right texture coordinates for the vertices on the box.
These days it usually makes a lot of sense to be doing 2-D stuff using a 3-D engine/library anyway, if only for access to hardware accelerated post-processing and hardware accelerated layering/compositing, in which case using actual boxes on a tilted camera and using "no projection" for your HUD/UI works pretty well.