I love how Carmack and Id since its inception has not only defined the cutting edge of game technology, but also repeatedly utterly disrupted industry business models (pioneering shareware distribution, the 'free demo' model, engine licensing), all the while making an absolute killing doing so. Most inspirational hackers out there, imho.
Different Apogee. Presumably you are thinking of Apogee Electronics, he is talking about Apogee Games, which more or less* became 3D Realms (of Duke Nukem infamy), and who back in the day published a lot of the classic shareware games from other developers as well, including id's early stuff like Commander Keen and Wolf 3D.
* "more or less" because the business structure is more complicated than that and there was still an Apogee Software alongside 3D Realms, but for all practical purposes, what was Apogee Games became 3D Realms and then died due to the Duke Nukem Forever project.
Whilst I have been so-so on id games since Quake 3 (Actually Quake 3 was excellent but it was just so overshadowed by the jaw-droppingly awesome Unreal Tournament) I have huge amounts of respect for Carmack as a programmer and also for the way id generally do business.
I remember scanning through the source code for the original quake some years ago (not really reading in depth) but I couldn't help but assume my download was borked and I was somehow missing 90% of it , the core of the engine is tiny, I get the same reaction when I look at the Linux kernel.
I've written business systems that are orders or magnitude larger in terms of code than probably all the id games put together but have nowhere near the awesomeness and variety in terms of functionality.
I wonder whether this is just how he programs from the get go or whether he writes something much larger and clunky in the beginning but is just ruthlessly disciplined at rewriting things to make them smaller and more efficient.
I think Quake's code represents a design methodology you almost never see in the wild. Essentially, Quake's code operates on the same principles that Apple's physical interface designs do (or perhaps singular programming language designs like Scheme or C).
Carmack, as dictator of id code (and thus id design) had the ability to say "we are not adding the following feature to our game because I don't like the compromises it will force in our technology and, thus, in other features we want to add down the road." It's basically as simple as that.
In almost all other organizations, someone holding the pot of money will trump the programming overlord at that point and say, "I don't care, XML is the buzzword this second, and all objects in the game need to be serialized to XML as web services - that's what the market is clamoring for right now" (or whatever the buzzword du jour is). And over time those political technical decisions add up to so much cruft that adding any new thing becomes a technical, jenga-style nightmare.
Not so with id's engines. I actually worked on beta versions of both Quake 2 and Doom 3 code bases while those games were in development, and one striking thing about getting updates from Carmack was his ruthless willingness to eliminate working, finished features / code if he didn't think the complexity or dependencies they introduced were worth whatever feature they added in the overall technical context.
Quake is what you get when a genius programmer who has mastery of his knowledge domains leads. Institutionally, there aren't too many other circumstances where similarly talented and sure-footed and, well, fundamentally, _right_ programmers get to specify requirements like that. I think the comparison to the Linux kernal is apt.
Interesting, I think you are right. When I first looked at the Linux source I assumed it would also be a monolithic jenga style mess since it was developed by so many different people with different motivations but I guess it must be a testament to Torvald's management/leadership skills.
I suppose I am probably jaded because I work developing bespoke corporate software with relatively few users compared to Quake. Therefor each user has huge individual sway over the functionality/design even if they don't really get the nuaces of building usable software. For example they will request additions of buttons in nonsensical places because it helps their personal workflow but then somebody else will come along and use it then complain that there's too many buttons and it's too complicated "can we please make this more like the iPad?"..
I wonder if there are any pieces of bespoke corporate software that aren't in fact giant piles of bloated dogshit?
All the books on this subject recommend doing thorough requirements analysis but that doesn't really work when you have to deal with people who can't really articulate what they want and actually have no idea what might suddenly become important in the future. Plus you have to deal with corporate politics of people with different roles and places in the heirachy who want and need diametrically opposed things, people will also often have good ideas but they won't tell you them until it's too late because they are worried about contradicting their boss.
It would certainly be refreshing to work with a Carmack but I doubt that I would have the focus or brainpower required to keep up!
This is very interesting. I (yesterday, coincidentally) finished reading Masters of Doom, the book about id by David Kushner. While generally a good read, it had a very fanboyish tone towards both the Johns. It really portrayed Carmack as absolutely superhuman when it came to coding. Given the tone of the rest of the book I'd assumed he was exaggerating — it sounds like he wasn't after all, about that at least.
I don't know C so I can't, let alone appreciate, the source he's released to judge for myself.
Most companies probably can, they just don't care enough to do so, seeing it as money spent on a last-gen engine that results in no new revenue streams.
I think id does it mostly because Carmack personally pushes for it. However it arguably makes some sense for them, relative to other game companies, because they've been particularly identified with the "genius hacks to make stuff work" aura, and every source release produces a new wave of discoveries/press about crazy things id did in the guts of one of their engines, which solidifies that reputation. Plus it's probably good for Carmack getting historical credit to put the techniques out in the open once they're no longer most-recent-gen.
Carmack also stated that these open source releases help them in the long run. E.g. they were able to port Wolfenstein 3D and Doom (the old one) to iOS much quicker by basing it on modern versions of those engines, created by the community.
Yes, and it also helps them keep the games alive. If you have the game data (CD, DVD, Steam, whatever), you can play every id FPS game up to Doom 3 in almost any platform you want, today. And you will probably be able to do it 20 years from now. I think it's fantastic.
Yeah, there's a lot of goodwill generated by them, too. I wouldn't be surprised if this also helps them with recruitment. I mean, this helps create more talented game devs by giving them stuff to hack on.
Bear in mind that this is inverse square root (a misnomer; it's really 1/sqrt, or the reciprocal), not just the square root. Square roots are decently fast as provided by the processor, but not when combined with the division needed for the reciprocal; combining the two is why FastInvSqrt is so fast.
No, it's not faster now and it's no longer really used probably (don't know if smartphones, tablets, various consoles, etc. offer access to any equivalent of rsqrtps), but it was brilliant at its time (well, still is, it's just not very useful).
Remember, Quake was released in 1996, SSE came out in 1999.
The SSE version is 16x faster than the x87 FPU version and almost 4x faster than the marvelous method. Further, the marvelous method is a loop and is much more resource-hungry than an instruction that goes away for 5-10 cycles (reciprocal throughput is now 1 cycle) but leaves you with most of your execution resources to do other useful work.
Vectorizing the marvelous method is very likely to be painful, as it has the treatment of a value as both a float and an int - while the xmm registers are dual-purpose operating on one successively as a int and a float causes some extra latency. The use of bit shifts and float operations require these cross-domain transfers...
Scorched earth. If id release an engine at a certain tech level, they put serious competition in the way of other would-be licensors of engines at or about that tech level. That stops smaller software houses from breaking through that tech level to compete at the higher level.
I don't know if it's a conscious policy or not, but it's interesting nonetheless.
That's an interesting point. I wouldn't immediately think of game developers as using someone else's framework to build an application with, but a lot of them really are, just like those of us building business applications with someone else's framework.
I wanna see how they did their shadow volumes. As far as I know, they did it without geometry shaders, which would mean that they had to do the shadow volume extrusion on the CPU. Or maybe there's some cleverness involved.
They probably used the CPU -- I remember reading somewhere that Carmack also wanted to implement a beam-tree (3D BSP) for faster culling of shadow surfaces, but I am not sure. I did some experiments with Shadows myself 10 years ago, on that era of CPUs and GPUs and the bottleneck for me was drawing the volumes in the stencil buffer.
I worked on Quake4 as a programmer for a bit over a year, though little of my work ended up in the final game.
The Doom3 engine is mostly C++. If I recall correctly, the renderer still looked more like a blend of C with some C++ slowly drifting in (which makes sense, given how good of a C programmer Carmack is), but the rest of the game code is very heavily C++, for some definition of C++, along with a scripting layer.
Having worked extensively with the Quake 2 engine previously, I really didn't like the Doom 3 engine, myself. It was radically less... pragmatic. Earlier id engines, to me, were masterpieces of concision. They weren't amazing feats of black box abstraction by any measure, and they implicitly assumed you needed a lot of domain knowledge to understand what they were doing, but they did exactly what they needed to do with a minimum of fuss. Features were included because they were used, generally with the smallest and simplest code and tool footprint possible.
Not so with Doom 3. Huge amount of code appears to have been written because someone prolific thought it would be lots of fun to write it, many features end up going unused or are in the way, and ultimately (because it's more typically software engineered in style), every feature requires about 30 times as much code to be written properly and has several extra unused layers of abstractions crufting things up. The game code (in C++) in particular bears an unfortunate resemblance to Windows MFC-style C++, with a giant 12 layer deep inheritance graph with everything inheriting from a few enormous base objects.
All of that to my eyes, anyway. But then, I really cut my teeth on Quake 1 and 2, and found that their concision and brevity really helped them stay out of the way when adding new, innovative features.
The funny thing about the code for Doom 3 is that, looking through it all, you'd get the impression that Doom 3 was a radically more complicated game that what you ultimately experience when you play the game.
The best way I could put it is this: Quake 1 and 2 and 3 were well made games that ended up being enormously fruitful for white box reuse. You couldn't do much with them without understanding fair bits of their internals, but they didn't actively resist being understood. Doom3 was built to be used in a more black box fashion (like the Unreal Engine), with all that entails from an architecture perspective. I'm sure others might disagree with me from a code ideology perspective, but it is hard to ignore how fruitful the Quake engines were for licensing, and how much Doom 3 sank like a stone.
The code bases make really fascinating studies in contrast about the consequences of different software engineering values, at the very least.
TL; DR: Quake 1 - 3 remind me of linux code, Doom 3 reminds me of MFC.
I have read most of the Q3 code and really like it. And as you said, not much (or any) overhead.
What you write about D3 (i.e. idTech4) makes me a bit sad. How comes it? Less influence by Carmack? (Because he still seems to be the pragmatic guy to me.) Too often, when C++ is involved, I see much too complicated code. Whereby I still have the meaning that you can really end up in having simpler code with C++. (Well but that goes into the usual C vs. C++ ranting discussion.)
Do you have any knowledge about idTech5 (or maybe even idTech6) which you can share?
I think by Doom3 id had reached a point where they needed a team of programmers, rather than Carmack + programmers orbiting Carmack. And so Doom3 represents their transition to dealing with all the software engineering / communication issues all of us on teams always have to deal with.
I don't have any knowledge of idTech5, although I would guess it goes further down the lineage of Doom3. id still has to find solutions to the problem of programmers working on teams, after all - no amount of Carmack genius is going to magically fix that all too banal (and inevitable) organizational challenge.
I had my fill of the industry by the mid-2000s and have been working on indie/experimental/violent educational games since, so that's roughly where my big iron engine knowledge fades...
Actually, that's not true. WebGL on Windows is implemented using D3D (ANGLE converts OGL ES calls to D3D9) in most cases, and WINE has done D3D on top of OGL for years. They're very different, but translating between the two really isn't as big a deal as you'd think. (Just keep in mind that XNA isn't D3D -- it may backend to it, but it's a very different API.)
It's unlikely that you'll be able to play TDM without a copy of Doom 3 for the foreseeable future. TDM reuses a number of Doom 3 assets and those are not included with the open source release of the engine.
(If you don't know, the Dark Mod is a Doom3 total conversion inspired by the Thief series, one of my all-time favorite games).
Me too! I just heard about the dark mod a while back and tried to get it working. I didn't want to go buy Doom3, because I don't like the game and I couldn't be sure whether or not TDM would work. So I tried with a pirated version and it didn't, so I'm stuck waiting for the open source release of D3 engine (and the subsequent "porting" of TDM).
(For some reason, PC games for me have a terrible track record of not working. I often try a pirated version of the game and if it works and I like it, I might buy it. If the pirated version doesn't work, I won't bother because experience tells me the legal one isn't probably better and I hate begging for refunds from Steam support).
I believe one of the major issues will be Doom 3's use of the "zfail" algorithm, which Creative claim a patent on. Generally, Creative license this patent to game developers in exchange for the game supporting OpenAL (a net win for Creative as it means their sound cards can make the game run faster and more people will buy them), but I'm not sure what the terms on a source release would be. Take a look at  for more info.
Oh this is just silly. Thankfully the patented algorithm is ridiculously simple, so someone can write an open source patch to use the better stencil algorithm. No doubt whatever derivatives will spring from the D3 source will use the single pass method.
While being at it, it shouldn't be too hard to use geometry shaders for the shadow volume extraction in the D3 engine. Hmm, maybe I should do it :)
I suspect this will be released, I doubt they'd get this far in announcing it to the public unless the remaining stuff was crossing Ts and dotting Is, but it is notable that there's a pretty big difference between this release and previous releases of their engine source.
Previously id was an independently owned company. Now they are owned by ZeniMax, a larger entity.