I've exchanged a couple of emails with the author a while ago. Reverse-engineering and decompiling DOS software running in real mode and especially video games is a particularly tricky endeavor, a notch above other similar projects on other platforms.
The combination of janky PC hardware, paper-thin MS-DOS operating system and prehistoric development tooling makes for an environment where anything goes. Real mode segmented memory and large memory models utterly confuse Ghidra's analyzers and decompiler, most common modern reverse-engineering tools either do not support 16-bit MS-DOS executables or have third-rate support.
It is more art than science in a way that I don't think reverse-engineering later software can match.
It's a little awful to think about just how much collective human effort was wasted by segmented memory. Everyone knew that 32-bit flat memory was the future; it just took a long time for the PC platform to get there.
> Why not just the full 64k. The sliding window trick was of such limited use compared to the headaches it caused.
As I pointed out in my other comment, the full 64KB would have made it harder to support child processes under DOS.
Suppose I have 300KB of free conventional memory, and run an 8KB program (something like a menu program, for example.) If it starts a child process, that child process has about 292KB of free conventional memory to use. With your idea, it could only have 236KB, because even though the parent process only needs 8KB of memory, it would consume a whole 64KB.
Yeah, but when you are shipping computers with 256kb like the original PC/XT those extra two bytes per pointer start to look like a major luxury. There was a heroic amount of effort in those days to make due with as little memory as possible because memory was so expensive that nobody had more than was absolutely necessary.
Yes, but if your program required no more than 64KB of data, then you could use near pointers everywhere, halving the pointer size from 4 bytes to 2 bytes.
Also, even if the program as a whole required more than 64KB of data, if you knew you required no more than 64KB of data for objects of type X, then you could use 2 byte pointers for all type X objects, with a fixed data segment. X here might, for example, be strings.
When you only have 256KB of RAM to begin with, the odds that all data overall, or at least all X object data for some X, will fit in 64KB is quite high. And if you have a lot of pointers, halving their size makes a big difference when you have so little memory.
But segmented memory isn’t a requirement, that technique works just as well with linear address space. You use a single 32-bit base pointer and then store 16-bit offsets for your data. We used that all the time on 68K and other architectures.
The difference is that 68K is designed as fundamentally a 32-bit architecture. (Even though the original implementation was physically 16-bit.)
Whereas 8086 is a 16-bit architecture with an extended address space.
The use of segmentation to enable a 16-bit architecture to address more than 64K was not original to the 8086, many 16-bit minicomputers (e.g. the PDP-11) used the same basic idea, although the specific implementation Intel chose was rather unique
Part of why the 8086 was 16-bit not 32-bit, was to make it easier to port software to it from the 8080, which was an 8-bit architecture with 16-bit addressing. It also was likely one of the reasons why the 8086/8088 was cheaper than the 68K, which is part of why IBM chose it over the 68K for the IBM PC
Motorola 68000 also had second sourcing options, just like 8088 - but 8086 was released a year earlier and built up on well-known successful design of 8080, even if they weren't exactly compatible. It built confidence about Intel's ability to deliver, even when 8088 was released the same year as 68k, as it was just bus-narrowed cheaper version, not an entire architecture.
Which reminds me - the bus-narrowed version was important, because they then could use cheap and off the shelf 8-bit components for the rest of the chipset.
Near pointers could exist in non-overlapping segments couldn't they?
With 254KB you'd use 4 segments tops. There was some valid reason to have them overlap, but it quickly became obsolete.
> There was some valid reason to have them overlap, but it quickly became obsolete.
Under DOS, the (primary/initial) code segment acts as the PID. So overlapping segments enabled less memory fragmentation and more memory for child processes. If your program used less than 64 KB, then the part of 64KB it didn't use could be made available to child processes to use. Important in the early days, where some programs were structured as a main executable (which displayed the main menu), and then separate child executables for each menu item–the main menu executable would stay in memory while the child executable ran. (Overlays were more efficient, but also more complex for the developer.)
People call DOS a "single-tasking" operating system, but that wasn't entirely true – processes could spawn child processes, it is just the main thread of the parent process was suspended while the child process ran, unless the child process turned into a TSR. Also, child processes could use interrupts–or even just CALL FAR (if you could get the target address somehow)–to make upcalls to APIs exported by their ancestors, which is a way ancestor processes could be reinvoked even while suspended. COMMAND.COM's INT 0x2E API is an example (something I wish Unix shells had–a child process can modify the parent shell's environment, by calling an API to make it run a built-in command).
Not standard terminology, but personally I think it should be called "semi-multitasking". CP/M-80, by contrast, was a truly single-tasking operating system – only one process could fit in memory at a time, and instead of TSRs which were just normal processes which had returned to their parent without being removed from memory, you had RSXs which were a radically different type of thing from a normal program. (Normal programs are allocated at the bottom of memory, RSXs from the top down.)
(Did CP/M-86, CP/M-68K, CP/M-8000, get closer to MS-DOS in this regard than CP/M-80 did? Obviously MP/M went further than mainstream MS-DOS ever did.)
This ability to load child processes and TSRs is mostly owed to the CPU architecture, not to the design of MS-DOS - CP/M-86 had it too.
With a single flat address space, programs need to be either position independent (not possible at all on 8080), load at a fixed address, or include some type of relocation info. The DDT debugger in CP/M started with a "loader" which moved the rest of the code to the top of memory, aligned to a 256 byte "page" boundary. There was a bitmap of which bytes of the code referred to a page number (i.e. the high byte of an address), so those could be fixed up. With MP/M, every program had to be in such a format, minus the loader, which became part of the operating system.
x86 memory segmentation is sort of a middle ground between that, and virtual address spaces for each process. No security of course, but at least some degree of isolation in case of bugs, and the convenience of having fixed addresses within each segment.
> This ability to load child processes and TSRs is mostly owed to the CPU architecture, not to the design of MS-DOS - CP/M-86 had it too.
Did CP/M-86 support child processes though?
Also, did CP/M-86 support TSRs? I thought it supported RSXs which just like CP/M-80 were in a different format and manipulated by different APIs than normal programs, whereas under DOS any program can turn itself into a TSR at any time, you can't tell when you start it whether it is going to turn itself into one (unless you know what its code does, of course)
Looking at https://www.seasip.info/Cpm/bdos.html#144 I see BDOS function 144 (P_CREATE), for creating subprocesses, was implemented on MP/M and Concurrent CP/M – but I don't see any mention of it being implemented on (non-Concurrent) CP/M-86.
CP/M-86 1.1 (but not 1.0) supported BDOS function 47 (P_CHAIN), for launching a new program – but like exec() on Unix, or the BASIC CHAIN statement, it terminated its caller before launching the new program.
That said, PC-DOS/MS-DOS 1.x didn't really support subprocesses either. There was no documented API for creating them. The program loader was actually in COMMAND.COM, and while it called the DOS kernel to allocate memory for a new process (the undocumented API INT 0x21,0x26), COMMAND.COM was responsible for actually loading the .COM or .EXE from disk into the process' memory (including relocating .ees), so creating a child process required duplicating a large part of COMMAND.COM's code. In DOS 2.x, the image loader was moved into the DOS kernel, and a public API to spawn a child process (INT 0x21,0x4B) was added. COMMAND.COM also implemented its own API to start a child process (which would be a child of itself not the caller), INT 0x2E, but I don't believe that was there in DOS 1.x either.
> With MP/M, every program had to be in such a format, minus the loader, which became part of the operating system.
You are talking here about the CMD executable format?
I'm fairly sure that the CP/M-86 documentation says that a chained program returns to its caller, not the CCP. And BDOS function 0 with DL=1 will exit while keeping the program resident in memory.
However, it also describes (PDF page 60) function 59 (PROGRAM LOAD), which can be used to load a child process. Like DOS 1.x, spawning a child process is somewhat of a manual process, you apparently have to use function 59 to load the CMD file, functions 51 and 52 to set the default DMA base and offset (which CCP normally does for you), and then manually do a far jump to the start of the program. At least, unlike DOS 1.x, the image loader is in the BDOS kernel not CCP.
But, this is not unique to 8086 CPU architecture, this function is also implemented by CP/M-68K–which (I believe) relocates all executables as it loads them – see PDF page 106 of http://www.bitsavers.org/pdf/digitalResearch/cpm-68k/CPM-68K... - (it is also implemented by CP/M-80 3.x, but there it can only load RSXs not standard executables.) And I believe CP/M-8000 implements it too
> And BDOS function 0 with DL=1 will exit while keeping the program resident in memory.
Yes, you are right, I wasn't aware of that. Now I read more about this, my impression is RSXs are CP/M-80 only, and CP/M-86, CP/M-68K and CP/M-8000 use TSRs instead.
Yeah. I’m not going to defend x86 segment registers and real-mode DOS too vigorously. (I programmed in x86 assembler for many years.) But I’m pretty sure the right approach wasn’t to say Screw it. Let’s wait until a 32-bit flat address space processor and OS come along at the consumer and individual business user level.
IDA Pro disassembler fully supports 16Bit DOS executables (they only removed the support from the recent free versions - thats why most using IDA Pro Free 5.0) but Hexrays the decompiler does sadly not support 16bit code (intentionaly) but i think the Hexrays guys could do wonders here (in comparison what Ghidra is able to do)
My beloved was a little bit newer DOS game F22 Lightning II. The best memories I have from it are when I was cycling through targets, and there was Air Force One which we were escorting. And just because I had armed missiles, AWACS detected radio pulses on the Air Force One, and every other aircraft in the air shot a missile on my machine. This caused first of all my computer to lag, then "incoming missile" from the speaker repeated hundred times, and after that I saw an animation of all the other missiles flying through the remains of my plane. Wonderful experience
When I was about 8 years old I can remember walking into the office at home this game was on the screen. One of the fighters was flying towards an airstrip and I thought I'd helpfully land it, I crashed. Turns out dad had it on autopilot to return to home after some mission which he had to repeat thanks to me :)
I've thought to myself that if I want a cool project that would motivate me to re-learn C just for the sake of making an NES game (ala 8-bit workshop - https://8bitworkshop.com/), I would really like to learn how to apply the pseudo-3d or 2.5d methods that were used by Microprose in F-19 Stealth fighter.
I don't know the first thing about 3D programming. I tried to follow a book on writing a ray-tracer from scratch just to see if I could pick up the principles and do it myself, but I got frustrated by the ambiguity of the approach. Like I was trying to use Ruby with Rspec, but I'm not even sure if I'd run into a performance issue or not be implementing the interfaces correctly.
I just want to know what method was used. Maybe 3D projection?
Kind of like how Myst was able to deliver ultra-high resolution 3D images in a video game in ways that had never been done before, even though it was really just a point-and-click slideshow with embedded videos and scripted interactions (Hypercard)... Hellcat Ace provided an experience far ahead of its time.
> I would really like to learn how to apply the pseudo-3d or 2.5d methods that were used by Microprose in F-19 Stealth fighter.
F-15 is not pseudo-3d at all.
Flight simulators were early innovators in proper polygon based graphics. Partly because there were so few polygons they could get away with the painters algorithm. Partly because the instrument panel could cover a large chunk of the screen. But mostly because players of flight sims were willing to put up with very low frame rates, even 1 frame-per-second would be considered playable for certain types of flights.
Anyway, the tricks used for the 8 bit computers and consoles were nothing more than distilling down the graphical complexity until all they really had to draw was a tilting horizon line and then use a few sprites for everything else.
The one notable pseudo-3d technique I'll point out is the voxel based terrain used by "Comanche: Maximum Overkill". That's well documented.
On the NES (or other 6502 machines), use assembly. That CPU just isn't a good target for C - pointers are awkward, you can't really do stack frames, and the addressing modes don't even support 'structs' very well. Fun to program assembly on, though. For your purpose the NES makes it double hard because it hardly has any ram (2 KB, and typically an additional 8 KB inside each cartridge except in the oldest games), and a character-mode graphics architecture rather than a framebuffer. If you do really want to do an 8-bit flight sim, the Atari 400/800 machines are an okay target. The Atari ST would be a better target.
Speaking of those machines, I'd love to see someone deeply reverse engineer the pseudo-voxel/fractal landscape engine that Lucasfilm Games invented for Rescue on Fractalus, Koronis Rift, etc. It still seems completely magical that they could pull that off on a machine with those specs.
Starting from zero, you might start at a high level and work your way down. Do a simple polygon flight engine using OpenGL and your favorite high level language of choice. Then write your own polygon rasterizer that you can overlay versus the OpenGL rendering as a reference. Then maybe rework it in pseudo 8-bit code (C constrained to only unsigned char variables, or similar), which should translate directly to assembly language on the target of choice. On a real 8-bit machine, unless you want low single digit frame rates, you probably have to pull a lot of dirty tricks. Maybe a 16-bit platform would be a better choice. This is speculative - I learned 3D (to a novice degree..) on a 486 PC under 16-bit DOS using C and assembly for inner loops (bitblts, texture mapping).
There's something about the aesthetic of late 80s PC flight sims (F-19 Stealth Fighter, LHX Attack Chopper, etc.) and their flat shaded polygon graphics that feels in vogue right now. Check out Thunder Helix on Steam.
Anyway there's definitely a magical feeling writing graphics code when you get your first feeling of realistic movement and rotation in 3D space working. Hopefully the ubiquity of ultra-realistic doesn't diminish that sensation too much.
If you're trying to do it all in software, you can get pretty far with a function to draw a solid colored triangle, a function to rotate 3d points using sin and cos, and some loops. Then the other pieces like lighting and texture mapping can be added pretty incrementally (depending on how obsessed you are with optimization).
You may like my https://github.com/pjc50/ancient-3d-for-turboc : a slightly more modern target than the NES, it targets Borland Turbo C for MS-DOS and renders to the standard game 320x200 paletted "mode 13" display.
Standard techniques of the time; project the 3D to 2D, cull backwards facing polygons, "painter's algorithm" (draw deepest first, in this case by object rather than by individual polygon).
> I tried to follow a book on writing a ray-tracer from scratch
May I pimp my own book about 3D rendering, covering both rasterization (including 3D projection) and ray tracing? There's a complete, free version at my website, https://gabrielgambetta.com/cgfs
I was just thinking of this book as a recommendation to the parent post, and here the author himself posts it. For what it's worth, I can totally recommend it. I had written rasterizers in the past but not ray tracers until I went through this one weekend.
As a pro game dev in my past life, I promise you can get a 3D engine up and running from scratch in relatively few lines of code. Especially something like a wireframe cube that you can rotate.
> Microprose had 3D flight simulators going back to 1984 on Atari 400/800, Commodore 64, and IBM PC. That's really impressive.
They weren't the only one though. Sublogic also had Flight Simulator II for those platforms. Which wasn't really combat focused (except for the sopwith camel minigame with the checkerboard and the cardboard mountains - lol). But it was also really impressive on such constrained platforms. Ok, it did about 1 frame per second and only rendered in wireframe on the Atari, but you could tune VOR and ADF radios, set weather etc. It was really impressive for an 8bit machine with 64kbyte.
Sublogic later licensed their product to Microsoft to port it to PC which became Microsoft Flight Simulator. And the rest is history :)
It was a bit weird because on the much more capable Amiga there was a flightsim game that wasn't even able to draw a tilted horizon if the gradient function was turned on. I forget which game it was, I think it might have been F-29 retaliator.
Wow... bookmarked. F-15 Strike Eagle II is one of those games I played the hell out of as a child. So much so I wore out two joysticks that my parents bought as a result.
I was a child so it took me a long time to learn to actually do what the game wanted, and then learn after taking out the targets you got more medals by then bombing or strafing other targets. I don't think I ever got the hang of landing back on the aircraft carrier though.
F-19 was my first flightsim (well, some on the Commodore 64 I never got the hang with were the actual first).
I had a pirated copy. Back then, nobody own legit games in my country. It was hilarious that the copy protection gave you a top down view of an aircraft and made you look up the name in the manual.
Yeah... for me, an aircraft-obsessed teenager who knew all the shapes, this was the same as no copy protection at all :) If anything, it trained me in recognizing the shapes of all aircraft of that era!
PS: I was sad when I learned the F-19 was never a real thing. That said, sometime later MicroProse came up with the F-117, which we know was a real thing.
I owned multiple Microprose games and the manuals that came with the games were simply fantastic.
There would be a section on the gameplay itself and then usually some historical reference about the setting of the game. E.g. if it was a WW1 flight combat simulator, there would be a section on the history of aerial dogfighting and then specs on each plane.
I know that nowadays it's common knowledge that most people don't read the manuals and it's easier to get people into a game with a combination first level /introduction/tutorial. That being said, I feel like we lost something by taking out the manuals with all that rich historical detail.
Yes the Battle of Britain game (not by Microprose but LucasArts) actually came with a whole history book. I read it for English class, my teacher deemed it decent enough quality for that (English is not my primary language so picking high-quality literature wasn't really a priority).
It really added so much to the game. Many other games came with cool things in the box too, like Wing Commander with the blueprints of each fighter.
I guess part of the reason was that the games themselves weren't all that immersive. There were no cutscenes or long narrative. The tech simply wasn't there yet. The included box items made it more interesting and immersive.
I miss the simulation games of that era. They’ve pretty much faded away. At one point I dove back in a bit but everything was so janky trying to run that I quickly lost interest.
Not this game, but DOS game + reconstruction made me think about something from my childhood.
I used to get around 1h of Internet time (modem) per week. I used this to download stuff that I could use for the rest of the week.
One time I started downloading a game, but a bit too late so I never got to finish the download before my time was up. Of course the game would not work as it was only partially downloaded (IIRC it was a zip file, but not 100% sure).
There was some message from Windows telling me that there was something wrong with the exe. But I continued trying to open the game over and over again (maybe fiddled with the properties?) and one day it magically started, ran for a while and then crashed. It was possible to restart it, but it would crash at the same point every time.
It's still a mystery how it was possible, I guess it only had "content" left and the exe itself was fine, but that doesn't really explain why Windows initially refused to start it.
The page "What does it take to take an old game apart? (Part 3)" mentions the 'restunts' project to reverse-engineer Stunts / 4d Sports Driving, and finding limited information about it.
Me too. My first “hacking” experience was loading the save files into a hex editor and flipping bits to award medals and ranks. I still win every the old way too. Bombed the heck out of Baghdad.
My understanding of PC game development at the time was that most games would re-implement their own drivers for system hardware, hence why you would often need to select what kind of graphics card, sound card, and their settings during the setup. As such, a game running from a boot disk is closer to just skipping DOS and having no OS rather than implementing a custom OS, although from another perspective you might just say that the game is the OS.
Sort-of. IIRC DOS games that wanted to use more than 512KB(?) of memory had to use one of a series of particular high memory access drivers. HIGHMEM.SYS ("XMS"?), EMM386.SYS ("EMS"?), etc. You had to load these and other drivers (in particular sound and mouse drivers, antivirus programs, etc.) when booting up in your CONFIG.SYS and/or AUTOEXEC.BAT. These were referred to as 'TSR' (terminate and stay resident) programs. There were all sorts of tricks to get things working... LOADHIGH, etc. In the end, most games required one particular approach to high memory, and you had to have a CONFIG.SYS/AUTOEXEC.BAT that left enough base memory free as well. It was a huge hassle. In later days most programs used DOS4GW ... https://en.wikipedia.org/wiki/DOS/4G ... which made things a lot easier.
I was a 90s kid, and was totally head over heels for Origin Systems' games (Ultima, Wing Commander, etc). Par for course, I had a selection of DOS boot disks to set up the right operating environment for each game. Ultima VII featured a memory manager[0] based on 'Unreal Mode', which made getting it running even more precarious. The quote at the bottom of that link from the developer is really telling.
Didn't get the Ultima series so never experienced that one, but an Origin Systems game I loved was System Shock. Still unbeaten in my book for raw experience as an early FPS RPG... so damn scary. It had so much complexity to the plot, it really set the bar high. I don't think it was matched until 15-20 years later ... many far more popular games were pathetically linear, simplistic and mono-dimensional by comparison. Other early games which left a mark for me: Heretic (which for unknown reasons performed very smoothly on my graphics hardware - Oak?), Descent (first really 3D game, excellent gameplay and many overnight sessions with friends), Dune II (defined the RTS genre on PC), Mechwarrior 2 (strongly immersive and awesome experience with a proper joystick), X-Wing/Tie Fighter (strongly immersive). Later, Command and Conquer (first modern RTS), C&C2: A Bridge too Far (awesome top-down historic battles via modem). Ahh, these days why do all the games seem so rinse-and-repeat... I think the last one I played that really impressed was Shenzhen IO.
Then making a boot disk using 4DOS to manage the whole lot of it. As each game had its own idea of what being 'setup' looked like with different other TSRs (mouse, cd, etc).
if you tried to look at the disk in dos or windows, it would report unformatted.
took a lot of convincing my neighbor's dad that the disk really worked, but only as a boot disk, because windows 3.1 would report it needed to be formatted
im not the author of these blogs but love to read his tales of reconstructing the C code part by part using IDA/Ghidra and some of his own tools - maybe others are interested in reading too :)
I never played that one, but the Microprose "look" just leapt out at me from those screenshots (similarities with F-19, Gunship 2000 etc etc).
I did play the later Janes F-15E sim - much more complicated/harder than the Microprose ones were. Not sure it was more fun though, lots more study needed and probably needed proper stick and pedals etc to get much out of it.
F22 Intercepter over here on the genesis. I recall the game over scene being a image of a skull on a ekg machine flat lining. Used to scare the crap out of me as a kid.
Went on to play a lot of F22 Lighting 2 / 3 by Novalogic. Then Delta Force Land Warrior.
There's a name I've not heard in a long time. I loved that game so much, the engine and gameplay was ahead of its time (or felt that way). It was sad to see NovaLogic fall behind in the years that followed.
There was also an Amiga version of this game, with a sanner 68000 flat memory architecture which could have been easier to reverse engineer (compared to real mode segmented x86).
I didn't play this that much back in the day, but seeing it now it looks extremely similar to LHX Attack Chopper from the same era. Same kind of displays and fonts, same kind of briefing.. Probably from the same studio I guess? Edit: Nope. LHX was not from Microprose but EA instead. Weird.
What I liked about LHX was that it was pretty versatile with its cameras and modes. You could focus the camera on enemy tanks etc.
It was very bad at actual helicopter combat though. The terrain was all flat and there was no scenery to hide in, no hills to sneak behind, no trees. The only objects were enemy tanks, camps etc. Later another game came out with a weird kind of voxel graphics, Comanche, that fixed all that.
Novalogic's one weird trick was using voxel terrains as much as possible, like Comanche or the Delta Force series. I'm not sure if "Black Hawk Down" still had a voxel level or if they have changed the secret recipe at that point.
Spice86 is some sort of tracking emulator like Dosbox giving you the ability to replace original code parts with C# - i don't think that Spice86 can emulate Win16 currently
Does anyone remember a PC flight sim from the (mid?) 90s that supported dogfights over a null modem cable? Have been trying to remember its name for ages.
Strike Eagle 3 (the successor to version II) could do this. It supported null-modem as well as Ethernet/IP connectivity to do dogfights, campaigns etc.
Yes, the game you're thinking of is likely "Chuck Yeager's Air Combat," developed by Electronic Arts and released in 1991. It was one of the first flight simulation games to support multiplayer dogfights over a null modem cable. Players could connect two PCs directly via a null modem cable and engage in aerial combat against each other. The game was quite popular for its time and is fondly remembered by many flight simulation enthusiasts.
IIRC there was one which was multiplayer pre Internet. You called in to a BBS like server with banks of modems. (I think there were 20 players or something like that in a game.)
The combination of janky PC hardware, paper-thin MS-DOS operating system and prehistoric development tooling makes for an environment where anything goes. Real mode segmented memory and large memory models utterly confuse Ghidra's analyzers and decompiler, most common modern reverse-engineering tools either do not support 16-bit MS-DOS executables or have third-rate support.
It is more art than science in a way that I don't think reverse-engineering later software can match.