The great thing about DOS is that it is basically not an OS, at least in the modern sense. It is little more than a bunch of 16bit routines that allow you to do useful things with hardware, like having a file system and handle some basic I/O.
If you have a toy OS project, check it out. Like so many people, I have one, and even though it is a protected mode multiple address space microkernel blahblah one, I not only start it from DOS, I actually still keep DOS running by hosting it inside a vm86 task in the OS. This means I can delegate all the stuff I haven’t bothered implementing, like file system I/O, basic video output and even a whole command shell, to the DOS instance (which is allowed access to the necessary hardware), and I can then focus on writing the parts of the OS that interest me.
It transformed an insurmountable project into a small and fun side project.
Mostly just in VirtualBox, which makes for much more convenient development, and also provides a debugger. When I need to run on real hardware, then just some regular PC (or Mac, with Bootcamp) will do.
Unfortunately I don't have anything out publicly right now. It's really a toy side project, and as such it's currently neither presentable nor in any stable shape. If it ever gets somewhere where I'm comfortable showing it, I'd love to put it up, though.
Sure! The sequence is roughly like this (note that DOS extenders are doing quite similar things):
1. From DOS, load your operating system kernel into memory.
How you do this is up to you, I simply use DOS routines to allocate memory and load the kernel from disk into that memory.
2. Before transitioning into protected mode, save away both the real mode stack and the CS:IP where your DOS task should later continue when your OS is sufficiently booted up, and pass that in to your kernel. I call this the “rendezvous” point. This could be any function in your loader program. What I actually did was to let it rendezvous into a routine that performs a “Terminate and Stay Resident” (TSR) DOS call, which returns to the DOS shell (e.g. COMMAND.COM), but keeps parts of the memory that the loader program allocated in memory.
3. Boot up your OS to the point where protected mode, page tables, interrupt and exception handling, etc., are sufficiently set up so that you can start creating and switching to tasks.
4. The most crucial step: Create your DOS task. That task needs to be a vm86 task (v86 in older 386 documentation), indicated by a bit in the EFLAGS register, and set up so that DOS exactly sees what it was seeing before (with exception of any stuff you want to virtualize away), e.g. the first MB of memory should be mapped to it (if you use paging), the stack is what you saved away, and set CS:IP to your rendezvous point.
5. Whenever desirable or necessary, switch to the DOS task, where DOS will just continue running, first starting at your rendezvous point, oblivious to the fact that it’s now just a task. You can then reflect interrupts back into the task if you want DOS to handle them (so that you e.g. don’t need to write any keyboard or hard disk I/O routines yet); you can invoke and communicate with little 16bit handlers to perform some jobs on behalf of your OS (so that you can e.g. use DOS’s file system, or even its memory allocator); or the other way around: You can make your DOS programs perform “system calls” that trap into your OS to perform actions there, implemented e.g. through software interrupts.
One hint that is easily missed: DOS has a “busy” flag that helps you figure out when it is safe to call DOS functions (through INT 21h). Because DOS is not reentrant, you have to finish your INT 21h calls before you can make new ones.
It’s sometimes a bit fiddly since you not only manipulate your DOS task’s registers for it to do your bidding, but also have to emulate interrupts and even quite a few assembly instructions in your general protection fault handler (in a part of it known as the “monitor”) due to the way vm86 works, but it’s not too bad and it was fun for me. Intel’s documentation about their x86 CPUs, even the original 80386 one, does a reasonably good job at explaining it, some of it in quite some detail, e.g. interrupt handling.
There are also many possibilities on which way to handle things from there. For example, your DOS task can be given direct access to the IRQ controller registers and let DOS handle that, or you can write a virtual IRQ controller for DOS in your OS and handle the real one yourself. The same goes for many other aspects, like video memory for example. Essentially you can move stuff into your OS at your own pace, and give DOS a virtual device if it still expects access.
If you want to learn more, assuming you already have a firm understanding of task switching on x86, I suggest reading the v86/vm86 chapter in Intel’s reference.
A few additions, in case someone goes back to this post for reference, and because I cannot edit it anymore:
* As far as I know, you are unfortunately limited to 32bit OSs, as x86_64 got rid of the vm86 mode necessary for hosting DOS. In 64bit long mode, you would probably need to run a full x86 emulator to achieve the same (of which there are plenty), although I haven't looked into that aspect of x86_64 yet. That being said, for a toy OS 32bit is plenty, and I haven't run into anything yet where I wish it was 64bit.
* The DOS busy flag's use is actually more to know whether DOS is still busy with a DOS function invocation by something other than your OS. Your OS code can trivially know by itself that it performed a call which did not complete yet (DOS is not reentrant, so you can only perform one call at a time). However, the DOS task is actually everything within DOS, including TSRs (which are essentially programs running in DOS's background, including many device drivers), or, depending on how you structure it, maybe even a plain old foreground DOS application that has nothing to do with your OS. Both will make DOS calls as well. The busy flag was actually introduced to support TSRs, which also need to prevent stepping on DOS calls performed by other TSRs or the foreground running app.
* Besides the busy flag, there is actually a second flag, the "critical error" flag, which you should just treat as another busy flag.
* A useful technique at the beginning is to have DOS programs that perform system calls (e.g. through an INTn instruction) into your OS, which then may call back into DOS in various ways to perform functions. That way you already have the ability to launch programs from DOS's shell which can do something related to your OS. But they will overall still be 16bit programs.
* To run "native" programs in your OS, have some kind of loader that just loads your program into memory, and then just calls into your OS to start executing it as its own task. You can either let your loader return to DOS immediately, going back to the shell independently of your new OS task, or have your loader stay running until the task is finished, which is useful if you want to do console input/output through DOS's console.
* Again, this has a lot of overlap with what DOS extenders (like DOS4/GW) do. The main difference is that DOS extenders more or less only provide everything necessary to build "32bit DOS apps" (and are likely mostly single-tasked, with the exception of the additional vm86 task), while your own OS can be pretty much anything.
I don’t think so, at least not in any “sensible” way. DOS, x86, and IBM PC compatibles (and what came out of that) are very much intertwined. You could make something for RISC-V that looks like DOS in a lot of ways (command prompt, filesystem...), and maybe is similarly structured, but no program, or driver, would work with it unless you have full blown PC emulation—in which case you’d just run PC DOS.
That’s part of what I originally meant: DOS is so barely even any OS at all, almost any software for it still does a lot of talking to the hardware itself.
> You could make something for RISC-V that looks like DOS in a lot of ways (command prompt, filesystem...), and maybe is similarly structured, but no program, or driver, would work with it unless you have full blown PC emulation—in which case you’d just run PC DOS.
There were some programs that would run fine with just a DOS-compatible API rather than full PC compatibility; WordStar was a significant program that relied only on the DOS API (and thus would run on otherwise relatively incompatible systems) , for instance.
If you have a toy OS project, check it out. Like so many people, I have one, and even though it is a protected mode multiple address space microkernel blahblah one, I not only start it from DOS, I actually still keep DOS running by hosting it inside a vm86 task in the OS. This means I can delegate all the stuff I haven’t bothered implementing, like file system I/O, basic video output and even a whole command shell, to the DOS instance (which is allowed access to the necessary hardware), and I can then focus on writing the parts of the OS that interest me.
It transformed an insurmountable project into a small and fun side project.