I'd suggest, make your first big goal to emit assembly, and use an existing toolchain to assemble and link. It gives you the opportunity to run your code much quicker.
if you can handle compilers, assemblers are a piece of cake. So you can add a flag to emit a .o instead of a .S when you get down to that level.
I've never actually written a linker, but when you're down at that level it's fairly obvious what needs to happen. There's some sort of binary prolog that indicates this is an executable. Then you have this table of symbolic names to offsets, you replace the symbolic references with actual numbers.
If you have something like a.o, b.o, and c.o, and a.o needs to call foo in c.o, it's kind of just sizeof(a.o) + sizeof(b.o) + offset(foo, c.o)
assemblers and linkers are kinda tough because they need to be perfect, but you don't have the hard algorithmic problems you have in compilers, like register allocation.
Also, if you limit yourself to a small subset of x86, you can pretty quickly verify correctness using existing tools, then make toy versions of your assembler and linker. Adding more instructions only means touching the assembler.
+1. Direct compilation is simpler and faster than layering everything. Abdulaziz Ghuloum wrote a great tutorial to "show that building a compiler can be as easy as building an interpreter": http://scheme2006.cs.uchicago.edu/11-ghuloum.pdf
A great compiler based on this principle is TCC: http://bellard.org/tcc/ The code generators are basically built around calling parser.next_token(). It is simple (highly recommend reading the source) and very fast. There is also a linker included.
To make things more interesting, it's "real syntax", i.e. C-like, rather than the "almost no parsing required" syntax of a Lisp, even smaller than TCC, and yet it can still compile itself.
C4 and the few variants it's spawned have probably become my favourite family of tiny, understandable, but complete compilers. The sheer elegance and simplicity --- one file of less than 1K lines --- makes studying them highly recommended. Almost all the other tiny/teaching/toy compilers I've seen are larger, less capable, or both. I don't know if anyone has made a variant of C4 that generates an AST, but doing that would probably not be too difficult either, and then you'd have a tiny compiler you could experiment on with optimisation and such.
if you can handle compilers, assemblers are a piece of cake. So you can add a flag to emit a .o instead of a .S when you get down to that level.
I've never actually written a linker, but when you're down at that level it's fairly obvious what needs to happen. There's some sort of binary prolog that indicates this is an executable. Then you have this table of symbolic names to offsets, you replace the symbolic references with actual numbers.
If you have something like a.o, b.o, and c.o, and a.o needs to call foo in c.o, it's kind of just sizeof(a.o) + sizeof(b.o) + offset(foo, c.o)
assemblers and linkers are kinda tough because they need to be perfect, but you don't have the hard algorithmic problems you have in compilers, like register allocation.
Also, if you limit yourself to a small subset of x86, you can pretty quickly verify correctness using existing tools, then make toy versions of your assembler and linker. Adding more instructions only means touching the assembler.
You can do it!