Most of the "make RPi Linux boot fast" guides out there use the same approach: Start with a tiny Debian and remove stuff until it's faster.
There's not a lot of good information about attacking it from the other way: Start with a bootloader, firmware, and a kernel, and add only the stuff that you need. I mean, it's probably out there on the web somewhere, but I've found it tough to find among all the "just strip down Debian" blog posts. If anyone has experience doing this and could point me to a few resources, I'd be grateful!
I'm not the best one to answer your question, but I know there's Buildroot and Yocto - the latter being a bit "heavy" in terms of setup. I think I would go for one of those distros that focus on minimalism (Damn Small Linux, Alpine, ...) which also have precompiled packages for your target (note: Debian does have a tool to build custom root filesystems [1]).
From there, get gcc and make for your platform, so you can carry on with compilation on-target, if possible; cross-compilation can be tricky. It should be ok if you don't want the heavier GUI stuff like X11/Wayland, browser etc.
There's not a lot of good information about attacking it from the other way: Start with a bootloader, firmware, and a kernel, and add only the stuff that you need. I mean, it's probably out there on the web somewhere, but I've found it tough to find among all the "just strip down Debian" blog posts. If anyone has experience doing this and could point me to a few resources, I'd be grateful!