Often the RTOS is not exactly free, but not entirely closed either. A while back, i used to work on Nucleus RTOS by Mentor Graphics with a pretty impressive global foot print http://en.wikipedia.org/wiki/Nucleus_RTOS. It used to be sold as an api (with source code given to customers) who developed applications based upon it. I have written portions (IPsec/IKE, SNMP, Ipv6) of its networking stack and at least all of its customers have access to source code. It is pretty well written with very decent coding conventions and can be compared to any good well known open source project (VLC, even Linux kernel). Then there are others such as Wind River's VxWorks among the more popular ones. Though i am not very sure of its licensing model, but it is pretty well recognized and established in the embedded world. Just that these are not as well known in the over all software community but rather more restricted towards those in the embedded industry.
Nucleus pretty much comprises of a very small foot print. With Architecture specific assembly isolated from like 95% of the code neatly. Rest is Ansi C. It contains tasks which are sort of equivalent of kernel level threads in POSIX but implementation logic is quite different i.e. RTOS constraints are handled by classifying interrupts at two levels. In terms of constraints, there is no dynamic loading i.e. you have to build a single binary. But at the same time it was pretty fascinating with os, networking stack, drivers all contained in a separate folder building up one project. Lately they have added power management, Android like UI and even some hyper-visor support. Most importantly, it is small and consistent enough for a programmer willing to learn through the entire stack. Helps with much better visualization from hardware to application. A couple of former colleagues (one of which incidentally now works with QNX and hence compared both) highlights both i.e. strengths vs weaknesses of each. But it didn't feel like one was superior to another. However, Nucleus severely lacks any certifications (and ability) to get into HARD real time industry such as aviation.
