The problem likely is that part of the public API is in DLLS that use a private API to talk to the system. For example, public class AddressBook extends AddressBookBase, with 'Base' having a couple of private fields that may come in handy.
Third parties could figure out that private API and call it themselves.
I do not think it is practical to defend the system against thos. Moving that base class into the kernel takes performance and includes its own security risks. Shipping multiple versions of the OS with different layouts or even implementations for such subclasses might help a bit, but also would open the system for bugs that show up on only 1/N of the machines.
Third parties could figure out that private API and call it themselves.
I do not think it is practical to defend the system against thos. Moving that base class into the kernel takes performance and includes its own security risks. Shipping multiple versions of the OS with different layouts or even implementations for such subclasses might help a bit, but also would open the system for bugs that show up on only 1/N of the machines.