Not really. If there’s an alternative App Store, then apps in that store can use private APIs and there’s nothing Apple can do to stop them. If there is only one App Store and it’s run by Apple, then if an app tries to use private APIs, then Apple can kick them out of the App Store.
This really is an all or nothing deal.
Once you jailbreak your device, or allow an alternative App Store, it’s game over for that device.
Arguably an approach that assumes there can be a hostile store operator would create a more secure and private product, as Apple would rely less on private entitlements and APIs in plists, and more on either technical measures to control access to these APIs, or actual security across them.
There's antitrust potential around private APIs and entitlements, like the background video access given to zoom before it was available to other developers. Arguably the "green dot" status bar warning approach helps alert users to abuse of this API, and a permission prompt before first use would let users choose.
Sandboxing binaries more than at present would also improve the general security posture of the device - I'd want my app sandbox to be secure even if a rogue app gets onto the device, and such a security posture would arguably better secure iOS for all users.
I could see a need to namespace keychain and team IDs and similar with a secure identifier (like the public key of an alternative app store's signing CA key), to protect keychain and other information from spoofed apps, but again this kind of change would arguably better harden iOS for everyone. The less that platform security relies on trusting someone else to validate and sign a plist, the safer the more secure the platform will be for users even of the default store.
This really is an all or nothing deal.
Once you jailbreak your device, or allow an alternative App Store, it’s game over for that device.