This is already the case with Proton Mail. Your encryption key is stored encrypted so that we have no access to it. It is decrypted when you enter your password, which we have no access to. The article above confirms that, in fact, and all of these cases are proof that our encryption cannot be bypassed by legal means. All the data stored encrypted on our servers (email content, calendars, attachments, etc.) is inaccessible to us: https://proton.me/blog/zero-access-encryption. All we can provide is the unencrypted metadata which we need to have access to in order for the services to work properly.
This is fundamentally incompatible with software updates. The keys must not just never leave the device, both the key distribution and what the software does with decrypted data must be locked as well. Which means it must be impossible for the software to update, as the government could use force to compel a company to hack it's own users.
This means that the cloud model, where code is downloaded through a web browser, is right out. Regardless of where the keys are.
> as the government could use force to compel a company to hack it's own users.
Currently, in the USA, the most generally accepted view among legal scholars is that this is not legal for the government to do -- mainly because of a view that it's a particular form of compelled speech / forced labor which is unconstitutional for the government to compel.
It is, however, an avenue that the law enforcement community does occasionally investigate as a possible route to get what they want. There's no clear court ruling as no cases have gone that far.
You either build your model to be resistant to a data request (user controlled keys that never leave the device), or you don't even come to the table.