There needs to be a button based passcode to view TOTP instead of just pressing one button once. That would add a layer of security. A combination of buttons and number of presses should still be somewhat added security.
What's the threat model here? The TOTP code is worthless without the password. You would need to get my password and physically obtain the watch; what threat does a button code protect against?
Someone who steals the watch from my house doesn't have the password.
Someone who phishes the password doesn't have access to the watch.
The government agent who has exerted enough physical force or legal coercion to get me to cough up the password can demand the TOTP code at the same time.
The only added security I can think of on a two factor authentication thing is a fingerprint reader on a physical hardware key, and even that's more of a gimmick than anything. And maybe TOTP codes generated from a password manager, but IMO that already defeats the purpose of two-factor because if your one device with password manager is compromised, they have both password and the second factor.
The four number code IS the added security already.
