I think you have a certain amount of fuzziness around the idea of an "interaction" with the data. It would probably help to think about compatibility and breaking changes in terms of reads vs. writes in order to get the determinism you're looking for and better alignment with SemVer.
That is, if a client using the previous schema can still do reads and writes without the data being invalid, you have forward compatibility, and this qualifies as an PATCH.
If a client using the previous schema can still do reads against the new schema without the data being invalid but not writes, that would qualify as a MINOR change.
(Aside: write-but-not-read compatible changes are possible, but are uncommon in practice)
A change that can prevent a client using the old schema from doing valid reads against the new schema (eg. a column is renamed or removed) would be a MAJOR change.
which has a very succinct explanation of forwards and backwards compatibility as it relates to producers and consumers.
It's high time we did a second draft of SchemaVer which explains it in terms of forwards/backwards compatibility; the actual behavior of it (when to bump etc) would barely change.
Breaking writes would not be considered a minor change in semver.