Most regulations require "reasonable effort" and a final judgement really only comes from an auditor / discussion with regulators.
Honestly even that is often not final except in the context of one particular audit, "reasonable" can differ from auditor to auditor and understandings can change over time.
Deletion is hard. For example, deleting a record from a database often leaves data on the underlying storage.
Database backups in cloud are often made at the volume level which poses obvious difficulties.
On an SSD or log structured fs, even when "overwritten", data may not actually be deleted until the device gets around to re-using a particular physical block.
If you are using a service like cloud-managed kafka ALL of these things might be in play at the same time! With the additional constraint that you have even less access to the underlying resources than usual.
The safest way to "really delete" say, PII in 2022 is only store it encrypted, burn the key on delete, issue deletes where you need to and hope for the best. Storing sensitive information in a dedicated place and tokenizing it / using links to limit data spreading and copying can also make your "compliance story" a lot simpler.
But no regulator I know of will make you do this, reasonable effort is the key. Almost universally "hard" db delete + an explanation of when stuff will age out of backups would be fine. It would not be "reasonable" to demand redaction of system backups.
Random thought: what if the criterion for deletion was the following?
"Made sufficiently inaccessible so that, even if validly subpoenaed by a court of competent jurisdiction, you would respond that you have no practical means for recovering the data."
It's not really a clear statement, and I think it also sort of misses the goal of deletion policies.
The above statement for me reads with the implication that it only checks accessibility by the party responsible for the data, as the example check is that the responsible party has no practical means to recovery it. But in fact this is not really a good goal I think as it doesn't really protect against what data retention laws aim for; avoid keeping data you don't need and most definitely don't sell it off.
Thus, an attacker looking to do a DB dump will most definitely engage in activities that are not practical to recover inaccessible data; a data sales team will certainly modify the data to still be equally identifiable and targeted but transformed enough that the original dataset is no longer required.
It really doesn't address what a lot of data deletion/retention policies aim to succeed with.
Agree I think - the most common mistake people make is just not doing the work. The work means:
Understanding the regulatory framework for your ecosystem (and partners), and maintaining policies that align with those requirements.
Ensuring that system owners understand the policies, and that they document how they meet the requirements.
Ensuring that actual behaviour corresponds to the design and keeping an eye out for errors (if you haven't had to scrub your centralised logging yet you should probably be looking harder...).
The intent is not primarily to sweat every detail but to limit gross violations and make sure you're not asleep at the wheel (which is common) or behaving badly.
What audits want is evidence you're "doing the work". Be able to clearly explain what you do and why you think it complies. Be able to prove what you say. The details tend get hashed out as a result of following a process.
The real-world problems people tend to have are more like "we don't know why we copied all our customer information to that bucket" or "hmm they might have onsold it" or "we don't actually know all the systems that read from this database or what they might copying out it" than "we are concerned about advanced storage necromancy".
Honestly even that is often not final except in the context of one particular audit, "reasonable" can differ from auditor to auditor and understandings can change over time.
Deletion is hard. For example, deleting a record from a database often leaves data on the underlying storage.
Database backups in cloud are often made at the volume level which poses obvious difficulties.
On an SSD or log structured fs, even when "overwritten", data may not actually be deleted until the device gets around to re-using a particular physical block.
If you are using a service like cloud-managed kafka ALL of these things might be in play at the same time! With the additional constraint that you have even less access to the underlying resources than usual.
The safest way to "really delete" say, PII in 2022 is only store it encrypted, burn the key on delete, issue deletes where you need to and hope for the best. Storing sensitive information in a dedicated place and tokenizing it / using links to limit data spreading and copying can also make your "compliance story" a lot simpler.
But no regulator I know of will make you do this, reasonable effort is the key. Almost universally "hard" db delete + an explanation of when stuff will age out of backups would be fine. It would not be "reasonable" to demand redaction of system backups.