The worry is that not only will we be utterly destroyed, but that the destruction will be so thorough it will blow backward through time erasing all events that ever happened and making it so that we never even lived. That means everything we experience right now didn’t happen, we are just seeing a probability of what could happen but didn’t because it’s all destroyed. These lives mean nothing.
Imagine they told you that when you die, not only will you be gone, but then they will go back and undo everything you ever did and basically make it like you never existed at all. It didn’t happen, you never happened.
It happened, therefore it happened somehow, someplace. What’s the difference between something having happened and not having happened? It doesn’t intrinsically mean anything.
In addition, most humans learn to accept that there’s a good chance nothing they do will have an eternal effect on reality without needing any strange frameworks.