1. What data can be leaked? (scope)
2. How difficult is it to construct a gadget?
3. What is the signal-to-noise ratio of the channel?
4. What is the bandwidth of the channel?
The original 3 Spectre variants were basically "whole process or whole of memory, easy, tens of dB, and many kilobytes a second".
If you're looking for binary safety w.r.t side channels, I think modern hardware cannot actually guarantee it.