As of today, we wouldn't be able to detect a same-tech-level civilization even at the closest star with the amount of effort we have given this, so there is no paradox in my mind.
All it takes is one sufficiently early civilization that goes full Neumann-replicator and decides to colonize every system it can for them to have already arrived here long ago. The fact remains that it seems like they didn't, unless they were very careful boy scouts and left no trace.
There are so many variables. For example, if they can't travel faster than the speed of light, they might not even think it's worth colonizing. They could also be on scales vastly different from our own, as microbes or as planets. We wouldn't even know what to look for in interstellar communications, let alone have the ability to make sense of anything they send. Our entire "search" has come with the assumption that alien life is going to communicate with radio waves, use sight (in the same range as our own), be similar in size to us, etc.
It just takes one weirdo civilization that decides to put up Ozymandias-style monuments to themselves everywhere- they don't even need to do it themselves. They can fire off a few Von Neumann probes and let them do the work to carve Mount Rushmores into every rocky body they can find.
It's not much weirder than things humans get up to, and if spacefaring civilizations that can build such probes are common at all (even if short-lived), you might expect at least one of them to have initiated such a project and been successful enough to have built a giant "we wuz here" radioactive pyramid on the moon. Evidently, none of them did- and depending on how many there were, this is either unsurprising or deeply unlikely.
Or if they were sufficiently advanced, they could entomb their bodies deep in a planet and observe the universe through VR and small undetectable probes. Or maybe the drake equation doesn't factor in drugs: that once a civilisation becomes advanced enough to stimulate their own pleasure receptors they quit bothering with growing and advancing?
The Fermi problem is a speculative theory based on extrapolating human civilization's use of energy to stellar levels (various hypothetical civilization "types" etc). It's plausible life at the scale of earth could be common, based on the principle of our natural non-uniqueness. So the simplest explanation is the speculative extrapolation to stellar energy uses is wrong - human already are more destroying themselves with excess energy use than preparing for any next stage, just as one datum.
