https://en.m.wikipedia.org/wiki/Side-channel_attack (especially the cache side channel)
TL;DR: CPUs are designed to "safely" execute next instructions before having received the results of conditional checks that might have take the code path in another branch (of the "if" statement).
The safety here is achieved by not "committing" any effect of that code to "real" state (aka architectural state).
The reason I put safety in scary quotes is that it turns out that while the CPU does manage to fully undo any writes to memory and other data buffers, the actions it has taken so leave a measurable trace.
It's called a "side channel", because it's not the primary I/O channel, but a different, unintended way for data to be conveyed from the CPU to ether another device or to another program running later on the same CPU.
Cache side channel work by leveraging the measurably quicker time it takes to read memory regions related to regions recently read.
Thus when a program reads something in memory, you can later take a good guess which locations of memory it read (not every single byte, but with the granularity of a "cache line" which is e.g. 64 bytes)
The next trick is to find a spot in a program that contains code that reads some data and uses it to compute an index in another table.
If you're lucky and you find some code that for some input value computes a location in a table, you will be able to guess that value by looking at which cache line is hot.
There is a lot more to say about this, in particular about the ways you can find code that does what you want in the victim program (especially about ways you can basically reorganize code in ways never intended by the authors; read more about Return Oriented Programming)
the interesting aspect that spectre attacks bring to light, is that you can cause memory accesses which are explicitly forbidden because the code performs e.g. bounds checks.
speculative execution "bypasses" the checks, because that's the whole point of speculative execution: speed up the execution under the assumption that one (or the other) case is taken. If the CPU guesses that it's more likely that the bounds check will be successful, it will perform better when the input is well formed, but it let potentially malicious inputs to cause side effects detectable with side-channels leak sensitive information that the bounds check design to avoid in the first place!
As you can see, there is no easy way out here. This is a fundamental feature that makes modern CPU fast. Making sure that code can continue executing only when the results of the check is known, will affect performance of the happy path because the CPU will sit idle waiting instead of doing useful work.