Voltage spikes from line inductance, voltage drop-outs from line resistance. Basically you have little reservoirs of charge scattered all around the board (current flow isn't instantaneous in a real circuit).
It helps to always think of current draw in a compete loop, out the "top" of the capacitor, through your IC, and back into the ground side (this isn't necessarily what's happening physically). Shorter loop means less inductance, shorter traces less resistance.
Decoupling can be used on signal lines, but we’re talking about power lines here. If you have something like an ATMega, it will change its power demands very frequently at high frequencies due to switching signals on and off. Without some additional capacitance on the board, the inductance will dominate at high frequencies. Inductance resists change in current, and you’ll get voltage drops.
Capacitance is in many ways the opposite of inductance. If you place capacitors close to the power sink (e.g. the ATMega), the traces will have lower inductance and be “decoupled” from the power supply.
Smoothing is part of the story: but the important question is what is causing the roughness? Switch mode power supplies have inherent output ripple that can be filtered, but that’s distinct from transient variations in the load. Decoupling capacitors are used to provide a low impedance path at high frequencies i.e. fighting inductance.
How/what does adding capacitance help with?