Perhaps the IO bandwidth between the RPi and its peripherals isn't high enough to enable it to manage an SDR. (I don't imagine you could connect a modern GPU, or a video capture device, to an RPi either.)
Speaking of: are there any SBCs that have PCIe-based daughterboard support? Or Thunderbolt-3-based peripheral support?
Those based on the Zynq-7000 line from Xilinx usually have ample IO, but start at about 100$, going up. You might need to do something in the FPGA part of them to use the IO though.
Plunk an rtl-sdr chip on a RPi board and Bob's your uncle...