Alas, if we really compare to solar, then keep in mind that for a solar cell you can use 100% of the refined silicon, whereas with uranium you end up throwing away a large fraction (over 95%) with low enriched fuel, and an even larger fraction (over 99%) with high enriched fuel.
Add to that that uranium is not among the most abundant elements on Earth (about 1ppm), whereas silicon is the second most abundant element in the Earth's crust (about 27%). That alone gives silicon a huge advantage in energy/chemical impact on the environment compared to uranium.
Oh, and maybe I should point out that the chemistry to work with uranium is also a lot more nastier than with silicon. Uranium is a heavy metal, so it all happens through complexes and acidic chemistry, which limits the options on chemical pathways. Silicon OTOH is very similar to carbon in its chemistry, so there are vastly more options to process silicon, and that alone allows for far more efficient processes.