The value of land is largely subjective. It is based on demand, among other things. If the government assigns arbitrarily high or low value to land based on hypothetical worlds where the land could be used for high rises, factories, or whatever as opposed to what economics has proven is the best actual use of the land and its best estimated price, that is just a terrible version of the same thing we have now.
Assessors observe market activity. The 'purchase price' when you buy land is based on the highest and best potential use of land. The "Land Value Tax" merely appropriates this observed value for public expense (and potential dividend distribution).
The fact that you compared 'prime real estate' to swampland or desert demonstrates a fundamental misunderstanding of ad valorem taxation and land rent generally. While the rate should be the same (ideally as close to 100% as we can get without causing issues) the actual -value- would be dramatically different.
Under your hypothetical question, the desert and/or swampland could very well have 0 rental value, which means the tax would collect 0 from these locations. 100% of $0 = 0.
