As a Google Voice user, the downside would be sad but totally worth it if it meant I could have GV accurately screen my calls. I could still get full caller ID information for GV calls received through VoIP, and when receiving calls through POTS I could at least know that the call is from somebody I'm willing to take calls from.
The other use case applies pretty much only to people I don't want to hear from in the first place, so I have no sympathy for how it would hurt their bottom line.
It sounds like the list of people you're willing to receive calls from is pretty small. Have you ever actually received a phone call which spoofed the caller ID of someone you know? I haven't - all the spoofed calls I receive are from numbers I usually wouldn't answer anyways since I don't recognize them.
Also, note that Google Voice no doubt uses many termination providers (considering they're giving the service away for free, they'll need some pretty sophisticated routing to take advantage of cheap rates), so the second use case applies to any Google Voice user.
Why can't we have a system whereby Google is required to prove to each of their termination providers that they legitimately own and control my GV number, and thus all of those providers can assure my outgoing GV calls have valid caller ID data? I don't see where it should matter whether calls into my number take a potentially different route from calls from my number.
The other use case applies pretty much only to people I don't want to hear from in the first place, so I have no sympathy for how it would hurt their bottom line.