Vacant housing is only a problem in constrained areas. The vast majority of the U.S. is not constrained. Your summer cabin in Montana isn't depriving anyone of a home, because it isn't driving up prices. Your unoccupied condo in Manhattan is depriving someone of a home, but I suspect that there are not so many of these.
Yeah, when most people talk about vacant homes, they don't understand that almost all of those homes are just between tenants. They are not actually being left vacant.