The "housing shortage" is due to a shortage of housing. This is entirely self-inflicted by various cities that have illegalized all the easy ways of increasing density (see San Francisco for the reductio ad absurdum case), at the prompting of deeply NIMBY voters who oppose anything that will increase construction regardless of claimed political beliefs.
I'm not an expert, so forgive if this is a stupid question...
What you just said might be absolutely true in the US. However, housing is at historic levels everywhere in the developed world. How do you explain that? Surely not all cities in the world made the same mistakes?
There are housing shortages in localities due mostly to distribution issues. Nationally, the number of housing units exceeds the number of households (families). In theory that market should correct this by causing places (San Francisco, in your example) to be less attractive and people will choose other places to work and live.
The work from home trend didn’t last and was largely unrealized. No, you can’t work from Butte, you need to be back in the office in Seattle.
Buffalo has about half the population now than it did at its peak in 1950. They just tore down housing because they could not afford to maintain them, and before they would fall into disrepair and became dens for crime and rodents. Similar de-housification is also occurring in Detroit, the market adjusts one way or the other.