Concrete is indeed a major polluter, your point stands. The world is already experiencing a severe shortage of concrete, it's that valuable. But it's also very hard to switch to alternatives as concrete has the best compressive strength per unit volume and is very well understood as a building material. Plus, it's hard to build high-density housing out of mud huts (buildings require concrete).
