Indeed, it actually seems like we've gone the other way. During the first half of the twentieth century, buying and living in Ikea-esque "Kit Houses" was actually not entirely unheard of. The Wikipeida page for Sears Catalog Homes  actually lists the price range as between "$12,590 and $55,390 in 2008 dollars," which certainly seems like it could be price competitive with other methods of new home construction, even taking into account costs like land, etc.
My grandfather did that in the 50s with my dad helping a little. You need to scale for sq footage, the average American house has increased from 1000 sq ft to something unaffordable recently. By sq foot its probably not so much of a deal.
Apparently it takes only a couple weeks for one man to build a house, but the labor involved in general contracting to get the subcontractors to do their thing exceeds the time spent swinging a hammer. He sub'd out to dig and pour the foundation, some electrical, gas, some plumbing...
For those of a certain generation it was how you proved to the employer that you could complete long complicated projects, basically the equivalent of completing your college degree (which was hard to complete during WWII)
This is still done on a regular-ish basis for large garden sheds and garages. Home Depot and others will dump a pile of wood at your location with some plans and bulk pack of nails and other hardware and a weekend or two later you'll have a garden shed.
The permitting and inspection process is more about revenue generation than safety.