Architects can design buildings that solve social problems. But they can't use Le Corbusier's authoritarian methods - or any derivative of that mindset - to do it.
This was a typical early 20th century worker housing project. It was pretty successful for the time - so much so it's now expensive and gentrified.
Most people want to live in low-density human-scale housing with plenty of greenery. There's nothing complicated or difficult about this. The problem is that architects want to Make A Statement, and you don't do that by giving ordinary people what they want.
Architects can design buildings that meet people’s needs, but that doesn’t necessarily solve social problems. That’s why it seems simple when in fact building someone a house is not the same as making them self sufficient or giving them the knowledge, skills, and habits to be successful, especially in wider social contexts where social problems live.
The problems that architects solve are much simpler.
Disclaimer - I studied architecture in university and then engineering, and I have a fine appreciation of art and good design but a poor opinion of most architectural theories, including the ones where they give themselves super powers while designing balconies or laying out kitchens.
Those look not unlike American social housing on the East Coast (not NYC). I suspect structural concerns outside the building's design (primarily support for the people living in and maintaining them) are an important input.
Edit: Here's an example, Highland Dwellings in DC.
It is worth noting that a lot of these tower in the park type developments were demolished and replaced with mixed-income housing due to the problems. NYC is probably one of the last places where many of them are still standing (for lack of money to develop them, and also because there are about 600,000 people on the NYCHA waiting lists). But there was a huge federal pot of money for doing this in the '90s and '00s and many authorities did so.
From an architectural standpoint, the difference is that the British homes front the street, whereas your lowrise homes are perpendicular to it, and the towers in the park are generally pointed inward, away from the normal street grid. This has pretty bad effects on crime, because it removes the "eyes on the street" effect from passersby.
That's a fair assessment. I have to imagine that the design distinction of "complexes" (with, as you said, limited direct access to streets) rather than "neighborhoods" contributes. Such layouts make it more difficult for "outside" assistance to "enter" or "retreat" in response to appropriate circumstances. It turns out that making the primary concern of design the warehousing and cloistering of as many human beings as possible in as small a space as possible, far away from employment and resources, does not make as many strides towards the goal of promoting stability and healthy socioeconomic development as may have been hoped.
I'm a 90s kid and so was not familiar with anything but low-rise housing outside of old TV shows and grandfathered infrastructure; your timeline follows my experience, and also the general trend of halting progress on such issues of social justice, where the answer is and has long been known but is warped in execution by a continued, if lessening, marriage to clearly-incorrect-but-politically-attractive notions about what "those people" need and/or deserve.
It's worth noting that HOPE VI and the demolition of public housing solved the immediate problem of "the housing projects are dangerous" but did not solve the deeper problem of the people in those projects and their communities having issues with poverty and crime. There is some debate about whether or not HOPE VI demolitions mostly just served to disperse crime around a city rather than keeping it in problem areas; I can't find the articles anymore due to Google's dislike of older material, but it was blamed for general citywide crime issues in Chicago, for example.
(It was also a political marriage of convenience because in the '90s, railing against welfare was in vogue, and HOPE VI generally did not result in 1:1 replacement of public units, so it was a backdoor way to reduce the population on welfare.)
I don't really think the issue is density. Plenty of low traffic, low density suburban areas have high crime as poor people move into the suburbs; public housing had few eyes on the street because the design intentionally kept out passersby, and if anything suburban neighborhoods with high walking distances, mazelike streets and inconvenient, unpleasant walking environments are more extreme than that, and also have lower amounts of residents to boot. At least in NYC (and probably true of a lot of very inner-city housing projects) the towers in the park usually replaced existing urban neighborhoods at far lower densities than what had previously existed, because so much of the land was just idle greenery (that usually wasn't useful, in the form of small, segmented lawns often fenced off). Compare that with, say, Vienna, where the masses are housed in public housing in large midrise (5-10 story) buildings.
Coincidentally, this large midrise form factor is also what dense gentrifier buildings look like in many cities around the US, from Seattle to DC.
This was a typical early 20th century worker housing project. It was pretty successful for the time - so much so it's now expensive and gentrified.
https://www.google.co.uk/maps/@51.5114036,-0.2416956,3a,75y,...
Most people want to live in low-density human-scale housing with plenty of greenery. There's nothing complicated or difficult about this. The problem is that architects want to Make A Statement, and you don't do that by giving ordinary people what they want.