The net speed of transit is driven by stop frequency and dwell time. Remember, you simply cannot make up for the time you're not moving -- driving half your trip at 15 mph and half at 30 mph doesn't give you an average speed of 22.5 mph ((15+30)/2), but of 20 mph -- it's the total time for the trip, divided by distance.
Acceleration (and braking) of LRV has to be limited on account of standees, which generally means limiting top speeds. The fact that most LRVs travel on shared roadways (with private vehicles, trucks, bikes, and pedestrians) means they have to be prepared for sudden stops. Exclusive RoW can avoid this problem, but it's expensive: you've either got to block off surface access, elevate, or tunnel.
Tracked vehicles cannot pass one another without sidings. A consistent issue with LRVs is that if there's a breakdown, or even a delay, of an up-track vehicle, the effects ripple back through the route. Sidings (or switches) are expensive. Trolley buses (which receive power from overhead electrical wires) have similar limitations.
Any transfers you need to make will also add considerable time to your trip, and again, dwell or stationary time _cannot_ be made up by speed.
In crowded or dense urban areas, priority signaling for transit, and the lack of need to find and pay for parking, can cut some time off of trips, but that's fairly limited.
I live in the Cleveland area, and even living in the downtown area, I could imagine it being a legitimate sacrifice to live without a car. While the downtown area is growing in residence and support services, such as grocery stores, are being built downtown, it's still tough. Also, while some businesses are moving their offices downtown, other companies, especially large ones like Eaton, have moved their substantial downtown presence to the suburbs, so the problem is happening in both the residential and commercial side of real-estate. I'd have to imagine many small and mid-size cities suffer from this.
I just don't see a solution that is going to get people to cluster more, given our culture. I think the long-term solution is electric cars and the possibilities that shared, highly convenient, self-driven cars present. So instead of changing behaviors drastically, we use technology to make the required changes much smaller.
You already feel a mild version of this in rush-hour on NYC subways. Your body will move slightly as the train breaks, often causing people to lose balance (particularly if they never experienced this previously)