The net speed of transit is driven by stop frequency and dwell time. Remember, you simply cannot make up for the time you're not moving -- driving half your trip at 15 mph and half at 30 mph doesn't give you an average speed of 22.5 mph ((15+30)/2), but of 20 mph -- it's the total time for the trip, divided by distance.
Acceleration (and braking) of LRV has to be limited on account of standees, which generally means limiting top speeds. The fact that most LRVs travel on shared roadways (with private vehicles, trucks, bikes, and pedestrians) means they have to be prepared for sudden stops. Exclusive RoW can avoid this problem, but it's expensive: you've either got to block off surface access, elevate, or tunnel.
Tracked vehicles cannot pass one another without sidings. A consistent issue with LRVs is that if there's a breakdown, or even a delay, of an up-track vehicle, the effects ripple back through the route. Sidings (or switches) are expensive. Trolley buses (which receive power from overhead electrical wires) have similar limitations.
Any transfers you need to make will also add considerable time to your trip, and again, dwell or stationary time _cannot_ be made up by speed.
In crowded or dense urban areas, priority signaling for transit, and the lack of need to find and pay for parking, can cut some time off of trips, but that's fairly limited.