It depends. In most places in the US, bas transit shares the same road space as cars. If almost all bus users switched to cars, not only would driving be slower, but riding the bus would also be made slower as well. So driving would still end up faster than taking the bus.
Or phrasing that differently, the more people take public transport the faster everyone goes. Which is basically the whole point of it, greater efficiency in moving people around.