Good point; there's only a single server handling all the requests so things can be a bit slow depending on the time of day. And there are browser limits to the number of concurrent connections.
Mobile browsers with native HLS support are faster as well. Those without need to load everything via https://github.com/video-dev/hls.js/ which slows things down.
I'll try to add some UI feedback so people don't give up too quickly.