Anyway, so I've tried many of the recent protocols. I was really hoping that HLS would work, because it's so simple. For example, I can use the gstreamer "hlssink" to generate the files, and basically deliver video with a one-line shell script and any webserver. But the 7 second best case latency is unacceptable. I really want 1 second or better.
I looked at MPEG-DASH: it seems equivalent to HLS. Why would I use it when all of the MPEG-DASH examples fall back on HLS?
I looked at WebRTC, but I'm too nervous the build a product around the few sample client/server code bases I can find on github. They are not fully baked, and then I'm really depending on a non-standard solution.
I looked a Flash: but of course it's not desirable to use it these days.
So the solution that works for me happens to be the oldest: Motion JPEG, where I have to give up on using a good video compression (MPEG). I get below 1 second latency, and no coding (use ffmpeg + ffserver). Luckily Internet Explorer is dead enough that I don't have to worry about its non-support of it. It works everywhere else, including Microsoft-Edge. MJPEG is not great in that the latency can be higher if the client can't keep up. I think WebRTC is likely better here.
Conclusion: here we are in 2019 and the best low latency video delivery protocol is from the mid-90s. It's nuts. I'm open to suggestions in case I've missed anything.
I had to hack it quite severely to get fast load with fair resilience for my usecase as the devices are restricted in performance and can have fairly low bandwidth. Since you're looking at a relatively fast connection, simply reducing the chunk size should get you to the target.
As a follow up - I've spent a couple years working on a video product based on WebRTC. This either works for a PoC where you just hack things together or on a large scale where you have time and resources to fight odd bugs and work through a spectrum of logistical hoops in setting it up. So unless you plan to have a large-ish deployment with people taking care of it I would stick to HLS or other simpler protocols.
RTMP protocol has a lot of implementations and is still widely used for the backend part of transmitting video at a low latency (i.e. from the recorder to the server).
RTSP with or without interleaved stream is another option.
DASH/HLS is a solution for worldwide CDN delivery and browser based rendering. Poorly suited for low latency.
If you need low latency and browser based rendering you need something custom.
I have used nginx-rtmp in the past - a very solid solution for my use case.
But this is also a little annoying to be forced to use flash to open the stream. I wonder if we could find something better now.
Speaking of nginx-rtmp, look like dev is stalled now. Anyone know if there is an alternative or someone who took over ?
Some VNC services like noVNC and Xpra also uses WebSockets.
Sadly browsers do not come RTMP enabled.
Contact me if you want to discuss it further.
"8-12 seconds End-to-End Latency" for community edition.
Actually a commercial product is not necessarily a problem, but the monthly fees are. If there was a one time fee version (perhaps with limited number of clients or something), then this might work.
I also poked a round with making a real time remote desktop client that could be access via a web browser for linux. It to -- at least on local lans got very low latency video. The link for that is below too.
Edit: I should mention latency were measured in ms, not seconds, even for many clients. I am sure to scale out to 1000's of users I would have to add a bit, of latency but not by much.
That being said, the ffmpeg solution is not using the hardware accelerator either, even though it does support MJPEG. But I think with work we can get a gstreamer based solution: the missing part is an equivalent of ffserver that works with gstreamer. The hardware vendors like to provide gstreamer plug-ins for their accelerators.
My code is geared to robots -- and has not been updated recently but there is at least a example of the simpler multiplexing in go.
I am not a fan when Apple obsoletes features that people love and use. But I always support when Apple forces everyone to upgrade because friction from existing providers is what keeps things slow and old. Once Apple requires X, everyone just sighs and updates their code, and 12mo later, we are better off for it.
That being said, I agree with author's disappointment that Apple mostly ignored LHLS instead of building upon it. Chunked encoding does sound better.
The obvious benefit of not using it is that you don't need your CDN to do TLS, likely to be utterly superfluous if video chunks are validated through the secure side-channel of the playlist already.
(I have no idea why video streaming might be better over HTTP/2 either.)
There's variation in the segment-to-segment sizes of video. Watching the stream of data, you can pretty easily find many of the segment sizes, and from there you just need a lookup table.
Figuring out spoken language or which web pages are in packets is fuzzier but still viable.
Now the best we can do is over 1 second, and closer to 3 seconds for something like satellite TV, where everything is in control of the broadcaster from end to end.
I suppose this is the tradeoff we make for using more generalized equipment that has much broader worldwide access than analog TV.
Unless your content operates in a very small niche, "real time" is far less important that continuity.
In rough order of preference for the consumer:
1) it starts fast
2) it never stops playing
3) It looks colourful
4) Good quality sound
5) Good quality picture
one of the main reason why "live" broadcast over digital TV has a stock latency of >1 second is FEC. (forward error correction) this allows a continuous stream of high quality over a noisy transport mechanism. (yes, there is the local operating rules for indecent behaviour, and switch and effects delays, which account for 10 seconds and >250ms respectively)
For IPtv its buffer. Having a stuttering stream will cause your consumers to switch off/go elsewhere. One of the reasons why realplayer held on for so long was that it was the only system that could dynamically switch bitrates seamless, and reliably.
There is a reason why netflix et al start off with a low quality stream, and then switchout to HD 30 seconds in, its that people want to watch it now, with no interruption. They have millions of data points to back that up.
There is just a broad lack of interest in reducing latency past a certain point unless there is a business reason for it. People don't notice 1 second of latency.
I remember Hangouts being better than Skype, but that's not high praise. Every calling service I've used has been laggy and disconnects often.
No the didn't. Early attempts at streaming videogame were unplayable even with a server in another room or a direct DC connection.
And they want it to work over an average internet connection in America.
No, the definitely did not solve the issue
It has been possible for years to get a total encode+decode latency of less than one frame with x264.
Meanwhile many people are gaming on TVs that impose 3-8 frames of processing lag.
And you can beat most current tech by more than half a frame just by supporting HDMI 2.1 or variable refresh rate. (Instead of taking 1/60 of a second to send a frame, you send it as fast as the cable can support, which is 5-12x faster)
my 144Hz screen would disagree.
That seems to be mostly solved with high speed links and better encoding technology.
I think youtube is the only streaming service that does it very well without any issues for the end-user, anywhere in the world. Mostly because of their free peering service that is extremely ubiquitous. https://peering.google.com/#/
For example, there were noticeable quality differences between MP3 coders.
Different encoders and standards will have different problematic scenarios. It seems like the number of problematic scenarios is decreasing.
I can encode DVD video (MPEG2) as h.264 at a huge bitrate decrease with no apparent quality loss.
Certainly streaming real time is harder with digital formats, but it’s generally good enough.
And don't forget how low and inconsistent the quality of analog TV was compared to what we can broadcast digitally.
The real story here is that latency isn't actually important to live TV, so it's a no-brainer trade-off to make. If you look at other transmission technologies where latency is more important, like cellular data transmission, latency has only decreased over the years.
Mixer can do about .2 seconds.
The technical writeup of this post are spot-on, though. I prefer less drama with my bias but I’m very glad I read this.
So with this, you can not have a manifest file that point to next future chunks (e.g. for up to next 24 hours of live stream) and delay proccessing of http request until the chunk became available. Like HTTP Long Polling used for chunks.
> On the surface, LHLS maintains the traditional HLS paradigm, polling for playlist updates, and then grabbing segments, however, because of the ability to stream a segment back as it's being encoded, you actually don’t have to reload the playlist that often, while in ALHLS, you’ll still be polling the playlist many times a second looking for new parts to be available, even if they’re then pushed to you off the back of the manifest request.
Which could be avoided if Apple didn't enforced the availibilty of download "at the full speed" once it appeared in the manifest. (long polling of chunks)
LHLS doesn't have this issue as the manifest file itself is streamed with chunked responses hence it makes sense. (streaming manifest file)
> For the time being at least, you’ll have to get your application (and thus your low latency implementation) tested by Apple to get into the app store, signaled by using a special identifier in your application’s manifest.
And this makes me to think about the implementability of the 1st and 2nd point on ALHLS. Maybe the current "implementation" is compatible but not with the specs itself.
> Maybe the current "implementation" is compatible but not with the specs itself.
It's perhaps worth noting that this is a "preliminary specification" and an extension of HLS. HLS itself is an IETF standard (well - an "Internet Draft"): https://tools.ietf.org/html/draft-pantos-http-live-streaming...
I don’t see why this would be the case. If you measure from the time the last bit of the playlist is returned to the last bit of the video segment is pushed to the client, you’ll be able to estimate bandwidth accurately.
Based on my loose understanding of HTTP/2 server push and ALHLS, the sequence of events will be:
1. Client requests playlist for future media segment/"Part"
2. Server blocks (does not send response) until the segment is available
3. Server sends the playlist ("manifest") as the response body along with a push promise for the segment itself
The push then begins with the segment.
The push stream can presumably occur concurrently with the response body stream. So I don't think you can wait until every bit of the playlist comes in. Likewise, you can't use the playlist bits itself to gauge bandwidth because the server imposes latency by blocking.
I mean... HLS predates DASH. It would've been hard for them to support a common standard which didn't even exist at the time. Initial release of HLS was in 2009, work started on DASH in 2010.
I'd also disagree with the characterization of DASH as "the commmon standard" - it's certainly a legitimate standard, but I feel like support for HLS is more ubiquitous than support for DASH (please correct me if I'm wrong).
I think you have the NIH cause-effect the wrong way around.
Citation needed :)
> The royalty for DASH Clients, which is now payable from January 1, 2017 forward, remains US $0.05 per unit after the first 100,000 units each year.
But the problem is the reverse here. Apple have patents on HLS itself which are not free, so it's not suitable for general adoption.
Apple however are the owners of HLS, and unlike some random patent trolls, if they are insisting on its adoption, they have to make sure their patents on it are royalty free. Not that it will protect anyone from further patent tolls attacks from the likes of MPEG-LA, but that's a requirement.
Has MPEG made any statement about the MPEG-LA patent pool?
I'm not sure what that means. Either they released it royalty free or not. Since there is no public statement about it, there is no reason to assume it's free.
> Has MPEG made any statement about the MPEG-LA patent pool?
I don't think anyone cares to make statements about patent trolls. The only effective way to deal with them is to bust their claims in court, which not many want to do.
Yes, and v7 of the protocol is documented here: https://tools.ietf.org/html/rfc8216
> This document is not an Internet Standards Track specification; it is published for informational purposes.
IETF standard should be free as far as I know. So the fact that it's not is already suspicious.
Knowing Apple, they probably patented it to the brink. So they released it royalty free? And when exactly?
Note the word 'and'. That link wasn't to explain that it's free.
Also "is not an IETF standard" is a pretty low bar for suspicion!