> Now that there's no product left to harm with outdated metrics: https://thume.ca/2022/05/15/latency-testing-streaming/ was about Mighty. They were nice to me, and had decent tech, but I did that testing cuz my intuition was that it would be a worse experience unless your laptop was very RAM-starved.[0]
This is where video games may be in a few years when governments and cloud services buy 99% of GPUs at volume discounts while miners and scalpers bid up the 1% that make it to retail channels. It will be cheaper to play games at max settings on a cloud desktop's $2000 GPU for a couple hours a day than to buy a $200 GPU from a scalper for $2000.
Governments aren't going to be buying GPUs in that quantity. What on earth for? And the crypto mining market is (fortunately) toast in the medium term, which gives time for proof-of-waste to be banned.
Physics and chemistry modeling, economic modeling, cryptanalysis, face/posture/gait recognition and behavior prediction, spatial inference and geolocation from video, autonomous drone swarms, war gaming, forest management, code enforcement, space object tracking, training simulations...
that would require a supermassive investment that I doubt anyone is willing to make anytime soon. Google has tried hard enough, and failed miserably, without making as much as a dent.
publishers love the idea, yeah, but people don't. until it's outright forced upon us, it will not gain traction
>Just in time for all the AI/ML startups!
thankfully, not a great time for bullshit startups, and hopefully won't be for a while longer
I don't know what the future holds, but right now, things are looking up. for example, a 3060 is $350-400 and there's no shortage of cards.
It all looks very nice and corporate with "saving RAM" or benefiting clients with low processing power but in the end solutions like this are very anti-consumer.
Why is it harder than pointing a smartphone camera at the screen and running a video conferencing tool? If the smartphone can do it, then surely your desktop can do it?
I'm an insider to this space, I have streaming software customers who individually exceed the revenue of Mighty's entire lifetime.
This Thume guy gets a lot wrong.
I'm not going to spend eons taking it apart: his ridiculous breadboard thing does not measure something precisely or accurately enough to be useful, and whatever it did measure he misinterpreted anyway.
> Encoding the whole window means latency scales with window size.
> something called “damage regions” that apps provide to the OS compositor
> possible to diff tiles of a 4k screen image in under 2ms
> One disadvantage of the hardware accelerated video encode approach is that using minimal damage regions is tricky, because all you have is a vendor-provided API to encode new full frames into a video stream, and you can’t count on the vendor to optimize their encoder for large pixel-equal screen regions getting encoded very quickly.
This is just so far off base. Until he checked the "Enable H264 encoding" box, the stream was compressed on the CPU, to VP8. So it's not even hardware accelerated. It's just slow to compress on the CPU, period full stop, which of course scales with screen size much more than the dedicated compression hardware on their VMs.
The low performance has nothing to do with "minimal damage" regions, that provides no meaningful benefit to a hardware encoded stream. An H264/H265 P frame is much more optimal than a "minimal damage" region, it's like comparing spaceflight to a horse drawn carriage.
Large pixel equal screen regions are of course encoded extremely well in something like NvEnc and Intel QuickSync. What vendors is he even talking about?
> Does it constantly send 60fps compositor updates? No. A lot of apps make this mistake but Remotey doesn’t. If you mess it up it causes WindowServer to take an extra 10-20% CPU and extra power
This gets the biggest facepalm to me. This is the same crowd that thinks Unity games are slow because of "Gfx.WaitForPresentOnGfxThread."
Enough about Thume. The simple reason Mighty failed is that it MADE. NO. SENSE. Their latency could have been zero, and approximately zero people would have still used it. I'm not going to toot my own horn here, there are definitely streaming business that not only make sense, but make a HUGE amount of $$$ SENSE, but remote browser streaming isn't one of them.
Perhaps you can tell us what's wrong with his "ridiculous breadboard thing" rather than just stating that it's ridiculous?
About damage tracking and hardware encoding: The encoder must always diff the whole frame. There is no magic involved here. It probably does it intelligently, e.g. by hashing blocks and comparing the hashes rather than the memory regions. However, comparing the whole frame still means that you must read the entire frame from memory.
He is correct in saying that an encoder that is aware of damage regions would be more efficient. This is because it would require a lot less memory bandwidth.
He is also correct about 60 fps compositor updates. You probably just misinterpreted his meaning. What he's saying is that when there is nothing happening on screen, you shouldn't tell the compositor to keep updating the content because that just wastes energy on compositing something that hasn't changed.
However, this is tricky when you have a constant bandwidth h264 stream because you will have incremental updates for the same frame. A trick I've used to get past this is to stop encoding when the size of the encoded packages goes below a certain threshold, at which point the image isn't going to improve much.
I'm not really sure what minimal damage regions are but I can imagine a case in something like scrolling where a context aware approach would beat a general purpose block based video codec, especially h264 at higher resolutions.
With a small macroblock size a large enough vertical scroll might cause the encoder to send fresh pixel data for the blocks with a large enough vertical translation. A context aware encoder would not only not miss the translation but also know where to begin its search algorithm, improving the encode speed.
In another case a window coming into view, misaligned to a macroblock boundary, would need to send more data than a context aware encoder that simply sent the window pixels and the location to draw them.
What exactly is so slow about encoding on CPU? In my career working with video encoding I haven't seen any big latency differences between where the encoding was happening - if anything CPU was often better at latency just because it supported more "wierd" encoding schemes (Intra refresh, partial encode) that were better for latency than GPU encoding blocks.
[0] https://twitter.com/trishume/status/1591995028842418176?t=Gz...