We developed this for Roam (https://ro.am/ [1]). Roam is a virtual office environment for real-time collaboration - audio, video, whiteboards, personal offices, team rooms, theaters and group chat with ML-enabled tools layered on top. It's based on Chromium & WebRTC so that we can ship a cross-platform app (Electron) as well as a nearly-parity Web client with a tiny team.
We've had a good experience with this approach, although it is far from plug-and-play. We do have to identity and patch items in Chromium/WebRTC & the server (Pion) interaction to get our video quality up to compete with Zoom/Teams/Meet. We are able to effectively compare video quality across these providers and expect to reach their video quality with this stack.
Disclaimer: I work on Roam's Chat, AI, and API.
[1] We are currently in closed beta so there's not much there at the moment.
Why are you using Pion instead of Coturn? Is there a lot of innovation left on table for TURN relays?
> ML-enabled tools layered on top.
In your opinion, have Encoded Transforms definitively settled the approach to this matter?
If I want to use free PyTorch garbage X on my camera stream Y, I can take a big hit on occupancy and latency, sometimes as high as 8 frames, by reading back the decoded video to CPU and then immediately copying back to the GPU there. Have you done any work to access the decoded frame in libwebrtc while it is still in GPU memory?
How are you guys going to force Mobile Safari to enable HEVC and VP9 by default in WebRTC? I am of course joking, you cannot do that, but if Apple doesn't want to support X in browser, and Zoom's janky MPEG-over-datachannel does, and it's on a tiny screen so video quality barely matters, do you feel like also building all of Zoom's approach is on your roadmap?
TURN is only one piece of WebRTC, and for the product we're building we want to be in control of our destiny on the AV side. Building on a fairly mature open source implementation of WebRTC was an early choice we made. We've already used this to build an international mesh architecture and few-to-many over WebRTC theater experience that can support thousands of viewers with very low latency.
The ML work we're doing is outside of video processing for other features of our platform, although we have incorporated some tools for features like background blur.
On mobile we've built native iOS and Android apps. Our goal in general is not to chase Zoom, but make a very different experience for distributed work. That said the AV has to work.
> Building on a fairly mature open source implementation of WebRTC was an early choice we made.
Hmm, you would know this best, having dealt with libwebrtc, that there's libwebrtc Chrome, libwebrtc Mobile Safari, and stuff that doesn't work. I'd say more, but if you say Pion 3-times, the Pion maintainer comes out in the comments and casts a spell on you and your jitter buffers become 10,000ms long.
> used this to build an international mesh architecture
AFAIK, Twilio, Amazon, Azure and Steam operate the only at-scale private-routing-over-TURN aka network-traversal service service for others to use, and Twilio and Amazon are the same network. Twilio doesn't even bother with Global Accelerators (aka anycast IPs), they route you via DNS responses, I doubt they've updated the code for years. Do you guys have your own private network? Surely it's "Amazon."
I suppose if you did that work, well you probably don't need 90% of the WebRTC featureset anyway, you might as well centralize it, which is what all the big chat vendors end up doing.
Anyway, it is extremely hard to innovate in this space, it's a lot of mashing together open source libraries and doing IT drudgery. Sometimes that IT drudgery is gluing C++ code together; sometimes it's deploying onto 15 AWS regions. I really appreciate the complexity of what you're doing and it all looks very cool.
Good on you for releasing a dependency to it so early. I like the approach. And kudos for bringing webrtc-node into the present (which admittedly on the web, moves pretty fast! haha, the "past" is not that long ago in real time).
Don't know why you got the downvotes--you just closed a 40M series A, right? Congrats!
Looks like MacOS/arm64 is missing from the readme/compatability table (or is x64 supposed to mean all 64bit macOS?) - and GitHub issues are turned off. Issues handled somewhere else?
Come to think of it, I guess Windows/arm64 is missing too?
That table in the README is definitely not up-to-date with the library capabilities. If you check the commit history, you’ll see a sequence of builds. Regarding the ones you’re specifically interested I’m not sure. Could be a good PR to make though if you’re interested? My feeling is that how they’ve updated it to produce recent builds has probably made targeting any missing platforms a lot easier, but i don’t know!
Regarding the issues being turned off, I think this maintained fork was not really intended to be some kind of widely used thing. I came across it today and it worked and I posted it, but when I found it, it had like one star.
I’m kind of hoping that the attention may convince the owners that what they’ve done is valuable, and may encourage the maintainers to open up issues or at least tend to it for a while but who knows? it’s OK if not. works now.
I just turned on issues, would be happy to take PRs. Our plan is to eventually get this current, but we will probably only be able to work on it from time to time and going from M87 to M94 got us what we needed.
Andrus, thanks for all of your original work! We're currently using this for some automated testing with some additional use cases planned in the future. If there is some other way that you'd prefer us to work with original repo we'd be happy to, i.e. submit PRs, take over some management of it on your original org, etc.
Hey, you're welcome! And actually, yes — I reached out to @duvallj from GitHub regarding this earlier today. He has my original email, or you could email me at my address in my HN account info.
Yeah I had a look at that too, but I'm concerned it may use a custom API that does not line up with the spec. WebRTC is confusing enough without an additional premature abstraction! Also, not to disparage, but it's missing: API compatible with browser RTCPeerConnection, which I think is essential for our use case.
Also, when you say native, I guess you mean just in Node, right? C++ or wasm binaries are going to be faster, and I'd just prefer the performance extra performance / lower overhead in something lightweight for real-time streaming that I believe webrtc-node offers.
> it may use a custom API that does not line up with the spec
Not sure about that, but it's been working brilliantly for me so far.
> API compatible with browser RTCPeerConnection
Fair enough, though I'd be curious what you'd be doing with Node not in a browser that requires this...
> Also, when you say native, I guess you mean just in Node, right?
Yeah, sorry I guess "native" was the wrong word there.
> C++ or wasm binaries are going to be faster
True, but if you are trying to distribute your NPM package to lots of users, and it requires platform specific binaries or, God help us, node-gyp, you're in for a lot of GitHub issues :)
RTCPeerConnection is central to WebRTC, it's the main p2p connection. In Node you use the same thing. The server is just another "peer". We wrap @roamhq/wrtc with SimplePeer, which, funnily enough heh, greatly simplifies things! :) RTCPeerConnection is central to simple peer.
The other library does say it has a different API, on its homepage: Easy to Use - The API design is similar to the browser's WebRTC API, which reduces the learning cost.
Regarding native binaries we have not had any issues so far, mainly as the prior major wrtc provided pre-built binaries for every major platform. I think the new one posted here expands the set of targets.
I do recall encountering a few obstacles with various addons from different libraries in different projects in the past, and node-gyp was occasionally involved. Tho I don't recall it being particularly troublesome, nor preventing us doing what we wanted. In fact I fondly recall node-pre-gyp or whatever usefully creating a build for the current target on the fly on install, which was fantastic and convenient, magic.
In the worst case to build for a custom arch, I consider these types of things a configuration issue, nothing insurmountable.
Stepping back, I think a lot of these concerns will eventually go away, or you can currently make them go away, with wasm. Which, while not at C++ perf level, is adequate and a significant speedup on node for some computationally intensive tasks.
I'm interested to know more about what particular issues have you had with native addons and node-gyp in the past and what you are currently using WebRTC for, if you're interested to share.
I maintain a Homebridge (community built system to bridge IoT devices to Apple HomeKit) plugin for Google Nest devices. The newer Nest cams use WebRTC for video streaming. This makes viewing the cameras in Apple Home a bit complex, as the bridge must setup and connect to a WebRTC stream, transcode it, and then send it out to your iPhone etc. as an RTSP stream.
Since this is for home automation the system runs on all kinds of weird devices, from RPi's to NAS's to old laptops etc. When I tried to use anything requiring platform specific binaries it was a complete mess, most of the time things would not build.
The problem may not be insurmountable if you're being paid, but since this is just done in my spare time it's not something I wanted to deal with.
Completely understand, yeah. It is annoying trying to get those builds to work! You'd rather just do something that slips right in. It's always good to shop around and find the lib that best suits your purpose!
Cool, these home automation projects seems pretty nifty, and like a compelling rabbithole where you can just keep adding stuff. Your set up sounds like a big project, well done! It's cool you created a whole streaming stack for this. The use of a dedicated streaming RTSP server is a high level of technical sophistication!
Hahaha! Thank you for saying this! Yes! Not to at all disparage the other suggested library, but I think its intended use case is more teaching and learning (at least in part). It says as much on its homepage: https://shinyoshiaki.github.io/werift-webrtc/website/build/
Ideal for learning: The code base is very small and all the protocol stacks of WebRTC are in one repository. It is also very easy to build, so you can easily add or modify features yourself.
It is undoubtedly a commendable and extraordinary achievement to create a largely compliant implementation of WebRTC in TypeScript tho.
You cannot use that library in the browser (as you cannot reimplement the primitives that use UDP as there are essentially no browser APIs for that (barring the Chrome-focused WebTransport I think)), but creating it in Node using udp libraries is possible. It's fantastic and dedicated stuff, but may not be the right fit if you want performance and reliability.
But why “WebRTC M94” specifically? M94 is just a version number, for a version of the WebRTC library that’s two years old and probably subject to known vulnerabilities at this point…
Good question I don’t know but I think it has something to do with They want to target that version of the API that guarantees will have features that are present in that milestone release.
Regarding vulnerabilities, do you know any that were present in that version? I’m not sure about what you’re saying, but I think it depends on how it’s implemented. If they’re using old libraries it’s possible but if they’re using up-to-date libraries for the builds but just targeting the M94 release with respect to the interface for API compatibility they may have sidestepped any vulnerabilities in M94 binaries.
I don’t know the details, but does that concept make sense? That there is a distinction between the implementation and the interface.
I suspect it may simply be because the people who created this had a need for whatever features were in M 94 and wanted to target that to do the minimum amount of work to get a compliant implementation without supporting features they didn’t need.
I recently discovered that the main web RTC library for node doesn’t work with node stable currently (20x) so and has been seemingly not updated for two years and discontinued by the author so I went hunting for this and plugged it in today and it worked so I’m just sharing it here.
This is Sean, one of the co-founders of Roam. It was at M87 when we started our work. Our plan is to get it to current at some point, we just haven't had time yet and this met our immediate term needs. Some version updates were easy, some have API changes that require real work.
I would be happy to take pull requests, I'll turn on Github issues now.
Hey Sean! I'm Cris at Dosyago. Fantastic to meet you here and great to see the work you have done on this. It's good someone has picked up the torch for this useful library.
This is great news about your intentions for it! I'm glad you jumped on here and congrats on Roam!
Not a stupid question at all. If you're building anything in the realm of real-time gaming or high-frequency streaming, WebRTC is your go-to. It operates on UDP, which means minimal latency and rapid data transfer, keeping user experiences smooth. Plus, it comes with built-in NAT traversal and encryption, saving you from some standard dev headaches.
While originating in the browser as a P2P tech, using a Node WebRTC library can gain you server-side control for even better performance. At the really sophisticated end that can be intelligent routing and on-the-fly quality adjustments, so you're not just faster but smarter in delivering content.
I think WebRTC is essential for any real-time project, and coupling it with Node.js takes your tech stack from good to very good. We use it in BrowserBox to stream the viewport of a remote browser: https://github.com/BrowserBox/BrowserBox
We've had a good experience with this approach, although it is far from plug-and-play. We do have to identity and patch items in Chromium/WebRTC & the server (Pion) interaction to get our video quality up to compete with Zoom/Teams/Meet. We are able to effectively compare video quality across these providers and expect to reach their video quality with this stack.
Disclaimer: I work on Roam's Chat, AI, and API.
[1] We are currently in closed beta so there's not much there at the moment.