My name is Hanseul Jun who is currently a graduate student working on augmented reality and telepresence systems. During my graduate studies, I have been working on code related to this app since four years ago, always wanting to bring telepresence technology outside of labs.
Hope you enjoy this! You can also share a RGBD video via a URL like this example using my app (https://apps.apple.com/us/app/telegie/id1593918560) using a iphone that contains a face ID camera or a LIDAR camera. As I have been enjoying reading comments from here and any comments would be appreciated!
Might I suggest slightly more interesting examples than kicking a ball from your desk. What about a cat playing with a toy? Or a model train going round a track?
Thanks a lot!! I am currently working on my dissertation, while developing this system, and kicking a ball while setting up the dissertation study kind of seemed interesting to me personally but I agree that it's not something interesting for others... Next time I'll at least find a cat!!
By default, mp4 files from iphones do not include depth. The RGBD videos from telegies are custom mkv files that include depth (done by directly using libmatroska, not via ffmpeg), which the webapp can display.
getting real braindance vibes [0] - for added fun try adding multiple sources with sequence alignment w/ e.g. dynamic warping based on the correlation between the audio tracks of 2 RGBD videos! Then look into NERFs! [1]
This is so cool! I’ve been playing around for a while now with different objects and types of lighting. Now I’m getting this error when I try to post:
{"timestamp":"2022-02-23T05:57:19.930+00:00","status":500,"error":"Internal Server Error","path":"/record"}
I'm really sorry about this... I don't think I can fix it at the moment but I'll keep work on making the app more stable. And thanks a lot for trying the app!!!
This is seriously cool, great work! You can get that “Radiohead video” effect by pausing and panning around a video of your face :)
I might suggest changing the name of this post as I had no idea what RGBD was initially, maybe “I made an iOS app recording videos with depth information and a web to play and explore them in 3D” or something (that’s probably too long for HN!)
Spending several years in a VR lab made me start thinking RGBD was some common term, but for sure I'll follow your advice and use more normal words next time. And you were right that it's probably too long since the above title was few letters short the limit already!
I remember watching Minority report again last year and being really surprised watching this scene!! I was like that looks like my research project but this movie is 20 years old???
Maybe we can use a neural net to fill in the shadowed regions. I have only looked at 2D inpainting algorithms and not sure what the state-of-the-art in 3D inpainting is though.
Right, the shadowed region does make the scene look less realistic, since there is no such thing in the real world. While the lazy part of me says it's okay to leave it this way, 3D inpainting is definitely something I should look into.
I think there are two missing things here. One is that this is not coming from a specialist camera, but from iphone cameras. Many mobile phones (including some android ones) already have hardware to capture depth. And the next is about color information. This video has no information loss in terms of color compared to ordinary videos. It's just that they are mapped into 3D space. The example you are watching has 720p color resolution.
If you could add a feature to measure distance when paused, it’d make renovation planning so much less stressful - I’d just walk around the place while recording a video and could measure anything post factum. No more ‘will it fit?’
Some places like IKEA will let you get a model of the furniture that you can drop into something like Blender to see how it fits.
When scanning my wife’s childhood home we found enough space for a large hidden room in the house. We couldn’t figure out how to access it without looking too suspicious (we didn’t want anyone to know we knew).
That sounds like a cool idea! Maybe I can make this website to support phone AR, then you can place the video on top of some ruler or furniture for your measuring purposes? It's nice to hear about potential use cases and good luck with your renovation!
Nice work! I’m pretty sure it will only work with the last one or two gens of the phone, but it gives you an idea of what Apple is doing, under the hood.
Apple’s ARKit is pretty cool. I’d like to do more with it, but I haven’t had an excuse, so far.
Thanks, I agree that what Apple is doing is pretty cool, both hardware and software-wise.
About supporting devices, while the better quality rear depth cameras are only available for 12/13 pro devices, the front depth camera is included in all iphones supporting face ID (3 years old), which is sufficient for running this app!
In case someone is interested in recording RGB-D videos for offline processing, I wrote an app for this https://apps.apple.com/us/app/stray-scanner/id1557051662. It simply writes the raw data files to disk, which you can export to the cloud/your computer.
This is really awesome. I think the ability to record the camera position / perspective changes atop the video (while shooting and after shooting) for export to flat video or gif would be really cool. It would allow people to record creative clips and share them on existing platforms.
The codebase was originally written for a HoloLens app, so you sort of correctly guessed where it is coming from! I think this app works the best with AR headsets and it also supports AR headsets through WebXR. It's just that the AR headsets are really hard to find outside of labs...
Between this app with RGBD videos and 3D reconstruction, I think in the future, there can be some software turning RGBD videos to reconstructed point cloud/mesh, so that you don't need to run a dedicated 3D reconstruction app at the time recording. Maybe I'll be able to work on it like... 3 years later...?
You should add a component that tracks your head in the browser and then re-renders the perspective. Should be fairly easy and would make it a lot more immersive.
While that is a great which I will definitely consider for the future, this website at least supports WebXR. In case you have a VR headset, please consider trying to run this video using one!
Thanks for trying it with the quest and for the feedback. I really haven't thought about this issue. Yes, you are right that I have held the camera with my hand. I'll think about this and probably will need to do something equivalent to video stabilization.
The easiest thing would probably be to just have the first video the people see be shot on a tripod, that will show everyone what it can be.
I have to say that I am actually thinking about using something like that for our VR fitness game. In that case the scenes would be shot from a fixed angle anyway.
I'd just have to figure out how to get that into GodotEngine and without killing the performance.
As I had prior experience with RGBD cameras, for iPhones, I've learned how to use their library than the fundamentals. For example, rear depth information comes from ARKit and more specifically from ARFrame and capturedDepthData property: https://developer.apple.com/documentation/arkit/arframe/2928....
While I cannot agree with deleting this site, which would be quite a sad thing for me, I agree that the website can be improved a lot regarding this. I am planning to make the website to download more gradually than downloading the whole video first, which will let you escape from the site without downloading that much!
My name is Hanseul Jun who is currently a graduate student working on augmented reality and telepresence systems. During my graduate studies, I have been working on code related to this app since four years ago, always wanting to bring telepresence technology outside of labs.
Hope you enjoy this! You can also share a RGBD video via a URL like this example using my app (https://apps.apple.com/us/app/telegie/id1593918560) using a iphone that contains a face ID camera or a LIDAR camera. As I have been enjoying reading comments from here and any comments would be appreciated!