Show HN: Maps and Splats – Mashup of 3D tile maps with Gaussian Splats

[kfarr]

https://maps-and-splats.glitch.me/

Hello HN!

I’m excited to share this open-source community mashup showing the power and possibility of combining 3D Gaussian splat scans with traditional mesh-based 3D tiles maps in the browser.

As you've probably seen on HN a bunch lately, Gaussian splats are a new method to allow custom 3D scans of scenes and objects [1] and Google Maps 3D Tiles API are the same 3D buildings and street views your familiar with in Google Earth or Google Maps, and recently they’ve been made available for inclusion in web apps.

Combining splats + 3d map mixes highly detailed real-world custom micro-scans with the macro-level 3D imagery from satellite data. This is useful for civil engineering, transportation planning, real estate, construction, creative production, and so much more – we’re just scratching the surface.

Demo1 with manual orbit controls:

- Video: https://twitter.com/kfarr/status/1773934700878561396

- Live Demo: https://maps-and-splats.glitch.me/

- Source: https://glitch.com/edit/#!/maps-and-splats?path=index.html

- Instructions: zoom in / out using scrollwheel and rotate with click and drag

Demo2 with animated camera path timeline:

- Video: https://twitter.com/3dstreetapp/status/1775203540442697782

- Live Demo: https://look-at-splat-maps.glitch.me/

- Source: https://glitch.com/edit/#!/look-at-splat-maps

- Instructions for using timeline to make your own camera flight path: https://www.youtube.com/watch?v=QDWKina45f4

These demos combine: A-Frame / three.js, NYTimes three.js 3DTiles viewer component with Google 3D Tiles, 3DStreet Gaussian splat viewer component from Luma Labs scanned scenes, theatre.js animated camera path and timeline editor, Glitch hosting / IDE because they’re awesome for hacking projects like this

These demos are results of R&D from a larger project: https://3dstreet.org

Some of these API cost real money so I worked to get a temporary quota of 25,000 sessions. After that the 3dtiles won’t load, you’ll need to get your own API key from Google :)

[1] 2 min intro to 3D Gaussian splatting: https://www.youtube.com/watch?v=HVv_IQKlafQ

[kfarr]

Next we are combining this work with an AR collection app for on-site capture and testing out workflows for "shared global anchors" (aka Spatial anchors or Cloud anchors) a recurring topic in the w3c immersive web working group: https://github.com/immersive-web/proposals/issues/82#issueco...

[spython]

Really nice! I did a quick test some weeks ago and apparently using google street view images to generate Gaussian splats works quite well: https://www.instagram.com/p/C2xozxVAdpG/

(I just took screenshots of the Street View, not the API, so the ground is falling apart a bit because of the street names.)

[kfarr → spython]

Agreed tons of potential here, however we may need to get our own geo data for this: Google was very clear during the API quota approval process that 3d tiles cannot be used for ML training or generation of derivative assets like Gaussian splats, at least with this specific product offering.

[spython → kfarr]

Have you looked into Mapillary? Could be a viable alternative, even though the image quality varies a lot.

[kfarr → spython]

Great idea... could even be generated "on-demand" only when the user requests high fidelity for a specific area!

[pimlottc]

It's not immediately obvious, but you can move the viewpoint via panning by holding down shift while clicking and dragging (on desktop, at least). My natural instinct was to double-click like in Google Streetview.

Not sure if there's a limit on the map area but I was able to navigate all the way to Ocean Beach!

[AlexAltea]

Congratulations on the amazing work/demo! An interesting extension of this would be integrating the "Level of Gaussians" approach presented at: https://zju3dv.github.io/LoG_webpage/

Also interesting would be to generate gaussian splats from Google Street View footage, and integrate it directly into the Google Maps 3D data as you have done.

[dang]

Related: https://news.ycombinator.com/item?id=39909110 (April 2024)

[theultdev]

This is the best real world (heh) demo of Guassian Splats I've seen so far.

Now to fork and enable the FPV camera controls.

Great work!

[kfarr → theultdev]

Yes, you can just remix and remove the orbit controls in lines 43 and 44 https://glitch.com/edit/#!/maps-and-splats?path=index.html%3... and then add `look-controls` `wasd-controls` to the camera entity. https://aframe.io/docs/1.5.0/components/wasd-controls.html

[theultdev → kfarr]

Thanks!

For anyone interested this is what I ended up with to fly around:

  <a-entity
    camera
    look-controls="pointerLockEnabled: true;"
    wasd-controls="enabled: true; fly: true; acceleration: 3000;"
  ></a-entity>

Demo: https://delicious-enthusiastic-accelerator.glitch.me

[tkzed49]

The video looks awesome! I'm on mobile and the first demo is loading 3D tiles but not the splats for me.

[kfarr → tkzed49]

I saw the splat "stall" during loading once while testing but couldn't reproduce. Curious if you reload after a stalled splat load if it works?

[jwilber]

This is great!

Unrelated - does anyone know a great library for building tile maps for 2D scatter plots?

[uncommonGeo → jwilber]

Check out tippecanoe for preprocessing. Alternatively GDAL also has a gdal2tiles script if your input is already a raster. An open source option for frontend viewing is maplibre.

[ghostly_s]

> [1] 2 min intro to 3D Gaussian splatting: https://www.youtube.com/watch?v=HVv_IQKlafQ

I don't know what the hell a "gaussian splat" is and I have to say this video did not help at all.

[ghostly_s → ghostly_s]

Here's a much better explainer from some people who seem to actually understand the subject matter and not just the buzzwords (and yes, the end result is basically a highly-detailed point cloud):

https://www.youtube.com/watch?v=VkIJbpdTujE

[geuis]

On mobile web you can't dismiss the text area. It covers 80% of the screen.

[kfarr → geuis]

Thanks I just shrunk it a bunch more. It's a tech demo so apologies for the basic UI. It's also open source so anyone can remix it and play with the UI too