Hacker News new | past | comments | ask | show | jobs | submit login
Plan2Scene: Converting Floorplans to 3D Scenes (3dlg-hcvc.github.io)
180 points by homarp 2 days ago | hide | past | favorite | 53 comments

I'm the first author of this, happy to answer any questions. It's great to see this on HN.

PS. I'll be graduating this July with my masters.


As far as I can tell it's not directly related to yours - but are you aware of any software that uses say, Android OS and a camera or pre-existing photos of the inside of a house/office/building to create a 3d map? Matterport seems like it might but their Android app is not going so well.

Ben Claremont has done a lot of reviews of 3D virtual tour software, see https://youtu.be/uKkQQ0aHRSc

Ben’s reviews are the best I’ve seen, but he is far from the only reviewer in this space.

Unfortunately, most of the software in this space does not actually generate a floor plan, even if they do create a 3D model of the space (like Matterport). There is one program he highlighted which is bundled with a service from the company that you can use to process your pictures and generate a floor plan among other things, but some of that work is still done by human beings.

I’m still looking for a good consumer grade program that can do photogrammetry of a space and turn that into a floor plan. I’ve seen attempts to do this with phones and cameras, and the LIDAR scanner that Apple has recently introduced has improved the situation, but IMO it is still not there yet.

But please correct me if I’m wrong. I would love to be wrong.

Did a bit of googling and this came up: https://canvas.io/. But, it's not for Android.

Ahh, yes — they used to support the Structure.io scanner that you could strap to an iPad or iPhone. I bought one of those. I could never get it to work.

Edit: the Canvas.io software really wants the latest iPhone or iPad devices with LIDAR scanners built-in. It will supposedly run on lesser hardware, but with greater error in the constructed 3D map. Fine for a sample to get an idea of whether or not it might work for you, but not really enough to actually use for anything useful.

Note that the Canvas.io software really wants the latest iPhone or iPad devices with LIDAR scanners built-in. It will supposedly run on lesser hardware, but with greater error in the constructed 3D map.

Fine for a sample to get an idea of whether or not it might work for you, but not really enough to actually use for anything useful.

I was reminded about a program called MagicPlan. See https://youtu.be/gEUk7YzfTcc for a video demo of using it to scan an apartment, using an iPhone 12 Pro with the LIDAR sensor.

But IMO, this still isn’t good enough. It’s a huge improvement over the corner mode, which is what you have to use if you aren’t using a device with a LIDAR sensor, but it’s still not enough.

I’m still looking.

Congratulations! Always great to see SFU students in the wild :)

Thank you!

You know it would also be useful to work in reverse. I get frusterated when house listings don’t include a floor plan. How cool if a program could go through the photos and guess a floor plan. Obviously a hard problem.

A hard problem, but back in 2008, Microsoft Photosynth could stitch together a user’s collection of images into a 3-D model. I remember seeing a detailed model of the exterior of the Notre Dame cathedral.

For people who have physical access to a property, I wonder if you could use a variant of SLAM like the robot vacs - maybe hack one of those laser tape measures to plug into your phone as you walk around the house.

Photosynth was a great app, really disappointed when it was discontinued. Does anyone have the backstory on that? It seemed to have a leg up on Google’s street view/cardboard app at the time.

Bringing a laser tape measure and a pen to a physical viewing, and just doing this manually, gets you 98% of the way.

It's not automatic of course, but you can do it pretty discretely and cover a whole house with measurements in ~ 10 minutes. I did it every single time when we were looking at houses.

Also always brought a powerful flashlight and a small spirit level.

Well if you take video or many photos of a house you can create a 3D model using photogrammetry. I was recently served an ad showing that I think Redfin includes 3D photogrammetry captures of some homes. So it’s perhaps already being done in the wild.

There are a number of companies operating in the space. Matterport is one some people in my work have been exploring for documenting projects in various stages of construction. There are definitely still a bunch of rough edges, but the results so far even just using a phone to capture the spaces have been acceptable to good.

Nice! Glad to hear it’s not just for research nerds these days. I’ve done a fair bit of experimentation with photogrammetry and it does work well for the built environment.

I have an application however where I want to be able to capture an outdoor forest scene with a video camera and convert that to a video game/simulation scene which renders efficiently. It turns out that is a more challenging problem that actually no one has solved as far as I know.

There’s lots of neat research tho and it will be possible some day. This bit of research is pretty cool: https://youtu.be/JuH79E8rdKc

This is actually a very interesting video. Reflection effects look very realistic.

Yah it’s cool stuff! Currently NERF takes a very long time to calculate and render but folks are working on improving that.

Do you think its possible to decouple lighting effects from the generated 3D scene? Currently it seem to simply copy the actual lighting condition in the environment. In a way, it's a limitation. It's much better if it can provide the RGB color + other material information (SVBRDF decomposition). If that's possible, then these meshes can be used in other environments.

Estimating the global camera pose using non-overlapping photos is a challenging problem. This is doable if the photos densely cover the house (e.g. video frames).

However, if the global camera poses are available, we can detect surfaces in photos and project them reasonably well into 3D. In the past, I have used this work to achieve something similar: https://github.com/NVlabs/planercnn

In addition to Matterport, mentioned in another comment, there's FloPlan, which has the agent walk around the house taking a video of the baseboards, and constructs a floorplan from that. Not entirely accurate, but good enough for a rough idea of the layout of the space.

I believe it can also use the LiDAR sensor in newer phones, but I haven't tested that.

An easier sell than Matterport, which (IIRC) is a $5K camera. FloPlan just charges a per-plan price (again, IIRC).


Floplan is based on the technology from Cubi Casa at https://www.cubi.casa/ but FloPlan is only available for MLS property listings.

If you want the generic service, you have to go to Cubi Casa.

Matterport will also work with a phone, so the fancy camera is optional.

I don't know if this is available where you live, but where I live the county assessor maintains a web application that contains information about each parcel and the buildings theron, including simple floorplans.

In Austin, Texas, you can get a “platte map” which will show the boundaries of the property and a simple outline of where the known structures fit in that map, but that doesn’t help with indoor floor plans.

You could try RealityCapture, https://www.capturingreality.com/, which is a professional 3d capturing software. It has been free to try since Epic bought it. Unfortunately, you'll need a lot of photos to get reliable 3d models. But playing with it will probably give you an understanding of how hard that problem really is.

So far as I can tell, the RealityCapture software only runs on Windows. You would have to feed it pictures or data from somewhere else (like your phone), but the software itself doesn’t run on the phone.

It also seems to focus on creating 3D models of things and not spaces.

A friend invented a device to do that. It was a small backpack you (or whoever is making the listing) would wear and walk into each room of the building. After gathering the data, software would turn it into a floorplan. It probably took less time per room than getting good listing photos. Not sure why she never went to market with it but I'm going to guess something similar could be done now with just a modern smartphone and the right software.

Matterport does this and uses it for real estate listings all the time.


The "dollhouse" view is really handy for being able to get a sense of the space.

There is a plenty of solutions like this already.

The earliest I am aware of was a specialized mini "cart" studded with video cameras that one would roll around the place and will get a textured 3D model that can then be walked around, Doom-style. It was made by a Vancouver startup was back in mid '00s.

There is also a specialized handheld "scanner" gadget that was marketed towards police use for capturing crime scene details. You'd just wave it around and see the resulting 3D phtogrammetry capture in real-time. Missed a bit - go back a wave a bit more. This was in the early '10s.

I'd imagine that this has been ported to smartphones already, because they now have sufficient sensor, storage and processing capacity for that.

> There is a plenty of solutions like this already.

I definitely feel like no one has won the market yet. Having something available or somewhat good isn't market dominance.

I wonder how it would compare to the floor plan my vacuum robot creates every time it is working. Or whether my Neato's floor plans are anywhere near realistic.

Or what Neato does with all these floor plans on their servers ...

The floor plans for every house should already exist in some database. Planners/builders at some point must had acquired them from the architect.

It would be nice if there was standardization for such things. No doubt most counties will have on file somewhere blueprints and drawings that were submitted for building permits but most likely haven't been digitalized. I'd hope that newer homes could have their blueprints submitted digitally by the architects but I suspect most places are still using paper centric processes.

About twenty years ago I worked for a big box store's corporate headquarters. One of my tasks was to add new stores to MapBlast so they would show up on the "Find a store" section of our website. Problem was that often stores were built in new sections of rapidly growing sprawl so the streets weren't in MapBlast yet. I'd have to make educated guesses as where to put the store on the map. Sometimes I could call the store and get some help that way but for our international stores, that was difficult because of the language barrier. We knew that there was some department somewhere in the company that had detailed drawings of each store, including precise location coordinates but the company was so large, we had no clue where to even begin to find that department.

There are some prior work that can digitize (vectorize) floorplans: https://github.com/art-programmer/FloorplanTransformation.

This can detect rooms as polygons, doors/windows as line segments and bounding boxes around objects (e.g. cabinets, sinks etc. indicated on floorplans).

Interesting startup idea to digitize the blueprints/floor plans by working with counties. I'd imagine if you offer them some money they won't refuse.

Very interesting. We actually are looking at viable approaches to get usable indoor maps for large buildings currently. We are building map based productivity software for workers in factories and offices. Maps are important for our use case but not the core thing we do. We actually partner with other companies for maps and positioning typically.

Indoor mapping is a hard problem because typical map providers tend to ignore private spaces, which is where maps are most needed. E.g. Apple, Google, Here, etc have great maps for a very limited set of public buildings but they don't really have solutions for indoors.

We know of a few companies specializing in this but it typically involves a lot of manual work and tends to have a high cost, require a lot of hand holding, and some very niche mapping skills. Also, a lot of indoor maps are intended for e.g. architects, contain a lot of clutter, or are generally not well suited for use on e.g. mobile screens.

It's an interesting problem to start tackling because we are getting kind of a critical mass of technology that will be able to start delivering pretty accurate positioning to commodity phones. There are all sorts of things happening in the industry related to that. It's a bit similar to when GPS became common on phones and it became a mass market in a few short years. Before that you had all sorts of niche stuff. After that you had ridiculous amounts of user generated content, open street maps, foursquare, etc.

Hi! If you're looking for a developer friendly indoor mapping suite (both for editing and viewing/navigating), check us out - www.mappedin.com

Totally agree that it's an overlooked problem and we've been quietly working to solve it for years. Early on we realized the key was to build "everyman" mapping tools that facility managers can use themselves to keep data up to date.

We're increasingly focused on our developer-facing tools and I'd love any feedback if you end up taking a look! (https://www.mappedin.com/mapping/sdks/)

BTW the demo link in the sample app does not work:


Thanks! fixed

We are in contact with mappedin but thanks for the tip!

Hey, producing indoor maps of private spaces is an area I've been working in for the past few years and I know exactly what you're dealing with.

I'd be happy to talk more about this if you're interested - email is in my bio.

Sounds like your company and the company I work for operate in similar areas. We focus on the indoor routing and positioning aspects, but are quite constrained by the effort of converting a floor plan or PDF of a CAD into a map. May be worth us connecting on LinkedIn to see if our companies could mutually benefit each other. My profile is my username here, without the “i am”.

This is neat, do you have any future ideas of making this into a product?

This is definitely a thought I have considered. It could potentially ease lot of hassle related to 3D scanning of houses by real-estate companies. Very often, those scans require expensive equipment, professional expertise to use those equipment and considerable time and effort to capture the entire residence. Even then,you will find many holes in those scans and floating debris. With our work, one only needs a sketch of the floorplan and photos taken from that house. Plan2Scene will use data to infer appearance of unseen surfaces.

For instance, look at this sample 3d scanned house: https://matterport.com/en-gb/media/2486. Make sure to switch to the doll-house view from the toolbar. There are many holes in the scan.

hi yasith, fancy running into you here :p

This seems to be really well done.

Thank you.

Rent3D++ dataset is not available for download, there is only a Google form they want you to fill out and no automatic email is sent upon submission. It's great that the source code is available, but it kind of defeats the purpose if the data it relies is behind a wall or at someone's whim before you can access it.

Hi, This is because the Rent3D dataset we extend has a "non-commercial use only" term.

How do we get access to it? I filled out the form and haven't heard anything back.

I did got a delivery failure on one of the emails I sent out. Can you submit again just to be sure?

Here is the form: https://docs.google.com/forms/d/e/1FAIpQLSfl4muDFf0qktqtWhj6...

Sorry for the trouble.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact