
AI for generative design: Plain text to 3D Designs - tf_12
https://blog.insightdatascience.com/ai-for-3d-generative-design-17503d0b3943
======
airstrike
I was _literally_ thinking about this in bed before falling asleep last night,
but I was imagining scenes rather than objects.

I'm in no way a ML developer (I'm not even a developer), but I was wondering
how hard it would be to generate an apartment / house layout from a (somewhat
detailed) description.

I'm sure the first use that springs to mind is for architects and real estate
developers / agents / brokers, but one of my pet-projects-I'll-never-really-do
is to recreate memories in 3D from descriptions and old photos. Imagine being
able to relive your childhood memories! Just throwing it out there in case one
of you smart folks want to take that and run with it...

~~~
tarsiel
That exists! The graphics lab at my old university published a paper on this
exact thing:
[https://manyili12345.github.io/Publication/2018/T2S/index.ht...](https://manyili12345.github.io/Publication/2018/T2S/index.html)

~~~
airstrike
Thanks! That is really cool and I appreciate you sharing it!

FYI I tried clicking on "Slides" but got a 404 error, so thought I'd flag in
case you know the authors.

------
rkagerer
Tea, Earl Grey, Hot!

I really want to try the live demo and make some melty furniture of my own,
but I got this error:

    
    
        ModuleNotFoundError: No module named 'plotly'

~~~
cheschire
Honestly surprised I had to scroll this far to find a Star Trek reference.

The scene where the crew describes their mass dream into a horror dentist
chair is forever burned into my brain as a technology I want in my lifetime.

~~~
starstorms
I loved that scene! It was actually a direct inspiration for my project here.
For anyone looking it was Star Trek TNG S06E05 (Schisms)

------
Lichtso
Really cool idea, I wonder why nobody tried this earlier.

However, the main problem I have with this approach is the voxels. They model
the geometry and only the geometry. The far more important aspect of topology
is left out. Thus, the results suffer the same problems like 3D scanning /
photogrammetry does: It is practically unusable as it can not even be
textured, yet alone animated or used in fabrication (except for Lego-models I
guess). So point 5 of the future work is the biggest one in my opinion.

~~~
Jack000
Voxels are directly analogous to 2d pixels, which means they benefit from the
myriad of CNN based techniques developed over the last few years (GAN in
particular)

3d manifolds are sparse, and more analogous to 2d vector graphics. There are
approaches for dealing with this type of data (eg. spectral graph NN) but they
don't work as well for 3d topology as CNNs do for dense pixel data, as far as
I know.

In the near term, it might be better to explore approaches that use voxels,
then generate the topology heuristically.

------
sgt101
There were a lot of evolutionary design algorithms in the 1990's with similar
results. This is fun - but I don't think it is going to go anywhere.

~~~
SiempreViernes
yeah, since all they wouls show were rather cruddy voxel shapes and didn't
start the text with "It actually includes the physics from the start" I didn't
bother read the whole thing.

But it very much looks like a very advanced way to produce something slightly
less useful than a sketch on a napkin, since the latter was made with an
actual understanding of what the description is supposed to mean.

------
heyitsguay
Amazing! I do work in 3D computer vision and I've really wanted to get into 3D
generative asset design as a side project at least. If the author or anyone
else is interested in collaborating, I'd be down to talk and shop something.

~~~
gaogao
Sure! I've been wanting to incorporate this exact thing for a "And let there
be light simulator!"

Essentially, people in VR would be able to point and speak to create objects
in a shared space.

~~~
heyitsguay
Cool! I've got some Unity VR experience, though i don't know much about
incorporating ML pipelines into it. I'm guessing there's resources out there?
So basically you'd want speech -> text -> object?

~~~
JabavuAdams
I've got some of this working in Python, outside of Unity. Last time I looked
at integrating with Unity there seemed to be some Google API requirements that
were hard/annoying to meet in my few hours of digging. Should take another
look. All the VR HMDs have mics.

------
mendeza
Great work! I wonder if using SDF as a 3d model representation would result in
higher resolution model generation (rather than voxels), some recent work in
that area here:
[https://github.com/marian42/shapegan](https://github.com/marian42/shapegan)
Adversarial Generation of Continuous Implicit Shape Representations :
[https://arxiv.org/abs/2002.00349](https://arxiv.org/abs/2002.00349)

~~~
starstorms
I hadn't seen that paper, looks really interesting! It definitely would
improve the resolution but I think fundamentally it still runs into the issues
I had with voxels where it mostly just learns low level correlations and not
high level topology.

------
afpx
Very cool. But, seems like a lot of work to order to conclude that tables are
rectangular flat surfaces and cups have handles and hold liquids.

------
tlack
Here's the github repo for this interesting work:
[https://github.com/starstorms9/shape](https://github.com/starstorms9/shape)

(I couldn't find it in the linked article, might have missed it)

------
raidicy
This is fantastic. Much better than my idea of just using text recognition to
search a database of models. I hope this research continues so in the years to
come I can slap on a VR headset and generate scenes like the holodeck.

------
whichquestion
It would be interesting to train the 3D Designer on more classes. Then have
the text generated from something like AI Dungeon 2 feed into it and see what
3D designs it generates.

------
Agebor
I've been thinking along similar lines on high level - this approach will
likely be very general and can be used to create different kinds of media, as
well as potentially behaviours in a later stage.

[https://metapresent.org/creation-engine](https://metapresent.org/creation-
engine)

Will be interesting to base it on a decentralised open platform that could be
"built-in" in the Internet.

------
bufferoverflow

        Importing libraries...
        ModuleNotFoundError: No module named 'plotly'
        Traceback:
        File "/usr/local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 314, in _run_script
            exec(code, module.__dict__)
        File "/app/shape/streamlit_app.py", line 15, in <module>
            import plotly

------
noajshu
Is anyone else getting ModuleNotFoundError: No module named 'plotly'?

~~~
ebg13
Yes

------
hyperion2010
Fun. Here are some sentence that come to mind.

Flat surface with bevelled edge and 4 Ionic columns as legs that fade into
nymphs at the base.

Table with caryatid legs.

Crescent wrench that doubles as a corkscrew.

Stapleremover.

------
abhinai
This work is inspiring and ground breaking. I can see a lot of text
description => X solutions coming up in the future. Absolutely awesome work!

------
vsskanth
This will be extremely useful in product search.

"sofa with 3 cushions and a round arm" should generate a model and match with
similar looking products

------
systemvoltage
Design is not just about how a thing looks or what kind of shape it has, but
the intelligence that goes into how it works.

------
ExSoax
I thinked about it so many times... Really coil. I hope someday will be
possible convert a sentence to a full scene

~~~
qayxc
That's possible even today, kind of. [1] [2]

[1] [https://t2i.cvalenzuelab.com](https://t2i.cvalenzuelab.com)

[2] [https://towardsdatascience.com/text-to-
image-a3b201b003ae](https://towardsdatascience.com/text-to-image-a3b201b003ae)

------
britmob
Can’t wait till this becomes applicable to something like AIDungeon!

------
camillovisini
Stunning work, congratulations. That was a pleasure to read.

------
windyfly
The ultimate tool for story-telling.

------
WFHRenaissance
Woot for Insight Data Science!

------
zarmin
This is amazing.

------
mikehollinger
Nifty!

