
PIFuHD: High-Resolution 3D Human Digitization - jonbaer
https://shunsukesaito.github.io/PIFuHD/
======
c22
I'm glad I clicked this because it's awesome, but it wasn't at all what I was
expecting. Maybe change the title to _PIFuHD: High-Resolution 3D Human
Digitization_?

------
TaylorAlexander
So I am building a farm robot, and for various reasons I think having a 3D
reconstruction of the plants would be useful. I’m hoping that this technique
might be useful for reconstructing models of plants from a small number of
images. The model doesn’t need to be perfect, just close. I think this will
help in doing reinforcement learning on simulated data, among other things.

Great research coming out of Facebook Reality Labs!

------
positivity10
More info on this blog: [https://ai.facebook.com/blog/facebook-research-at-
cvpr-2020/](https://ai.facebook.com/blog/facebook-research-at-cvpr-2020/)

------
makeworld
This is amazing. Does anyone have any insight on what makes the back view
possible?

~~~
swframe2
It requires a video and the person needs to rotate. It doesn't imagine the
back view. It appears to be doing optical flow and photogrammetry.

~~~
TaylorAlexander
It is not doing photogrammetry and it works on a single image. It has learned
what people look like (and training may have used photogrammetry in some way)
and it can imagine a good 3D model from even a single image. The last line of
the abstract:

“We demonstrate that our approach significantly outperforms existing state-of-
the-art techniques on single image human shape reconstruction by fully
leveraging 1k-resolution input images.”

------
adhoc32
The pretrained model is unavailable.

edit: downloading now, maybe the server was overloaded.

~~~
swframe2
If you try it your own video, make sure the person is rotating so the model
see all sides.

~~~
dorkwood
I'd recommend watching the "1 minute presentation" video linked at the top of
the page. It mentions generating the model from a single image.

[https://youtu.be/-1XYTmm8HhE](https://youtu.be/-1XYTmm8HhE)

------
monkeydust
Application - better fitting of clothes brought online? Consumer happy,
company returns rate drops. Seems like an obvious one no?

------
coronadisaster
High resolution doesn't mean high accuracy in this case.

~~~
swframe2
Similar to how the early GANs for generating faces were low res. I would
expect follow up work to get much better fairly quickly.

~~~
coronadisaster
You can't create an accurate 3D model from what you can't see. It can be high
resolution though...

~~~
dogma1138
It’s not about being “accurate” as in representing exactly what it can’t see
but rather be “authentic”, since you don’t care about things like patterns or
logos clothes aren’t that hard to figure out and so are body shapes.

Backpacks, and other attachments that are on the back or occluded ofc can’t be
predicted or fitted with-in the model to represent their real life
counterparts well not unless you train the model on every possible backpack.

That said training this model on people’s photographs and their 3D scans and
then possibly on a dataset of clothes and accessories from big brands might
actually allow you to much more accurately predict the unseen parts of the
image by not only filling the gaps based on predictions but by actually
specifically matching the items of clothing and accessories to draw additional
information that way.

Most fashion brands already have detailed images of their products from
multiple angles and even have 3D views of many items.

Combine that dataset with this model and you’ll probably be able to increase
the accuracy of these models considerably.

------
jcims
Tweet of Obama reconstructed as Larry Bird in 3...2...

