
UnrealCV - sytelus
https://unrealcv.github.io/
======
Endy
Aww, I don't get to create a walkthrough of my resume in Unreal? I'd love to
let recruiters shoot my bosses.

(nevermind that the last Unreal I played was in 1998.)

~~~
soverance
I could actually make this, and it sounds like a cool idea. But I'm wary about
putting the time into it.

My concern is how many recruiters would actually download and run a 1 or 2 GB
application simply to see my resume, when they instead could visit my website
and watch some quick videos.

My guess? None.

~~~
TrevorJ
there's an experimental HTML5 feature so you could ostensibly embed it online.

------
AndrewKemendo
This is a great example of how we (collectively) are using virtual worlds as
"dreams" for Deep learning.

In the Deep Vision world we as a group are trying to segment, classify and
reinforce our NN training on labeled real world data. The challenge is, it's a
very manual process to label data - specifically images. The more that we can
do inside the computer, for example automatically labeling pixels inside an
image, without having to acquire and label real world data (or making it
easier to do with real world data) the easier and faster training becomes that
can be applied to real world use cases.

The trick is making the virtual world match the real world as closely as
possible so that the nets we make are accurate representations of real world
scenarios.

~~~
bonoboTP
It's not just for deep learning, it's for any computer vision research.

Synthetic datasets have been used in the past too, but this makes it much more
accessible.

~~~
AndrewKemendo
Indeed...

However CV is becoming more and more Deep Net based.

------
johnhenry
I was hoping that this would be someone's really cool resume, but this is
interesting, nonetheless.

~~~
AtheistOfFail
Was actually hoping for an alternative to OpenCV that wasn't created in the
90s.

~~~
mynewtb
What problems do you see with opencv?

~~~
AtheistOfFail
Nothing, actually, it's a quality library, just old.

I can think of a few algorithms that have changed or been created since the
1990s so I have to assume there's something that can be done to optimize it.

There's also a severe lack of "in-browser" CV solutions (which could be fun),
instead opting for binding OpenCV in the server-side.

------
djhworld
Can someone explain what this is for?

Is it so that computer vision researchers can create scenes in UE4 that can
then be fed into their application?

~~~
dharma1
You could use it to say, auto create large labeled data sets of images and
their corresponding depth maps. Or generate large labeled data sets of objects
and their segmentation masks. Or for training an autonomous agent (say self
driving car) in a simulation environment

~~~
qiuwch
Exactly. That's the goal for this project.

~~~
dharma1
Many thanks for your work, will be putting it to use! Please consider
integrating it with [https://gym.openai.com/](https://gym.openai.com/) for
easy reinforcement learning tasks.

Some similar efforts (though not with open source engines)

[http://download.visinf.tu-
darmstadt.de/data/from_games/](http://download.visinf.tu-
darmstadt.de/data/from_games/)

[https://arxiv.org/abs/1608.01745](https://arxiv.org/abs/1608.01745)

------
bashtian
I thought about something like this for autonomous cars. The game engine could
simulate all the sensors in a car. Then it would be possible to use this as
training data. Would be fun to create a racing game out of it to let different
algorithm compete against each other. Does this already exist? Google should
have something like this, but probably not freely available.

~~~
gondolgames
AdasWorks is doing this, check the videos 'Data Generation Tool - Annotation'
and 'Real-time Data Simulator' towards the bottom of this article (text is in
Hungarian):
[http://index.hu/tech/2016/09/13/obudan_epul_az_autoipar_noki...](http://index.hu/tech/2016/09/13/obudan_epul_az_autoipar_nokiaja/)

------
highd
This is a brilliant idea, IMO, and a real game changer (in principle, not
necessarily tied to the implementation). Without this, CV problems are largely
only limited by how much data you can get. If you can simulate arbitrary scene
configurations it's now possible to solve almost any non-human CV problem,
even ones with sparse data.

For example, in security applications you have basically zero data on the
sorts of anomalous events you want to trigger on. If you can simulate example
cases of those things that means you can actually start applying supervised
machine learning.

~~~
qiuwch
Many researchers have done interesting work using synthetic data (a lot of
them are mentioned in this post). This tool just tries to make the creation of
virtual world (which can be used by computer vision) easier.

------
malux85
Does anybody know if there's a plugin for unreal, like this, but supports
creating geometry inside the game?

If not, is there a way of generating levels with code?

~~~
TrevorJ
I don't think the plugin needs to directly support those features, as they are
already in the engine natively.

The engine supports generating meshes dynamically at runtime:
[https://wiki.unrealengine.com/Procedural_Mesh_Generation](https://wiki.unrealengine.com/Procedural_Mesh_Generation)

It is also fairly easy to generate random or pseudo random levels using
blueprint (Unreal's visual scripting API) or C++

If you have any other questions I'd be happy to help.

~~~
malux85
Thanks for your response, great info.

The reason I ask is that I have a large dataset that I would like to
visualize.

The ideal situation would be: I could use python to create a giant flat plane,
and then iterate over the dataset to generate the level (which initially would
just be quads on the 2d plane).

The dataset would be too large to do in-engine in realtime, and the quads are
static - I would like to generate the level as a 1 off process, and then have
Unreal do any optimisations that it can by "compiling" the level (e.g. frustum
culling, quad tree calculation etc) I might have some of those optimizations
wrong since my knowledge of engines is still stuck in the Quake 1 era, but you
get the gist :)

Do you think this is possible? If this tech doesn't exist, I would pay someone
to develop it for me....

~~~
TrevorJ
anything is possible :)

How many total datapoints are you trying to represent at once, ballpark? My
first instinct is that it would probably be most efficient to feed the data
into Unreal and build the physical representation there VS importing a large
monolithic 3D model, because then you could make use of instancing, distance
culling, etc. This would also mean you could animate the objects separately as
well, and manipulate scale, color and the like in realtime.

Again, probably depends heavily on the specifics. In _general_ though, modern
gaming GPU's can handle just an insane number of polygons so either way, you
should be able to represent a ton of datapoints.

As far as getting data into the engine, it supports .csv importing where you
can define your own row types etc:
[https://docs.unrealengine.com/latest/INT/Gameplay/DataDriven...](https://docs.unrealengine.com/latest/INT/Gameplay/DataDriven/)

I believe there's also a SQL lite support module for the engine as well, if
you want to deal with really massive sets of data, though the documentation
there is sparse.

My contact info is in my profile, feel free to hit me up if you have any other
questions, happy to help.

------
zitterbewegung
This would be cool if openai would participate in it.

------
ktta
Anyone know of something similar for Unity? Unity has a bit of a steep
learning curve of you want to do stuff like importing external data and
working with outside programs

~~~
qiuwch
The CVC lab and Virtual KITTI use Unity in their projects. They might have
some internal tools. The advantage of UE4 is open-source, so that we can fully
understand what the engine is doing.

------
dharma1
Love it. Anybody use unreal engine with OpenAI gym yet?

------
zawarudo
I, like many others most likely, thought this was gonna be a cool way to
present your resume, or someone with a demo of theirs in Unreal Engine...
Still, I was not disappointed by the plugin.

------
kevinmgranger
I thought this was going to be Unreal Curriculum Vitae, and it was going to be
a playable resumé. I'm slightly disappointed.

