I think it's worth taking a step back and just marveling at what's now possible to do with an off-the-shelf software.
Yeah, it's "just" 50 fps, but ... would you just look at it!
4 years ago there was an equally fascinating "Paris apartment" VR demo, but the movement was limited and the amount of work that went into appears to have been orders of magnitude more - https://www.benoitdereau.com/
Go back 10 years and seeing this back then would've been absolutely bananas.
Makes you wonder what we'll have in 10 years from now.
Off the shelf yes, but wow…. The requirements for Unreal are unreal.
Recent versions handle real-time global illumination (Lumen) and dynamically render geometry to insane levels of detail (nanite). Both of these are groundbreaking achievements. But having had to install this in a bunch of university computers I can tell you that it comes at a cost.
Interestingly, the most significant issue is the space these installations require. We have optimized our installation as much as we can, but we nonetheless had to introduce systems whereby local storage was wiped clean every two weeks.
Our lab computors have 500gb of space. Each time a new profile is activated by a student, that eats up around 5 to 10 gig. In addition… Video editing eats up storage space like there is no tomorrow (the Cache requirements of Resolve are another issue). Nanite supports scene with very dance meshes (downloaded directly from Unreals online store). These add up quickly in terms of file size, especially as each comes in several versions, with multiple levels of detail.
Ideally we would have liked to install more than one version of Unreal. Unreal updates regularly and files created with each version cannot be guaranteed to be opened by subsequent versions. However, this was one saving we made.
There are sample files which are actually very instructive but which we did not install.
All of these issues compounded and towards the end of sem we were having a tough time.
There's your problem. Development and content creation machines need far more disk than this. It doesn't have to be the OS drive; a data drive or two is usually fine, but 500GB in total is far too little.
Exactly. I don’t understand why universities don’t factor in human costs. If it requires manual intervention every 2 weeks, investing $100 should be a no brainer.
I couldn't find the exact video, but Digital Foundry did a great breakdown on the performance bottlenecks in Unreal 5. They showed how disk speed directly influenced asset streaming, and it was pretty interesting to watch. Their HDD-speed drive choked constantly when running the UE5 demo, but a SATA SSD had enough bandwidth to handle the assets.
So, it's kinda a weird spot to be. Not every scene was streaming 250mb/s of assets, but when it did the worse drives took a notable hit to performance.
Depends on the system, AFAIK. Modern consoles have so much disk bandwidth that compression is completely unnecessary, but PS5 and Xbox both have APIs for streaming and decompressing compressed assets. I haven't seen any titles using them, and it's doubtful that we'll reach a point where it's required with modern systems.
That being said, compression would be an interesting avenue to explore for SD cards and SATA drives. The performance add can be pretty marginal, but sometimes that's just what these drives need.
Yep, unreal is surprisingly efficient on the graphics side of things, but it also wants 70-something GB just to install and that's before it starts setting up the graphics cache which adds another 40 GB iirc.
And thanks to inflation, while $90 used to be like six to eight bags of groceries not that long ago, now it's only two or three. So that's not like you can buy much else with that $90.
True. But the recent releases are a little better. We looked into using Unreal at school a few years ago but rejected it. This year was the first time we felt that our lab machines were equal to the task.
I think it will be commonplace to virtually walk around in movie scenes turned into 3d scenery by AI by the end of next year. I can see this turning into a real industry. Who wouldn't like to wander, fly and swim around Pandora, the world of Avatar?
It will also lead to questions about the copyright of the resulting models, probably somewhat dependant on how automated this process is.
> It will also lead to questions about the copyright of the resulting models
I'm sure there will be no questions there.
The questions about the copyright of models like DALL-E, Stable Diffusion, etc. exist solely because it infringes on the rights of thousands of random artists all around the world. There is no one single entity with enough power to clearly oppose it so people making the models can get away with "questionable copyright", "legal gray area", etc. kind of talk.
Try the same with a model trained on a single movie created by a multinational with infinite money and all this will be resolved and defined real quick and strongly worded cease & desist letters sent out by end of work day.
> The questions about the copyright of models like DALL-E, Stable Diffusion, etc. exist solely because it infringes on the rights of thousands of random artists all around the world
How does it infringe? Are your memories of a movie copyright infringement?
I do enjoy that video, but also, despite my deep cynicism about copyright, I can't help but think this is not in any risk of happening. If you trace the economics it makes no sense. Why would a creator make a thing, then sell you the experience of that thing, then be upset that you have memories of the experience of that thing? If they do try to claw it back in an era where memory editation is possible, why would anybody shell out any money to have an experience if they won't be allowed to remember it tomorrow? Why would I use copyright law to prevent word-of-mouth marketing, the most powerful form of marketing there is?
The point of copyright in general is to protect the economic value for the producer. Being so grabby about "IP rights" that they literally claw back the experience of consuming it means that the economic value of the IP drops to zero. It only has value in the context of being consumed, remembered, etc., and where the person purchasing the experience has confidence that such a thing will be allowed.
I suppose hypothetical future simulated humans could be shaped and molded until they give up some real value for an "experience" they won't be allowed to remember tomorrow, but by then they have passed so far beyond what is "human" that I have no further guesses what they may act like. (That's the true meaning of the "singularity" term, not any particular future, but the point at which our predictions are meaningless.)
If I am an artist and I go to art school, I'm going to spend years studying the masters to learn their techniques and style. If I then produce an impressionist portrait on canvass, informed and influenced by the years of study of existing works.... am I violating copyright?
If you're creating new works inspired by copyrighted works, no that's not (or should not be) copyright infringement. If you're creating exact reproductions of copyrighted works (e.g. The Gates of Argonath from Peter Jackson's adaptation of Lord of the Rings), just in a different format (3d instead of 2d) then yeah, that probably is.
Right. Most of the conversations about IP on this site seem to rely on extremely abstract reasoning or idealist thinking about what constitutes "property" or what a "copy" is. My understanding is that judges are not so easily misled. If you are selling a "replica statue of two kings" online that is clearly intended to resemble the Gates of Argonath, you are setting yourself up for a lawsuit. It doesn't matter if you built it from your memory of the LOTR films, used an AI to generate a bunch of similar looking statues and picked the ones most similar to the films, or what have you. In practice any judge will see that you are attempting to make money off of the LOTR IP. (Note: not legal advice.)
Note that this doesn't mean that Stable Diffusion (and friends) are copyright infringing just because they're trained on copyrighted material. My brain doesn't infringe the LOTR copyright because of my memories of the films. If I turn my experience of fantasy epics into a new novel with a different storyline than LOTR, that's not copyright infringement. That's creativity. But those same memories can likewise be used to make works that are infringing. The question of infringement isn't in the creative act, it's in the artifact. I think it's plausible (though not certain of course) that IP questions will be settled for AIs in the same way: it's not infringement to train your AI on copyrighted material. It is infringement to use your AI to generate works that a reasonable person would conclude are intended to replicate or imitate copyrighted material.
"My brain doesn't infringe the LOTR copyright because of my memories of the films. "
I guess you can pirate Lord of the Rings on bitorrent all you want then? Cause computers are just a bunch of brains remembering things?
This is not a winning legal argument. However this is resolved, no court is going to treat an intentionally designed piece of computer software as a "brain". Or the machine storage of information as equivilent to a human being's "memory".
I didn't say that? The act of piracy is an act of copyright infringement. The act of watching a legally owned copy of LOTR until my brain remembers almost every single detail down to the smallest element is not copyright infringement, obviously. I don't see how piracy is relevant here at all.
Obviously if OpenAI or other AI researchers are committing media piracy as part of their research, that is something they're legally liable for. It doesn't mean that their AI "remembering" it is inherently infringing.
My point is the brain metaphor does not work. The copyright act does not regulate "brains" but it does regulate software so what your brain remembers or does not remember is irrelevant.
I predict A.I. trained on LOTR will be found to be infringing because it's a derivative work unless the court determines that use of the original work constitutes a fair use.
My general point in my intentionally silly brain metaphor is brain metaphors are inapplicable and unconvincing.
> I predict A.I. trained on LOTR will be found to be infringing because it's a derivative work unless the court determines that use of the original work constitutes a fair use.
Okay, but that's just a prediction, not an argument. I don't agree with the prediction. I think it makes some sense to think about the AI training data on analogy with memories, because they contain impressions of a thing without being able to reproduce it exactly. They can both be the source of both real creativity as well as the source of infringing works.
I don't see an argument for considering the training data produced by AI training a derivative work, at least not in any copyrightable sense. ChatGPT is "software"; the training data that it operates on is not "software" in the strictest sense and exists in an uncomfortable middle space at present with respect to copyright law. That's what the memory analogy is supposed to show: just like human memories aren't subject to copyright, AI training data that is similar in structure and capabilities to human memories is likewise not subject to copyright. More precisely, my point is that that's one entirely reasonable way courts could rule: AI training data represents "fair use" of its sources (provided they are legally obtained of course), but the products of an AI trained on those sources might not be.
> it's not infringement to train your AI on copyrighted material.
If that's the case, it should probably should be codified in law. Copyright has a lot of exceptions, and if the existence of AI models trained with copyrighted data are in public interest then it could be an exception too, but this should not be up to the courts.
No, because it took years of study and producing a new portrait will take weeks. If you were able to study a master's entire portfolio in a matter of hours and then instantaneously generate 10,000 portraits in their distinct style, yes that would be unethical. I don't know enough about copyright to say if it would be illegal and I don't think that's a particularly interesting topic.
You just described some pretty amazing technology that can quite possibly move the human race forward and I’m surprised your first thought is that it is unethical. New technology often displaces old methodologies by disruption. It is not unethical to disrupt, but it may be unethical to let those people be disrupted without compensation, which is where I feel a lot of the anti-technology sentiment comes from.
As a society we ask people to branch out into various expertise… through no fault of their own maybe someone’s expertise is randomly obsoleted by a new technology. Just as copyright laws incentivize new creations, we as a society should incentivize people to embrace new disrupting technologies by safety-netting those displaced by them since it’s a gamble more or less of who is next to be displaced.
So your primary complaint here is the amount of time it took? Why shouldn't a bike be illegal as it allows you to go places in weeks that used to take years?
I’d like to add that all creative work stands on the shoulders of predecessors by being trained and based on (even if indirectly influenced) previous works. This really is unavoidable on a human level because we can only do what we see and experience.
> people making the models can get away with "questionable copyright", "legal gray area", etc. kind of talk.
Where are they saying that? Nobody thinks it's a gray area, they think it's explicitly legal and they're probably right too. Claiming it's a copyright violation is actually a rather weak case artists are just saying is obvious.
Microsoft has licensed DALL-E. They're not venturing into legal gray areas.
The argument I heard most often goes along the lines of: "people also learn from a lot of examples and other artists, and that is not copyrighted".
It is an interesting argument, but AI is not people and the argument could easily be the opposite (not taking sides here). They might think it is legal, but it is far from clear, IMHO.
IANAL, but it will be interesting to see if this is enforceable and by whom.
The day after movie-to-VR becomes easy to do, movie owners will monitize it, which will end the current free-for-all. Want to walk around The Shire in VR? You better have paid for the Hobbits Online subscription package.
> The day after movie-to-VR becomes easy to do, movie owners will monitize it, which will end the current free-for-all.
The movie making studios already own those 3d "digital assets" from their films. IDK if there is a real "free-for-all" right now, but the monetisation play has already been planned for.
The copyright protected material isn't in the model. This can be figured out easily because, if they did, the models are so much smaller than the input data that it'd be literal unobtanium for compression algorithms.
Copyright also has nothing to do with the tool.
What a piece depicts and how it is used are much more important than how a piece was created. If I paint a picture by hand of a particular scene from a Disney movie, that's clearly a derived work and may be subject to copyright protection (it depends, there could be a fair use argument in specific situations, so even "it's copyright!" is not a cut-and-dried argument). If I paint a picture of Hercules in the style of Disney, that's trademark violation. But I can paint Hercules in any other style and Disney can't say shit (though they certainly try).
This is an age-old argument. Some artists think they should have a monopoly on certain ideas. As a society, we've already said no, you only get to own your expression of your ideas.
And as ideas go, "like a trending artist on ArtStation" isn't a particularly strong one.
> There is no one single entity with enough power to clearly oppose it so people making the models can get away with "questionable copyright", "legal gray area", etc. kind of talk.
Never underestimate the power of a sufficiently motivated rich individual to create legal hell (see Gawker). There are many very rich artists. There just needs to be an obvious large enough irritant which will happen as these generative models start getting used beyond toy amusements. The class action lawsuit against Copilot was filed just as pricing models started.
This is going to be an interesting area to watch. Use in commercial projects at your own risk of exposure to liability.
The wrong and easy reaction to this is to limit these services to not include copyrighted content. The more right and really really difficult solution would be to give the original copyright holders partial rights to the output.
It really feels like the textile factory worker problem all over again. We can't let the big players be the benefactors every time.
> The more right and really really difficult solution would be to give the original copyright holders partial rights to the output.
That is impossible (assigning credit to training images from the output is arbitrary, not deterministic) and wouldn't help anyone (imagine Spotify except it pays you even less).
> It really feels like the textile factory worker problem all over again. We can't let the big players be the benefactors every time.
Industrialization is better for society because it produces more customer surplus. Even Marx thought the Luddites were wrong.
>Industrialization is better for society because it produces more customer surplus.
Also my point.
>Even Marx thought the Luddites were wrong.
Sure they are wrong in that they confuse what the problem is. If workers owned the factory they would be happy to automate the work. But their feeling of being left out has some validity because if they weren't the total value would improve.
The real issue is that the revenue brought back through automation doesn't benefit the worker. If it did, the automation would happen sooner because incentives would align better and the total value for society would thus likely improve faster.
I think it would be almost trivial to build in a “generate 30% different than” (the legal amount of difference before a work is considered different than the original). That could either sidestep the entire legal question, or throw Hollywood in to a blender since so many movies/shows have been made on the back of that difference.
I think it would be the other way around. A lot of movies(and tv shows) have games these days. You can already experience Star Wars in VR with “Star Wars Squadrons” and I’m sure we will start to see a lot more games with VR support in the future, many being based on Movies. MNCs are there to make money and they will just license it.
> I think it will be commonplace to virtually walk around in movie scenes turned into 3d scenery by AI by the end of next year.
I would take the opposite bet, I doubt most movies have complete enough 3D scene to be AI enhanced, let alone something good enough to walk around, you wouldn't model what you won't film after all and nobody wants to see hallucinated aliens in middle earth.
> Who wouldn't like to wander, fly and swim around Pandora, the world of Avatar
This is a good example of the fundamental limits of headset VR. Jumping into a lake will never feel real. Not by touch, obviously as you can’t get wet. Not by physical movement because you almost certainly can’t move downwards irl. Not by the weight of water around you or the buoyancy you expect to feel. Not by the rotation you would expect to have in water. Probably not by the stroke propeller movement or the correlation of your breath. And of course probably overly smooth movement to avoid motion sickness.
The difference between controlling a flying camera through water and believing you’re swimming is pretty big.
Exactly. We don't have virtual reality at all. We have virtual remote viewing with sound.
We are basically at the Eliza stage of VR with a bunch of people pretending Eliza is not that far from passing the Turing Test.
Once we have virtual skin sensation and virtual physical movement there is no intellectual debate about any of this. Just instruction tutorials about how to have the experience because no marketing needed.
Once we have this I can think of far more interesting things than swimming around in the Avatar movie.
You don't need practice for that, all the signs are in English and you just use Jorudan for route planning. Or Google Maps if you must, though it's gotten worse over time.
nb: you may need practice to navigate Shinjuku station
I don't know but then agree. The general concept sounds boring to me but I can see how this can be interesting for many people. For example as a Starwars fan someone must want to fly the same scenes in VR for a few times.
But I think the better is to train AI to generate scenes based on description such as "dogfight in P-51 with German ME-109 in WW2 for 15 mins" and after some computation the player gets to play this game in VR for 15 mins. The problem is how to make it possible at low cost as player will probably just pay a few bucks for such experience (but you can always resell the popular ones many times)
Do you feel the same way about real life locations? Exhibitions? Galleries?
Obviously nobody is saying "VR is the same as real life" but surely to some degree it's the content of the experience that determines whether something is interesting. Some locations would be fascinating to walk around even if there's technically "nothing to do".
I'm not much of a gamer so my interest in VR has always slanted towards towards the more passive experiences. In fact quite often don't want my enjoyment spoiled by having to complete tasks or solve puzzles just to get around.
Once that stops being exciting we could tell AI to create new worlds combining others such as mixing the Smurfs and Pocahontas worlds... I wonder what it could come up with /s
It will lead to questions - but those questions will be answered by the platform owners not the courts. The platform owners will just remove your models even in cases where a court would rule it fair use.
VR of real places is going to be huge - bigger than for imaginary places. My guess is that most VR worlds will be a combination.
Here's a question. What if you model the real world and an architect of a building. I know that copyright doesn't apply to buildings or other functional works. But again, this will be decided by the platform owners and not the courts.
Clearly there are good reasons to not have "platform owners" in the metaverse.
The Trek community created a pretty detailed model of the Enterprise-D, but took it down due to copyright concerns. It's still floating around if you know where to look but it's a real shame that development on it had to stop out of legal fears, well founded or not.
I can see this turning into a real industry. Who wouldn't like to wander, fly and swim around Pandora, the world of Avatar?
Me. It sounds boring. I couldn't give a damn about exploring someone else's fantasy world. This is a bit odd because I love open world video games. I guess I need a reason to spend time there.
I think ChatGPT showed exactly that we are not that far away. ChatGPT fine-tuned on the specific world could generate quests and make every NPC able to get interacted with
> It will also lead to questions about the copyright of the resulting models
I very much doubt it. Within the Movie business, the ownership of "digital assets" such as 3d models, who keeps what, and what they can legally do with it, is the subject of long and detailed contracts between the film studio, VFX houses, subcontractors etc. Ownership is well-covered, if not exactly "clear" if you're not versed in the law regarding it.
A third party using any technique (AI included) to re-create a look-alike of such an asset, and to monetise this, would surely attract the "eye of Sauron" of the big studios to turn it's withering gaze upon them, and to send out a well-armed detachment of lawyers.
Why by the end of next year? That seems optimistically short.
Why hadn't it happened this year, or last? Nanite is great, but IMO doesnt change the equation that dramatically.
The cost for a studio to manually decimate game quality versions have been small enough that of the will had been there, we could have been doing it for years.
Having loosely followed this space, I think I might actually have some input!
First off, you need a VR-ready environment. Not many companies are making these, so the only people equipped to roll this out is Valve, Meta, and possibly Unreal/Epic. Valve already did this alongside Half Life Alyx; they released certain map portions so you could freely walk around them with friends without playing the game. Meta is getting there, but the combination of low-power hardware and headset attrition doesn't leave them with much of an audience. There's also Epic, but they don't really do much in this space yet either.
So, obviously a paradigm shift is needed. And frankly, I think Universal Scene Description will be that paradigm shift. TL-DR: Nvidia came up with a new standard for 3D environments that interoperates game technology with digital effects. In theory, you could take your favorite Andor or Rings of Power scenes, drag their file into another app, and just start walking around in-scene. You may end up adjusting a few things vis-a-vis scale, but once it reaches adoption I can see it being pretty easy to work with and popular.
Maybe 'next year' is a little close, but the stars have definitely aligned already.
WebXR feels like a solid building-block to me. Adoption will be (and has been) shaky, but with Apple being forced to adopt third-party browser engines it's revival seems likely.
WASM is neat but extremely finnecky at the moment. I've seen cool stuff done with it, but I'm not convinced it's funny production ready yet.
WebGPU is sorta in the same boat as WASM, if a little more realistic of a bet. People will want hardware-accelerated browser experiences soon, and WebGPU will probably do most of the lifting there.
The author predicted that players will play a game where they will be dropped into their favorite movies, and will be given points for exactly mimicking the dialogue, inflection, and movements of the characters.
90 FPS for redrawing the image according to the motion of your head, but the actual rendering of the scene can be much slower. Motion reprojection is basically the only way to make GPU-intensive games like MSFS playable in VR on < $1000 GPUs.
Most VR headsets/runtimes support this but it only goes so far. The closer things are the more occluded sections will need to be in-painted. The farther the scene is the more it might as well be a skybox at infinite distance.
One thing that sticks out like sore thumb are those relatively low-res rock textures, especially close-up. Would be neat to see some procedural generation there.
That screenshot does not look rendered in game, and no idea what the resolution is - at least not by an Xbox One. If current games are indistinguishable from single-frame renders that take minutes if not hours, I'd say that's a massive win!
What stands out to me is the low resolution textures everywhere, obvious seams between terrain and assets (runtime virtual texturing resolves that), light seams which have odd seams, and the entire ambience being vaguely similar to how light acts, but not really - primarily the lack of bounce lighting affecting anything. With the move to HDR, the more binary lighting of previous games looks weird too now.
Nanite, especially foliage, is something you have to experience for yourself though. Foliage pop-in not being so stark and alarming is huge for me, and the rich lighting capabilities improves immersion massively.
Regarding assets, high polygon assets have been created for ages too, but were never actually brought in to games. You bake those details into your normal maps, but with Nanite you can just bring them into games, and get the full detail when the player is up close to them. When games can fully embrace Lumen and Nanite, the developer iteration improvement will be huge.
The quality of nanite heavily depends on your model. You can use photogrammetry scans or CAD exports directly, and nanite will make sure they'll look good. But if you just use existing videogame or fanmade assets (like here), the quality obviously suffers.
The technology at play is about handling scaling detail. You can now use very large and very detailed models so your scenes look good from close and afar. A single shot misses the point.
What makes that shot look good are just the post effects, eg. depth of field and god rays.
There's no accounting for taste, but another reason why the Unreal advancements are a big deal is improved productivity for artists. The geometry in that Xbox One screenshot was painstakingly optimized by some poor overworked artist, whereas this guy just threw together a couple billion polygons and UE5 didn't even flinch.
edit: not to mention the lighting in the Xbox One screenshot was also pre-baked and probably manually tweaked, compared to real lighting updated in realtime in UE5.
VR capable computers are quite cheap. Cheaper than a low spec apple macbook pro running m1. For 2-3k you can get a beast of a machine running everything on real high settings.
Right. You only need a card that's more powerful than the GTX 1060 for VR though, so the actual price-of-admission is more in the $250-400 price range.
The parent I'm responding to was interpreting "VR capable" as "top of the line" which is demonstrably false. You probably can't even get this scene in the first place, so it's kinda a moot point.
This is super cool. Performance doesn't really matter, as long as it can keep 45 fps, the experience would be good enough with motion reprojection turned on.
I sometimes fear I shouldn’t play driving games too much, so that I don’t get desensitised against those intrusive thoughts I sometimes get on the road. I wonder how irrational that actually is, because it feels like pretty much the same thing as your idea. I mean it doesn’t seem far fetched that a very realistic VR driving simulation with subtly “easier” physics might make you a worse driver in reality. More closely related to the fear of heights, we also jump from greater heights in video games than we would in real life, and much more frequently. I practically never have a reason to jump down from anywhere, so I could imagine making some wrong estimations when, after years of doing it in VR, I’m somehow faced with a real-life situation.
If you haven't played it, I highly recommend Richie's Plank Experience [0].
The main focus of the game is you take an elevator to the top of a building, there is a plank an you jump off it.
I've found depending on the age of the participant the experience ranges from light fun to harrowing encounter with our own fears of death. I've seen people that, if it's their first time in VR, can take nearly an hour to finally do it.
Yeah, it's "just" 50 fps, but ... would you just look at it!
4 years ago there was an equally fascinating "Paris apartment" VR demo, but the movement was limited and the amount of work that went into appears to have been orders of magnitude more - https://www.benoitdereau.com/
Go back 10 years and seeing this back then would've been absolutely bananas.
Makes you wonder what we'll have in 10 years from now.