They're spending about 14ms on inferencing and 3ms on everything else, which technically is about 60fps, but in any real world application you're probably not going to be able to blow almost your entire GPU compute budget on shading a few pieces of cloth.
To their point, 30 fps is commonly acceptable, in which case you're spending 7-14ms on this, and 26-19ms on everything else. You can also use frame gen cheat 30 fps back up to 60 if one is so inclined.
Interesting to hear that, consumer market has been well past 30 fps for years, to my knowledge. For example:
- Phones are widely at 120 fps as of a couple years ago.
- Gamers have been targeting 240 fps for years and years, since 2018?
- On the low end, ex. Steam Deck requires you manually opt-in if you're happy with less than 60.
I'm curious for some groups you're aware of that commonly accept 30. (note: not a trick question, ex. we know movies are 24 fps, and professional 3D renderers are happy with not-real-time, but we're thinking about 3D rendering for games)
It may be the case that high end PCs with the latest expensive graphics cards can run modern games like cyberpunk at 240fps using things like DLSS, but console games regularly run at 30fps in “quality” modes and 60fps in “performance” modes. 30fps is absolutely an acceptable frame rate for the most people. If we didn’t have Digital Foundry and the like nitpicking dropped frames and counting pixels for dynamic resolution, I think even less people would care about high framerates, as long as the game is fun.
PS5 came out 4 years ago (November 2020) and has 4 games at 30 fps. Of 3140.[1]
PS5 and Series X both have an infamous problem with ports from older generations coming over at 30fps, because they were render-locked to that in the previous console generation, and no upgrade work was done during the port.
Beyond that, 30 fps comes about via opting into it via choosing raytracing at 4K.
It's a massive reach to describe 30 fps as commonly accepted. And that's without considering the context: it's being used to argue for the usefulness of a cloth simulation neural network that takes 18 ms to render on a 3090, not 2020 consoles.
Moreover, those screenshots aren't showing a game with some clothing simulation. They're showing a nearly-empty render except with some cloth simulation.
I think steelmanning is good and important for conversation, but its hard to swallow here. Blithe while taking too many liberties that are likely to shade rather than shed light.
> It's a massive reach to describe 30 fps as commonly accepted
Toy Story was rendered at 24 FPS and I don't know a single person who refused to accept it as-is. In fact, I hear more people complain when cinematics aren't run at a filmic framerate.
The end-to-end latency is horrible with 30fps. Move your mouse and you can feel the lag. In theory you can over-render the sides of the screen and move the rendered image around with the mouse but almost no one does that.
> - Phones are widely at 120 fps as of a couple years ago.
They're only briefly at 120fps during important things like full screen scrolling. Phones are battery powered and passively cooled, so rendering at 120fps would be very wasteful even if you can do it.
The important thing about 120fps is that a lot of things are factors of it; you can't play a 24fps movie on a 60fps screen.
All fair points. Phones have 120fps displays, but obviously can't run much at that rate. Some web pages, basic note taking, etc. But pretty much anything 3d is 30-60fps.
Consoles, hand helds usually aim for 30+
Real time rendering, video streaming etc, are 24-30.
If the nit is my phrasing, let me rephrase, the point is that it's usable in an interactive/real-time application. It's perhaps at the lower bar, but 30 fps is far from a rarity. Though not always ideal, it's certainly acceptable.
30 fps should not be acceptable given a 4090. Remember, most people have cards that are barely 15% the performance of a 4090, or less. The fact that almost all modern titles target the 4090 is a huge problem; only a very small number like Roblox actually understand how to optimize.
As a professional I usually use a 3090. I'm sure if I worked in fields that heavily relied on this sort of thing I'd have a 4090 or two no problem, but it's not that safe of a base assumption to make.
perhaps if someone builds a liquid NN version, can reduce the size dramatically. but that has it's own problems, e.g., operations in that network class less supported.
> works by encoding the patterns and parameters of fabrics as a small latent vector, which can later be interpreted by a decoder to produce realistic representations of various fabrics.
> Woven fabric patterns […] tend to be regular and repetitive. Chen and his colleagues set out to leverage their repetitive nature
Is this “just” using the NN to approximately compress and decompress things? That doesn’t seem to explain the performance gains, unless some work can be done in the reduced space?
Everything NN-related is compress and decompress if you squint hard enough.
You are trying to map inputs to outputs in an efficient way space-wise or time-wise. The NN is doing something which allows it to skip complex calculations at the expense of fidelity in a way which is not far from a more advanced look up table.
The fact that what it does is so opaque remains the main flaw of it.
I believe that results like these show that our algorithms are woefully inadequate, and something much faster has to exist, if a NN can do so much better with all the overhead.
This isn't efficient at all. It requires a 3090 to get 60fps - and presumably it requires all the GPU's power to do that for the simple scene in the article: a couple of sheets and cushions on a sofa.
Possibly it's RAM rather than compute limited but still. You couldn't use that in a game. Existing techniques like PBD or verlet integration would be far more efficient.
Maybe on future devices with separate neural processor units this could be useful. But even then I bet those same chips could run standard real-time cloth algorithms really really fast too.
Edit: I downloaded their video and there's no movement at all, only still images. Maybe I misunderstood the purpose of the technique. But I re-read the article and it certainly seems like they're claiming to do full simulation, not just static scenes.
That scene with the sofa isn't even getting 60fps! It kinda feels like a disingenuous article after watching the video.
I don't see how something like PBD helps at all with rendering.. are you thinking that this work relates somehow to physics simulation? Considering that the result is pretty much just a shader, I imagine it works just as well on animated scenes as static ones.
Well, technically cloth simulation and cloth rendering are separate. But in real-time rendering the terms are often (generally, even) used interchangeably - it's rare that you'd just want static cloth without movement - so at the very least I'd expect a sentence or two clarifying that they don't do any movement simulation here, it's a pure light transport technique.
Light/material calculation in shaders requires exponentiation, which is a costly operation. If a clever lookup table can get around that, it sure can be faster.
Yeah it’s kind of like a FPGA implementation: it proves the concept but is by definition not optimized for efficiency. There must be a way to go from the weights to an algorithm.
The classifier still has to learn a model, which is a lossy, compressed representation of the world. Classifiers can be very lossy.
The outputs from Google's Deepdream are the kind of things that a classifier would "uncompress" to.
There are plenty of compression methods that have zero generalization. There are also methods of generalization that do not compress at all. It seems to me that they are orthogonal concerns, even if they may make sense to use together.
> As for NeuMIP, it can produce results with a higher quality than ours in terms of MSE, at the cost of a higher storage and per-material training.
> Furthermore, our network can be applied to three typical types of woven fabrics once trained, but NeuMIP have to be trained per material which takes a long time
> Specifically, we introduce a simple encoder-decoder structure, where the encoder compresses the fabric pattern and other parameters into a material latent vector, and the decoder interprets the material latent vector with a spatial fusion component and a small angular decoder. Thanks to the lightweight encoder and decoder, our network is able to achieve real-time rendering and editing. Meanwhile, our network only occupies a small storage of 5 MB, even for scenes with multiple fabric materials.
Yeah. This is really just a simple solution that can be quickly trained and that compresses the problem space better at the cost of some fidelity.
We’ll have made it as a species when the harvester in C&C no longer gets stuck or wanders through the enemy base.
In all seriousness though, I think RTS games (I only do single player) would be more fun for me personally if I could leave the economy/build order stuff to a reasonable AI and focus instead on military unit control. I’m just not someone that can handle all of that at once.
I think the potential for an enemy tailored to challenging and occasionally surprising me without simply overwhelming me would also be great.
thankfully, the genre has been dissected so there are games like revolution that focus on the warring armies aspect, and games like sim city that focus on the building aspect. and then there are factory games, which is a whole genre unto itself.
Sim City is older than RTS, or at least older than the genre defining RTS of Dune 2. Herzog Zwei release at about the same time as Sim City, and definitely didn't have any influence on it.
> In all seriousness though, I think RTS games (I only do single player) would be more fun for me personally if I could leave the economy/build order stuff to a reasonable AI and focus instead on military unit control. I’m just not someone that can handle all of that at once.
Just pick an RTS game without base building? They exist.
What about RTS games specialising on the military part? I haven't played them myself, but I think the Company of Heroes series is like this, light on base building and focusing more on the military aspect.
WARNO and it's predecessors have zero base building. You can place down a single Forward Operating Base that is just a supply truck without wheels, but everything else comes from off screen onto the game.
With more and more fixes for “hallucinations”, and with smaller models getting more capable (or bigger models getting more efficient, depending on how you look at it), I have a feeling that eventually it’ll be cheaper to render things neurally rather than rasterized.
Imagine if you could describe what is happening on the screen, including the state of the world and everything, with a really long prompt. You dynamically alter that prompt continuously based on inputs received.
There will be a point in the future where the prompt will render with 99% consistency. You’d probably still need some sort of barebones state server for things like hitboxes and to prevent cumulative divergence.
Well, the holy grail in games for AI will always be ... AI.
Like, actually really pseudo intelligent companions you can converse with. Not gonna happen in the foreseeable future, and the job markets are gonna be... Well... essentially gone once that exist, so it's probably for the best that this tech isn't actually anywhere close to AGI
I’d love to see a real improvement in games where the NPCs think further than a few if statements. If AI could track context of conversations and include player actions such as quests, relationships, deaths etc to influence decisions or outcomes it would add lots of enjoyment for me.
For me it’s either Stardew Valley or Paradox type games for my single player itch or in the past I’ve ventured off to MMORPGs like Eve or Wurm and play in “single player” mode.
MMORPGs have a greater sense of world change as players can carve mountains and build estates while slaying champions and such in Wurm online and you can stick to your plot of land and have nothing to do with it while enjoying the benefits.
This doesn’t exist in single player games but it could and should.
Although I don’t agree AI will be taking jobs but it will just be another tool used in the industry. We need to embrace change and use these new tools to make better content and not just assume jobs will be lost.
> If AI could track context of conversations and include player actions such as quests, relationships, deaths etc to influence decisions or outcomes it would add lots of enjoyment for me.
From what people say about them Bethesda games try to do this, but it seems like the result is they're impossibly buggy because noone can test all of that.
Undertale is low tech, but does track player deaths and use them in the story.
Didn't downvote you (and not sure why anyone would), but I still disagree: Aren't games ultimately all about interacting with other people?
It's already possible to play with actual other people, and quite popular too. Sure, those guys might not be very in-character or prioritise you having fun over them, but there's genuine intelligent interaction.
The other way in which games make us interact with other humans is by experiencing things designed by them. Story, dialogues, game design, atmosphere etc. I think we might see more ad-hoc generated world/story, but I think many people would want to experience the _same_ game, in some way, being social animals.
I'm not sure if and in what way generative AI or even AGI is going to disrupt the game industry. I can certainly picture the jobs of many people changing, from artists to voice actors.
But I'm not sure I see a holy gray here. Lots of interesting novelties that I'm sure will be tried, but not really a fundamental change to why we play games and consequently, what games are like.
There are plenty of games where the main feature is emergent gameplay resulting from a world simulator and the interplay of basic or complex rules. Dwarf Fortress is the OG here but also Sim City and Elite. Single player experiences are a huge segment and these will get better with LLM integration for world building and NPC agents. Fine tuned or custom models that operate on some basic state and rules could also be explored. You could train BERT family models to choose interesting state changes or actions based on the current game state for simple things like a dog NPC that would seem more intelligent by choosing the next action it should take e.g. deciding to chase a squirrel or growl at a new NPC you encountered based on its awareness of the context rather than following some rule system plus RNG. You could get rid of a lot of the stupid seeming game AI behaviour.
True, though in my mind I put these things into my "lots of interesting novelties" category. Everything in games (including the physics engine!) is typically fine-tuned to provide fun, more than realism. The same is currently true for AI (especially enemy AI), and I believe it'll continue to be that way.
Good point on the procedural aspect, Minecraft and similar games came to mind when I wrote my first post. I'd argue they are still fundamentally a designed, shareable experience, and that's my main point really. In Minecraft, you can't talk about the world much with someone else since it's generated, but the mechanics, materials, recipes etc. are the same, and provide exchange between humans playing the same game.
I believe we'll even see some successful AI heavy games in the near future, but I also believe we'll cycle back to games being, first and foremost, designed upfront. I might be wrong of course and we might see some kind of holodeck 100% custom game experience become dominant. It's just my hunch that we won't, even if it's feasible. Thinking as far back as I can, people always seem to have wanted two things from games: Fun, and connection to other people.
That’d be great for an RPG game, but if I’m playing a action adventure game or a style fighter, I generally care about how cool it looks and how smooth it plays than I do about the intelligence of the side characters.
I'm curious if the neural net ran wide and used the majority of the VRAM to achieve 60fps, or if it ran narrow and had more of a long-thread-length dependency.
For some reason I thought this was about simulating the fabric in a physical sense. Having watched the videos from the supplemental material (downloadable from the SIGGRAPH paper page) it appears to be about "just" rendering. The fabric does look nice though for some reason they chose very shiny fabrics for their video.
There was also another neural cloth rendering paper released recently, focusing on more micro-scale details (and not being real-time) https://doi.org/10.1111/cgf.15156
I think that the 60fps on a consumer grade card is the win, but that says a lot about the power of consumer grade cards in 2024.