Implement virtualized geometry rendering

[novemberist]

• Production edit: Related to Implement mesh shading in RenderingDevice #6822.

Describe the project you are working on

none specific, but would be relevant for any 3d project

Describe the problem or limitation you are having in your project

Producing high quality game-ready 3D assets is generally a very labour-intensive task for any engine, requiring many specialized steps (sculpting/high-poly modeling/photogrammetry, retopology, baking/creating texture maps, optimizing/tweaking poly count etc.)

Describe the feature / enhancement and how it helps to overcome the problem or limitation

So I guess most people have already taken a look at UE5's impressive new features by now. Now, I know that UE has a very different focus (and a amount of resources) than Godot.

Nevertheless, it is never a bad idea to draw some inspiration from other technology for Godot's long-term future. So maybe this is a good place for discussion, even if implementing similar features in Godot is not realistic at this point.

One of the big new advertised features of UE5 is Virtualized Geometry (called Nanite)

https://docs.unrealengine.com/5.0/en-US/RenderingFeatures/Nanite/

These are the advertised benefits of this new technology:

• Multiple orders of magnitude increase in geometry complexity, higher triangle and objects counts than has been possible before in real-time
• Frame budgets are no longer constrained by polycounts, draw calls, and mesh memory usage
• Now possible to directly import film-quality source arts, such as ZBrush sculpts and photogrammetry scans
• Use high-poly detailing rather than baking detail into normal map textures
• Level of Detail (LOD) is automatically handled and no longer requires manual setup for individual mesh's LODs
• Loss of quality is rare or non-existent, especially with LOD transitions

While the increase in visual quality of games using this technology in the future is immediately obvious, the more interesting fact is, that developement time for 3d assets might be significantly decreased, since the usual tedious process of optimizing and tweaking models will mostly disappear (at least this is what this technology promises)

This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I have no idea, but this is how Nanite in UE5 works according to Epic:

Nanite integrates as seamlessly as possible into existing engine workflows, while using a novel
approach to storing and rendering mesh data.

-During import: meshes are analyzed and broken down into hierarchical clusters of triangle groups.

-During rendering: clusters are swapped on the fly at varying levels of detail based on the camera view,
and connect perfectly without cracks to neighboring clusters within the same object. Data is streamed in
on demand so that only visible detail needs to reside in memory. Nanite runs in its own rendering pass
that completely bypasses traditional draw calls. Visualization modes can be used to inspect the Nanite
pipeline.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, as this is rendering technology.

Is there a reason why this should be core and not an add-on in the asset library?

This would have to be a core engine feature

[2plus2makes5]

Yeah it's huge for sure, but i think we should first see if it really delivers what it promises and what are the dowsides.

One last thing:

"This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience."

Realistic assets need realistic animations, sounds, voices etc, all things that a small team can't afford.

[novemberist]

Yeah it's huge for sure, but i think we should first see if it really delivers what it promises and what are the dowsides.

One last thing:

"This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience."

Realistic assets need realistic animations, sounds, voices etc, all things that a small team can't afford.

One of the caveats seems to be that, since mesh data is streamed from disk, users will probably be required to have a fast SSD to make the most of this technology. However, SSDs are of course relatively cheap compared to GPUs.

[2plus2makes5]

One of the caveats seams to be that, since mesh data is streamed from disk, users will probably be required to have a fast SSD to make the most of this technology. However, SSDs are of course relatively cheap compared to GPUs.

It makes me think 2 things:
1)it was tailored for the PS5 and XBS/DX12.
2)I can see this being used to push some sort of partly installed/partly streamed(under subscription obviously) hybrid future for games. :(

[mrjustaguy]

Partly installed/Partly streamed future ain't coming, it's going to be one or the other, however Streaming has the huge downside of latency and need for good infrastructure so a local pc market will always to some extent exist.. What is actually going on is there's a move to use huge amounts of highly compressed data, not to mention, SSDs are getting faster, and the capacity is going up.

Honestly atm, I don't see a point in Godot Trying to support it unless it isn't too much trouble to do so, which I doubt, also I think Nanite is Proprietary, so we'd need to make our own implementation.. And really like why? Godot isn't a AAA game engine, it's not aiming to be... Not even larger AA teams would don't need That sort of stuff.

[novemberist]

Honestly atm, I don't see a point in Godot Trying to support it unless it isn't too much trouble to do so, which I doubt, also I think Nanite is Proprietary, so we'd need to make our own implementation.. And really like why? Godot isn't a AAA game engine, it's not aiming to be... Not even larger AA teams would don't need That sort of stuff.

Of course I wasn't talking about implementing UE's Nanite in Godot, but rather discussing the feasibility of similar technology in Godot at some point. Why? Because, as I mentioned above, this is not just about AAA-quality graphics, but much more about the workflow when creating assets.

If this technology manages to establishes itstelf, in the future traditional asset creation pipelines (texture maps etc.) might become obsolete quickly, not only for big studios.

Things like photogrammetry (and directly using the resulting models in-engine) are especially interesting for developers that don't have a lot of dedicated 3d artists.

[mrjustaguy]

I'm sorry to drop the ball but, Photogrammetry in the way you're thinking about it is dead on arrival for multiple reasons...

1. High poly counts
2. Hard to Properly Use
3. Is already becoming obsolete (AI)
4. If you can't photo it, you're toast.

all 4 of those points are being addressed with AI/Machine-Learning which in turn negates the argument for supporting Nanite-like Tech. How is this possible?

1. AI can generate less points to get the same visual quality and triangulate that
2. AI can guess most of the Occluded parts, and has a better understanding of Camera Angles, Lighting and such, meaning Less Input is needed for the Output
3. Photogrammetry is Computationally VERY expensive, AIs doing this stuff, Ain't nearly as computationally expensive.. Here's one example (there are many more different ones too) https://www.youtube.com/watch?v=548sCh0mMRc
4. Describe what you want - AI will generate. This is already possible for images, and code (search for GPT3)

While yes, these aren't all available yet and there is still a ton of work to do, they're getting better faster and faster, in both Results and Efficiency.

Also one thing to Note, Godot 4 comes with an Auto LOD system, so worrying about Polycounts isn't as big of a deal as it used to be, unless you're real close to the Mesh, and eventually we'll get Mesh Shaders to further drop the polycount rendering issue, as things that ain't on screen, won't be rendered, or well, to a much less extent compared to now as it allows for much much better culling, and instead of having a single mesh that is entirely drawn even if it is only contributing a pixel (if even that) to the picture, it'll draw the meshlets (a part of a mesh) that are contributing to the pixel, that are on screen, instead of the WHOLE mesh.

[2plus2makes5]

Partly installed/Partly streamed future ain't coming, it's going to be one or the other, however Streaming has the huge downside of latency and need for good infrastructure so a local pc market will always to some extent exist.. What is actually going on is there's a move to use huge amounts of highly compressed data, not to mention, SSDs are getting faster, and the capacity is going up.

Right now a Netflix-like game service either:
-completely streams the game with all the problems we know.
or
-makes you download dozen or hundreds of Gbs.

Instead they could make you download only the base files and then stream all the rest, unlike frames assets could be donwloaded before they are actually needed and once dowloaded they can be used as many times as needed.

I'm ok with this being used only for subscription services, but the many examples of big game companies' greed make me think that it won't stop there.

[mrjustaguy]

I've never seen a netflix-like game service that makes you download dozens or hundreds of GBs, that only sends assets as needed.
I've only seen something similar when downloading games to allow you to play while downloading, but it is a full game download still, just allows you to play things as they have downloaded.

Could you please give examples of the 2nd type of game services, that makes you download dozens or hundreds of GBs the way that isn't actually downloading the Game?

Also: I did look up at Nanite, and according to them, the size increase isn't that big, because it does save room on Textures and LODs, and it does seem to be compressed, so on that end it isn't actually as terrifying as it sounds in terms of the actual size added, and for some assets it might in fact decrease the size, as 4k normal maps are HUGE in terms of space usage, and doesn't have the fidelity of an Actual Mesh, and Nanite offers the fidelity, and removes the need for such huge textures, and the actual size is in the ball-park of the current stuff, going both above and below the current equivalent-result technique file sizes depending on the scenario.

[aaronfranke]

Could you please give examples of the 2nd type of game services, that makes you download dozens or hundreds of GBs the way that isn't actually downloading the Game?

Sure, two examples:

• Microsoft Flight Simulator 2020
• Outerra Anteworld

Note that neither of these games are feasible to make in Godot for other reasons.

[2plus2makes5]

@mrjustaguy
-i guess i wasn't clear, when i was talking about downloading hundreds og GBs i was referring to the download of the entire game, but think how incovenient it is, people has to wait to play, has to have lots of space on the hard disk and once downloaded a game could be pirated...

On the other hand as you and @aaronfranke pointed there are already examples of partial stream of a game, so it's not a new thing. With partial installed partial streamed there would be less wait and space for the user and since the data that's not useful anymore could be removed the game could be never completely on the hard disk so it can't be pirated(well i guess it could still be but it would be harder).

For streaming services that's absolutely a good thing, but my fear is that this could lead to a future where instead of buying a full game with DRM we will buy only the base files and the rest being streamed.

[mrjustaguy]

My only issue with Games as a Service are the fact that they in no way guarantee that I'll be able to play the game in the future, and could die at any moment... I mean when you've got the whole game locally, it's only about your PC being able to run it, and dealing with any DRM or Emulation issues that come up.

Your fear that Games will be insanely large because of Nanite however, are unfounded as Instead of using Normal Maps etc to store data, you store data in actual polygons... and Trust me, Normal Maps take Exponentially more Space AND lose some fidelity.. and what I mean by that is, from 1k x 1k, to 2k x 2k, to 4k x 4k, to 8k x 8k and beyond, the number of pixels to store goes up really fast, and isn't efficiently used in terms of data storage, compared to a well designed Mesh storage system (like Nanite)

The only reason why we use Normal Maps and such is because it's cheaper for the GPU to texture stuff, compared to pushing tons of triangles, and not because Textures are more Space friendly, because spoiler alert, they're often not. It's simply Cheaper to have a Million triangles, compared to 1/10th as many triangles plus LODs and a 4k normal map (equal-ish results) in terms of data storage.. I mean it's like 3 million Vector3 points, versus 300k Vector3 Points, plus like half as many points in LODs, and 4k x 4k pixels which is some 16 Million Pixels, and it goes the same for larger meshes..
10m triangles vs 8k or higher res normal map required, which would be 64m+ pixels for equivalent output..
Also Let's not forget, that instead of Triangles, you can do some impressive compression of the mesh by instead of having multiple copies of the same vertex, one could store all the Verts, and the data to connect them, separately and gain quite a bit of extra space if there's a ton of vertex sharing among triangles, which means that the actual size of storing the Triangles is even smaller.

But still, atm, I don't see the point for Godot to go down that Route, as the work to make something like this is just too large and not that worth the effort for a small benefit (if any) for the types of projects most are using Godot for and it's not like traditional methods are Horrible on the small scale, it's just that they Poorly scale up with the Increase in Visual Fidelity of modern AAA games.

Really You can think of it as simple efficiency calculation. Current methods are like 40% efficient, new method is 50% efficient, but who cares if you're 10% more efficient if it already takes you practically no effort at 40% efficiency to pull something off? Where it starts to make a difference is when it's at a larger scale, where the effort lost to lower efficiency actually starts to matter.

[2plus2makes5]

@mrjustaguy the tecnique in itself seems fantastic and i don't have anything against it, but i fear how some greedy companies could use it, the step from buying games that stream from the hard disk to buying games that stream from internet seem way too easy and obvious,

[mrjustaguy]

They don't need the technique to do such a greedy move. they really don't, they could do so easily with current stuff if they wanted to (and much MUCH older stuff), so really no need to worry about the technique incentivizing that move especially as the space required is on par with current techniques, so it really doesn't change anything in that respect.

Are Companies more likely to go down that Greedy route today? Yes, But not because of the technique, it's because They are getting Greedier, because why make a good product and sell it for a reasonable price if you can make a horrible product and sell it for an absurd price? This is something that the Internet allows, especially Digital goods, and they've been inching to the Horrible Product for an Absurd Price for quite a while, it's just that at the pace at which they're doing it they ain't spawning enough outrage to force them to back off, and people start normalizing the new, worse products instead.

Can this be changed? With Capitalism, a hard No, but does it matter? again, no because within a few decades Capitalism will Be dead because it simply won't be able to function because it revolves around the Idea of People Having work to do for which they in return get money, something that won't be true when AI & Robots are able to do everything humans can, and far better than humans can..

However I think this whole segment is Off topic, with the "Partly Installed-Partly Stream" topic in mind that is.

[Wolfe2x7]

@2plus2makes5 -- I wouldn't worry about gaming turning to streaming. Maybe the likes of Microsoft or Sony would try to go all-in, but any suggestion that Nintendo ever would is hilarious to anyone familiar with Nintendo's ethos and stubbornness, and some genre communities (like the fighting game community) are never going to adopt streaming -- and you can bet they will make a lot of noise about trying to force streaming upon them.

I'm convinced there will always be developers who perceive the negatives of streaming games, continuing to make games for those of us who intend to keep playing games "locally".

On topic, I had one comment:

"This sounds like a big deal for small teams or solo developers, which makes it in turn potentially very interesting for Godot's target audience."

Realistic assets need realistic animations, sounds, voices etc, all things that a small team can't afford.

Not necessarily always. You can simulate a vehicle or make a super-pretty modernized Marble Madness or Bust-a-Move clone (or more ideas) without necessarily animating or voicing a single character, and there are free resources for sounds, much like textures. Some individuals are very talented with animation; I've been seeing that in the Godot community.

[mrjustaguy]

For creating games (but in general too), the quality of the game will primarily be the result of the following, roughly in this order:

1. Team Talent (Natural Competence with Computers and such)
2. Team Problem Solving Capabilities (Intelligence & Creativity)
3. Team's Dedication (How much they care and put in to the Project)
4. Tools (But only to a degree, there's a point of diminishing returns with High number of tools & tool features, and too much can turn into negative benefits)

So yeah, a Solo talented, intelligent, dedicated person with decent tools -CAN- beat a group of not so talented, not creative, not too dedicated people with lots of specialized tools... So it's really Mostly about The person doing the stuff, with the tools being complementary to the person, and not being The thing that makes them automatically capable of doing something, they just help

[mrjustaguy]

Ok, so it's possible that I was wrong with the way they're implementing nanite, as they Might not even be Storing Geometry in a normal sense (#2793 (comment)), as Juan pointed out in his Speculation on it's implementation on Twitter - https://twitter.com/reduzio/status/1398467670056570880

However Considering the Paper that's posted in the Speculations, I Don't think they're using that approach, as Geometry Images are actually Significantly Smaller, compared to Nanite, as 70k faces taking up ~25kb, assuming somewhat linear Face-Size ratio scaling, would mean ~500kb (20x, as 70k x 20 is 1.4m triangles which is close to 1.6m used in UE example that takes ~19mb)

I mean 19mb is a 38x size increase over 0.5mb, so unless UE is using the method, just with much Higher Res Geometry Images, It's probably Not the same method.

UPDATE: Yes, Nanite doesn't use Geometry Images

Ofc, this data is According to The Paper and UE for their respective methods, and who knows how they Actually work & scale irl...

[2plus2makes5]

I'm happy to see you are all optmist so i hope you are right, so let's return to the topic.

I saw just a pair of videos comparing ue4 and ue5 and well the difference in quantity of stuff on screen is immense.

Unless there's a huge catch or drawback Nanite is clearly the future of 3d graphics, Unity and other 3d engines won't lag behind and will implement their own version, if that will be the case like it or not Godot will have to do the same, not immediately because we still know almost nothing about Nanite, especially the drawbacks, but in a not too distant future.

[anunknowperson]

Essentially, Nanite is just an automatic LOD system. I think it's not that hard to implement. But you need to create some analogy of the Decimate modifier from Blender. When importing meshes, automatically create a certain number of LODs for them. While the project is running, you need to load these LODs from the disk, depending on the distance. Of course, this will be a useful technology for the engine.

[mrjustaguy]

That is not how Nanite works, it's a bit more complicated.

Also Godot already has an Automatic LOD in 4.0, and it's quite visible that it's not the same as Nanite, both visually and performance wise.

[TechnoPorg]

There's going to be a talk at SIGGRAPH 2021 by Epic explaining in detail how Nanite works: http://advances.realtimerendering.com/s2021/index.html

[TechnoPorg]

The slides from the talk have been posted, which include the script, the illustrations, and links to other work that was referenced during the talk, as well as some extra stuff that didn’t make it in due to time constraints. Kudos to Epic for freely sharing this amazing technology, I probably wouldn’t have if I’d come up with it 🙂. http://advances.realtimerendering.com/s2021/Karis_Nanite_SIGGRAPH_Advances_2021_final.pdf

[generrosity]

There's going to be a talk at SIGGRAPH 2021 by Epic explaining in detail how Nanite works: http://advances.realtimerendering.com/s2021/index.html

I hope this is of interest to this subject. Appologies if not. New SIGGRAPH video on Youtube by a Epic engineers - A Deep Dive into Nanite Virtualized Geometry, 1hr.

https://www.youtube.com/watch?v=eviSykqSUUw

[Mikeysax]

I just saw this and could be useful for a future implementation in Godot given this Nanite-like implementation is made with Vulkan and C++:

https://github.com/milkru/vulkanizer

From the repo:

"Rendering engine prototype made with Vulkan 1.3. Code is written using a simpler C++ coding style. Project is written for the C++20 standard and x64 system. Currently the code is tested only on Windows, using MSVC (Visual Studio) and MINGW (Visual Studio Code) compilers. Linux is not completely supported at the moment, but it should be easy to port, since all third party libraries are cross platform."

[Zireael07]

@Mikeysax What's "Nanite-like" about this? Looks like yet another Vulkan renderer...

[mrjustaguy]

Meshlet culling, which is basically the same principle as Nanite, just different and simpler implementation that also requires Turing/RDNA2 GPUs because of mesh shaders unlike Nanite which has a Compute Fallback that allows GCN & Maxwell to use it.

[CK85]

Hi, I'm currently looking into Godot, with the intention to port NanoTech to Godot. NanoTech is similar to Nanite and is mainly based on C# and compute shaders, so it should not be too complicated to port it. For more informations you can join our discord server: https://discord.gg/kKA9UyR4qg

[Broken1334]

WebGL2 Virtual Geometry (some info about implementation)
https://twitter.com/twodelab/status/1732212197894603186

Cluster-based rendering, Nanite-style (200M triangles in 1ms), based on "Multiresolution structures for interactive visualization of very large 3D datasets" paper, author wrote "Most of UE5's Nanite pipeline pretty much exactly follows that paper."
https://www.youtube.com/watch?v=7JEHPvSGaX8

[VoidEssy]

Nanite like technology would be a very welcome core addition to the engine allowing high-detail assets to be considered when developing the product. Even from indie perspective if you want to attempt something of higher visual fidelity UE5 allows you to save time and resources offering nanite meaning you don't have to spend time and resources working on and implementing LODs.

I'm glad to see @CK85 in this conversation having tracked his Unity NanoTech work for quite a while before Unity decided to throw everyone under the bus including themselves.
PS. Unity China supposedly packs Nanite in itself which is off-topic comment on the matter

[JMS55]

I've been working on an open source implementation for Bevy. Feel free to copy it. It doesn't require any recent (More than WebGPU baseline) hardware features at the moment, although I will eventually add faster paths for when mesh shaders, atomic u64 texture min/max, subgroup ops, etc are supported. I'd be happy to know that my work helped out Godot. If you have any questions, feel free to reach out (the Bevy discord is the easiest way to get in contact with me).

There's also Carrot and a couple of other projects (both open and closed source) recreating Nanite as well.

The siggraph presentation Unreal gave really explained all the tricks very well. The only secret magic is their simplification and error metrics, and all the various tweaking and heuristic tuning they've done to get it to perform well in real-world scenes.

[Broken1334]

Detailed article about Virtual Geometry implementation for Bevy by @JMS55
https://jms55.github.io/posts/2024-06-09-virtual-geometry-bevy-0-14

GDC 2024 presentation: Maximizing Graphics Performance with Flexible Virtualized Geometry (by VIRTUOS)
https://gdcvault.com/play/1034621/Maximizing-Graphics-Performance-with-Flexible

GDC 2024 presentation: Nanite GPU Driven Materials (by Epic Games)
https://epicgames.ent.box.com/s/uepi1joznb0a3mhw7hu5a4m2d6vm61pi/file/1495820437550

[luxun0517]

I've been working on an open source implementation for Bevy. Feel free to copy it. It doesn't require any recent (More than WebGPU baseline) hardware features at the moment, although I will eventually add faster paths for when mesh shaders, atomic u64 texture min/max, subgroup ops, etc are supported. I'd be happy to know that my work helped out Godot. If you have any questions, feel free to reach out (the Bevy discord is the easiest way to get in contact with me).

There's also Carrot and a couple of other projects (both open and closed source) recreating Nanite as well.

The siggraph presentation Unreal gave really explained all the tricks very well. The only secret magic is their simplification and error metrics, and all the various tweaking and heuristic tuning they've done to get it to perform well in real-world scenes.

Great Work!

[VeinSyct]

Similar techniques is implemented in threejs, see topic https://discourse.threejs.org/t/virtually-geometric/28420/58

[novemberist]

Maybe I'm wrong, but I'm of the impression that Nanite in UE5 has by and large not turned out to deliver what was promised and in reality often causes more problems than it actually solves. And it is basically a high-end feature for top-tier graphics cards, not a general purpose performance saver.

Although there is always room for improvement, Godot currently handles high-poly assets relatively well even without virtualized geometry. So I'd like to see improvements to Auto-LOD much more than a Nanite-clone at this point, I think.

[JMS55]

Nanite has mostly delivered on it's promise imo, which is letting large, open world and dense games just chuck a bunch of geometry on screen without worrying about tweaking LODs. Performance is worse than if you spend time carefully tweaking things, but it's a mostly effortless solution with a very, very high ceiling on how much geometry you can render.

The main thing it struggles with is aggregate geometry like foilage, but I know Unreal is working on some new voxel-based additions to Nanite for that.

Regardless, if Godot doesn't copy Nanite, I still highly suggest writing a meshlet based rendering system similar to Alan Wake 2. Even with traditional LODs and no software raster you can still get a ton of mileage out of clustering and compressing your geometry, doing two pass occlusion culling, and then feeding the resulting clusters to mesh shader invocations. With Nvidia's new CLAS stuff for ray tracing, it'll also allow for memory savings and faster BLAS builds if Godot add ray tracing support.

[mrjustaguy]

Calm Disappointed Tone: As both a Gamer and a Game Dev UE5 has failed on all counts to deliver on it's promises.

Frustrated Tone: As a Gamer, UE5 has failed hard to deliver the "easy realism" due to all the blotchy bubbling noise artifacts of lumen, and the insane amounts of upscaling required to get frankly unimpressive looking shading to even run on anything that's not an RTX 9090 and the no LOD pop in promise of nanite has failed time after time (as not everything is viable as a Nanite mesh still)

Mocking/Joking yet Frustrated Tone: As a Game Dev, UE5 is only useful to those that should be banned by law from being in game dev for their sins in choosing the most impractical, idiotic ways of creating the simplest of things such as the floor in https://www.youtube.com/watch?v=UHBBzHSnpwA&t=160s which is clear evidence that IQ can indeed be negative, It's just a level of Intelligence that's out of reach for your average Meat bag.

Serious Tone: Godot should experiment with Meshlet rendering, but instead of going for the ambitions of using trillions of polygons like what nanite attempted, should instead be focused on having game ready assets cull and render more efficiently, and should be constantly compared to the traditional render pipeline in real world scenarios to ensure there is actually a benefit to using the technology.

[2plus2makes5]

UE5's focus is clearly AAA games(and maybe movies even more than games), I don't know how much UE5 deserves the criticisms that is getting, but lots of people are tired of stuttering, graphical issues, and super heavy games.

Instead of chasing UE(and inevitably its problems) I propose Godot to clearly position itself as a "solution" to the current games performance problems by being a more performant, lightweight and general purpose game engine with clean and good enough visuals, focus on implementing all the "essential" graphical features that it's missing, fixing what it needs to be fixed and making everything as performant as possible.