Design of VisualAddon/ShapeNode for renderer

[mklingen]

There was a lot of discussion here: https://github.com/dartsim/dart/pull/608/files#r52828569 about how to design the VisualAddon/ShapeNode to support stuff like vertex colors, texturing, etc. for the renderer. It probably makes more sense to have an issue for this.

I think I can best summarize this with a table:

Vertex Properties

Property	Stored In	User modifiable?	Shared among instances?	Notes
Vertex positions	MeshShape	yes?	yes?	Vector3 array
Vertex normals	MeshShape	yes?	yes?	Vector3 array
Vertex colors	MeshShape	yes	yes?	Vector3 or Vector4 array.
Vertex UV coords	MeshShape	yes	yes?	Vector2 array.
Index buffer	MeshShape	yes?	yes?	uint16_t or uint32_t array.

Shape Properties

Property	Stored In	User modifiable?	Shared among instances?	Notes
Shape local transform	ShapeNode	yes	no
Shape scale	ShapeNode	yes	no

Material Properties

Property	Stored In	User modifiable?	Shared among instances?	Notes
Shape base color	VisualAddon	yes	no	Vector4
Textures	VisualAddon	no	yes	What format???
Materials/Shaders	VisualAddon	no	yes	What format???
Material Uniforms	VisualAddon	yes	yes	generic key/value store

My view is that the vast majority shouldn't be represented anywhere in dart but should just be loaded by the viewer from the uri associated with the visual mesh. The only exception to this would be procedural generated meshes, which can have a very simple structure.

[mxgrey]

This summary looks really good to me. The only thing I'd tweak is that Vertex positions, Vertex normals, and Index buffer should be stored in the MeshShape rather than the ShapeNode. Granted, the ShapeNode is holding a reference to the MeshShape, but that MeshShape is potentially getting shared among multiple ShapeNodes.

[mxgrey]

Also, I'm not totally clear on what is meant by

the vast majority shouldn't be represented anywhere in dart

Is this referring to the textures, materials, and shaders which simply contain URIs? From the summary, it looks to me like everything else would have a native representation within DART.

[mklingen]

I'm saying for most meshes we wouldn't make an attempt to load most of the visual data into dart at all (why should dart have this data? That should be the job of the renderer). That includes all data on the table not needed for dynamics/collision. That would completely eliminate the need for assimp outside of renderers.

VisualAddon could then be seen as a way to modify the way the renderer draws the mesh as defined in the file it loaded, just to allow the user some minimal control over what gets drawn. They would also be useful for creating dynamic procedural objects to draw in the viewer.

[mkoval]

@mklingen thanks for putting the table together.

I suggested a more extreme version of this in #453, where the viewer would ignore the geometry loaded by DART and independently load the mesh from Uri with whatever library it chooses. However, @psigen raised an important downside:

Consider loading an object from a file URI but then changing the object on disk (I was trying to adjust some objects in a scene while I had it open in rviz). There is zero guarantee that each loading operation will have the same result, especially if assets are lazily-loaded, which can lead to out-of-sync visuals or even more pathological consistency issues.

The problem is actually worse for embedded textures. We would have to either invent some custom Uri specification to refer to a specific texture inside a mesh, which would likely be quite specific to Assimp, or extract the textures and host them at known Uris.

This would also make it difficult to programmatically generate materials, e.g. creating a texture to visualize a color map, which is a common use-case. It would also be interesting to, in future versions of DART, use the same mechanism to support variable material properties (e.g. friction coefficient) across a Shape.

I'd prefer one of the two extremes, either: (1) force the viewer to re-load the entire mesh as in #453 or (2) load all of the textures into memory when we load the mesh. The former is much easier to implement, but the latter avoids the limitations mentioned above.

If we go with (2), I like @PyryM's suggestion of allow arbitrarily-named vertex properties and materials to avoid getting in the business of defining a standard set of material properties. I am not sure whether these should be attached to MeshShape or VisualAddon, but am leaning towards the former.

Thoughts?

[mklingen]

where the viewer would ignore the geometry loaded by DART and independently load the mesh from Uri with whatever library it chooses

yes, that's probably the right thing to do.

Consider loading an object from a file URI but then changing the object on disk (I was trying to adjust some objects in a scene while I had it open in rviz). There is zero guarantee that each loading operation will have the same result, especially if assets are lazily-loaded, which can lead to out-of-sync visuals or even more pathological consistency issues.

Is this a problem? Deleting a mesh and re-loading it from disk is a problem the renderer should be designed to deal with. In the case of rviz, pushing the "reset" button does this already. Other viewers could have similar commands. I would probably go with option (1).

That said, I've often found it annoying that OpenRAVE doesn't actually store visual geometry. That made it difficult to do offscreen rendering and other tasks that require querying the visual geometry. It also made it impossible to generate visual meshes. It would be very useful to have this data under user control, but I can't stress enough how much of a pain its going to be to write a custom asset/material system for dart.

[psigen]

Is this a problem? Deleting a mesh and re-loading it from disk is a problem the renderer should be designed to deal with. In the case of rviz, pushing the "reset" button does this already. Other viewers could have similar commands. I would probably go with option (1).

The problem is the assumption that everything else gets reloaded at the same time in the same scope. In my case, the geometry in the collision checker did not update, since it was already in memory, but the viewer reloaded the same geometry at arbitrary times when it reset, making it out-of-sync. In addition, rviz implicitly assumes that every single viewer has access to the same URI scope, which fails across the network and even across workspaces on the same machine, and can't be easily rewritten since mesh file formats often contain internal references to more geometry and textures in arbitrary other files.

Our current implementation of dart_rviz both manually rewrites package URIs and runs a micro webserver that serves the entire host catkin workspace over the network in order to sort of workaround this issue. Our current implementation of or_rviz doesn't do any of this and just gets out-of-sync as a result, and almost never works properly across machines due to differing URI scope.

That said, I've often found it annoying that OpenRAVE doesn't actually store visual geometry. That made it difficult to do offscreen rendering and other tasks that require querying the visual geometry. It also made it impossible to generate visual meshes.

I agree, this makes programmatic and embedded mesh data pretty impossible, which is unfortunate. I ran into this problem recently when I was trying to visualize a mesh into OpenRAVE that was being regenerated programmatically from point cloud data.

It would be very useful to have this data under user control, but I can't stress enough how much of a pain its going to be to write a custom asset/material system for dart.

😓 Yeah... I completely agree. But I think it's still worth at least seriously considering given the huge amount of hackery that we currently have to do to avoid the above issues.

[mklingen]

I see. So I guess the question is: do you want dart to be more like box2d, bullet, havok (i.e a physics simulator) or do you want it to be more like a game engine? Because adding visual elements that can be controlled by the user is going to move it into that space, perhaps at the detriment to using it in other applications that don't require a viewer.

[mxgrey]

perhaps at the detriment to using it in other applications that don't require a viewer.

I think if we utilize Addons and library components in an appropriate way, we can avoid penalizing people with things they don't need.

One option might be to utilize a third-party library (whether it be assimp or something else) for handling visual assets, but have the interface for this be an optional library component. That way, the core kinematics/dynamics/optimization components of the library don't depend on assimp any longer (which I think is the primary goal here), but people will still have an easy and ready-to-go way of utilizing visual assets through DART, as long as they're willing to accept the third-party dependency. Of course it should be approached in such a way that the users can create and embed their own extensions for visual assets, so they aren't tied down to whatever route we decide to go.

[PyryM]

Whatever you decide to do about materials and textures, it would be good to keep all of the vertex attributes together in the MeshShape rather than splitting them up, and to have dart responsible for owning all of them. This would avoid the worst of the simulator/viewer consistency issues (e.g., world scale in dae files, up direction in dae files, assimp loading flags like vertex order optimization, etc.).

[psigen]

@mklingen: That's an excellent way of putting it. I think the reason that we are stuck in the middle is because:

• DART core is focused on kinodynamic simulation, and thus primarily cares about just the physical mesh data, much like the other physics simulators.
• Unlike the above libraries, we also seem to want DART to manage loading assets, which is why it is becoming responsible for this extra data that the other libraries can ignore.

Basically, if DART didn't do any loading and always received assets from another library, the original simplified API would be fine because the other library could provide the additional information to other components (e.g. normal game engine architecture).

If we want DART to do saving/loading as well (because no other library managed environment representation works sufficiently for our purposes), then it also needs to be responsible for the other data, because no one else will be able to get it consistently.

But in general, I agree with everything in the last three comments:

• 👍 for a simple mechanism to store loaded visual data
• 👍 for that mechanism to not add cost if people don't need it

I think this problem is going to come up in various forms so long as DART is loading environments, so it's worth some thought.

[mkoval]

@PyryM and I discussed this a bit in person today. Here's a quick summary of our conclusions:

• We need to support three types of data:
  1. mesh-wide properties (at least of type Vector4d)
  2. vertex properties (at least the Vector4f properties supported by OpenGL and DirectX)
  3. textures (at least two-dimensional textures of type Vector4d)
• Data is accessed by string name; DART does not define a canonical set of strings.
• This data should be stored in the MeshShape, not in the VisualAddon.

We'd modify loadMesh to extract this data from the aiScene and attach it to the MeshShape using some naming convention (this part will be a challenge, but hopefully a surmountable one). The user could do the same when programmatically constructing a mesh. The viewer can inspect the MeshShape for any properties it supports and use them for rendering, ignoring any properties that it does not recognize.

I think this roughly amounts to @mxgrey's suggestion of making MeshShape an AddOn. This gives us efficient access and type-safety for free. As @mklingen suggested, we can keep VisualAddOn as a few simple, high-level tweaks to rendering (e.g. visibility status, tint color, whether to render textures), that do not require (expensive) copying of data to the GPU.

Any objections?

[PyryM]

Well our conversation got cut in half and that isn't quite correct. Shader uniforms ("mesh-wide properties") are usually vec4f, and textures are normally rgba8 (4 bytes/pixel).

But how I see it:

• Vertex attributes are straightforward: whatever object stores the vertex positions should also keep the other vertex attributes.
• Material properties (shader uniforms) are lightweight: a handful of vec4's and perhaps a string or two. Shader uniforms are normally set every frame anyway, so (ideally) the viewer doesn't even need to be notified when they change.

The big question is what to do about textures, because they do need to be uploaded to the GPU, and updating them is a comparatively expensive operation (so it would be bad for the viewer to assume every texture is changing every frame).

[mklingen]

Okay, let's go down that rabbit hole then. I like Unity's asset system way of dealing with materials. (though most game engines have similar systems).

Shaders can be glsl, hlsl, or cg. They have uniforms stored in configuration files (or on disk, or embedded into an asset) that can be any float, int, bool, Vector , string , Matrix, or a pointer to a texture. This is implemented as a generic key/value store. A "material" is just a shader and its associated uniforms. Materials and textures are stored in a central repository and are associated with asset tags -- so a visual mesh is just an asset tag that says where its mesh data is stored and a material asset tag that corresponds to the uniforms/shader. To keep things simple, if no material is specified the renderer can fall back on a simple lambert shader and read ad-hoc uniforms like diffuse, specular, from VisualAddon.

As for textures, we can just store them as arbitrary uint8_t arrays with an arbitrary number of channels/bit depth. The renderer will then have to associate these with texture samplers that are passed to the shaders. This is definitely the trickiest part, especially if the models we load in have embedded textures that will have to be converted to the desired format.

EDIT: it's also important to note that some meshes can be designed with multiple materials. So we should be able to support multiple materials per mesh. The way this is usually done is the mesh is broken up into sub-components each which have their own material.

[PyryM]

This is reasonable, except I wouldn't try to actually have dart directly deal with the shaders-- just have the material specify a "material type" as a string and let the actual viewer handle the details of its shaders. In a simple viewer the material type might map directly to a shader (e.g, "diffuse" --> "diffuse_vs.glsl","diffuse_fs.glsl"), but we shouldn't mandate that.

I also don't think it's worth trying to directly support multiple materials per mesh; it will add a lot of complexity for a feature that will be only rarely used. And if somebody really needs it, you can get the same effect using vertex attributes and shaders without having to design dart around the possibility.

[mkoval]

I know little about rendering, so here are a bunch of questions for @mklingen and @PyryM. 😓

Shaders can be glsl, hlsl, or cg.

To be clear, these are all domain-specific languages for writing shaders? Where are shaders loaded from?

A "material" is just a shader and its associated uniforms. Materials and textures are stored in a central repository and are associated with asset tags -- so a visual mesh is just an asset tag that says where its mesh data is stored and a material asset tag that corresponds to the uniforms/shader. To keep things simple, if no material is specified the renderer can fall back on a simple lambert shader and read ad-hoc uniforms like diffuse, specular, from VisualAddon.

I'd prefer to not define a central asset repository in DART. Doing so introduces global state and potential concurrency issues. Instead, I'd rather have the assets managed by shared_ptr and and provide a callback for when the asset is modified to address @PyryM's efficiency concern. Each renderer can handle de-duplication however it chooses, e.g. by updating an internal asset repository when necessary. Is this a reasonable solution?

I understand that textures should be de-duplicated and only transferred the the GPU when modified. This could be enforced by the asset repository. However, I don't understand the motivation for the additional level of indirection introduced by "material." What is the difference between: (1) having four objects that each reference textures A and B and (2) four objects that reference material C, which references materials A and B? What advantage does (2) have over (1)?

On the same note, it should be (relatively) efficient to render the same mesh with multiple textures. In that case, it seems that vertex properties should be stored in MeshShape and all material properties (e.g. uniforms and textures) should be stored in VisualAddon. I think this agrees with what @PyryM said above.

it's also important to note that some meshes can be designed with multiple materials. So we should be able to support multiple materials per mesh. The way this is usually done is the mesh is broken up into sub-components each which have their own material.

How does this work from a rendering perspective? How do you tell the GPU to apply different shaders to different parts of the same mesh?

---

I noticed that @PyryM drew a distinction between uniforms and textures (because uniforms can be efficiently updated on every draw, but textures can not be). @mklingen mentioned that textures are by pointer in a uniform. What am I missing here?

[PyryM]

You have it right: textures are effectively a pointer uniform. Changing which texture a uniform points to is cheap: if you have ten textures already loaded on the GPU, you can change which texture a mesh uses to one of those ten every frame without penalty. Changing the actual texture data of any texture is expensive, because it has to be uploaded from the CPU to the GPU.

[mkoval]

Thanks for the clarification - that makes sense.

@PyryM Could you also respond to my question about materials? I've duplicated it below:

However, I don't understand the motivation for the additional level of indirection introduced by "material." What is the difference between: (1) having four objects that each reference textures A and B and (2) four objects that reference material C, which references materials A and B? What advantage does (2) have over (1)?

[PyryM]

It makes it more convenient to create a bunch of objects with the same appearance if you can create a single material and assign it to all of them.

[mkoval]

@PyryM Do you think this is something we should include in the DART API? Or leave that up to the user, since it's only a matter of convenience (not efficiency)?

[mklingen]

To be clear, these are all domain-specific languages for writing shaders? Where are shaders loaded from?

Yes, glsl is for OpenGL, hlsl is for DirectX, cg is supposed to work for both. The shaders are literally source code that gets compiled at run-time and loaded into the GPU.

Each renderer can handle de-duplication however it chooses, e.g. by updating an internal asset repository when necessary. Is this a reasonable solution?

Okay, as long as DART isn't creating new meshes every time you clone an object or something its fine.

How does this work from a rendering perspective? How do you tell the GPU to apply different shaders to different parts of the same mesh?

You don't. You break up the mesh into multiple parts and render them in separate passes. In the ADA mesh, for instance, the hand has two sub-meshes with two different materials (one for the white rubber thing and one for the black plastic wrist). This is a pretty common thing to do with 3D modelling software, because it can be easier than texturing. rviz breaks these into two meshes and renders them seperately.

However, I don't understand the motivation for the additional level of indirection introduced by "material." What is the difference between: (1) having four objects that each reference textures A and B and (2) four objects that reference material C, which references materials A and B? What advantage does (2) have over (1)?

I have four objects with textures, but how are they drawn? How does the object respond to lighting? These are the kinds of questions a material answers by providing a shader and uniforms (parameters) of the shader.

By having all four objects reference the material, we can change the way all four of them respond to lighting. It also allows the renderer to batch together all objects that have the same material without repeatedly sending new data to the GPU (this is a pretty big performance boost). Ogre (rviz) already has a material system for this purpose. rviz loads materials from .dae files and creates materials that it finds stored in them. For meshes without materials, rviz uses a default lambert shader.

mentioned that textures are by pointer in a uniform. What am I missing here?

The uniform stores a pointer to the texture as loaded on the GPU.

[psigen]

Having to maintain lists of models that use each texture in userland and write signal listeners to keep them all in sync is insane.

Nice-looking materials need to use a lot of textures and vertex attributes together. If it's going to be handled by some other rendering library, it might be fine to leave it out of the core library, but some library is going to have to do it.

[PyryM]

Which proposal are you arguing against / for?

[psigen]

Just pointing out that a material system will emerge somewhere in library form, so it's either going here, or there will have to be a tightly coupled library that does it. It's not going to be a thing that end users want to deal with.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[PyryM]

This showed up in my notifications, so here's my updated suggestion: implement materials to match how the GLTF 2.0 format does them, which is a metallic-roughness physically based (PBR) model that is commonly supported across a wide variety of rendering environments.