Project map imagery onto reality meshes

[pmconne]

Is your feature request related to a problem? Please describe.

Some scenes include reality meshes representing portions of the earth's surface. Currently, map layer imagery is only draped onto the global terrain/ellipsoid, not onto reality meshes. Unlike the global terrain, these reality meshes usually have their own baked-in texture imagery too.

Describe the solution you'd like

Provide a way for a reality mesh to specify that it should be treated as terrain, with map layer imagery draped onto it. Allow it to optionally specify that the base layer (e.g. Bing map imagery) should not be applied to it.

Describe alternatives you've considered

None.

Additional context

Global terrain uses the same shaders as reality meshes, just sampling the textures differently.

A "treat as terrain" flag could also be useful when the reality mesh represents a large area of the earth's surface, to help the display system reduce precision issues caused by excessively-expanded frustum planes. On the other hand, we might want to avoid conflating that flag with a narrower "drape map imagery on me" flag.

Consider whether we may one day need to support projecting map imagery onto point clouds and design the API accordingly.

[alarkbentley]

@danieliborra @markschlosseratbentley Any status update / estimates on development for this?

[danieliborra]

We're currently working on this issue (In progress) and it is high in our priority.
No estimates yet.

[markschlosseratbentley]

@alarkbentley As we work on this, it would be very helpful to have a testcase that represents the scenarios you expect to encounter. Basically: a reality model + map layer to project onto it. Do you have anything you can provide us?

cc @MarcNeely @danieliborra

[alarkbentley]

@markschlosseratbentley

Here is an example from iTwin Experience QA.
Select "Groups & Layers" -> "Reality Mesh" and it will try to render a reality mesh with a WMS map layer on-top.
See video.

https://qa-itwinexperience.bentley.com/dcd5e824-d9fe-40db-961f-83fd5ae95484/map?featureFlagOnCsv=should-ocp-use3dtiles&sceneid=db54b083-e71e-43f3-8b4c-d10c5df9059c

maplayer_on_realitymesh_241113.mp4

Expected:

[markschlosseratbentley]

@alarkbentley Thank you! I'm searching through Chrome dev-tools' network tab, trying to find the sources of this data, but not having much luck.

Could you tell us where we can locally download the reality model itself? Or a URL directly to it?

Could you also direct us to a URL of the map layer we would be able to attach in our own app?

Basically, we want to 1) load the reality data in itwinjs-core display-test-app 2) attach the map layer
so we can test our projection development.

[MarcNeely]

I have a command implemented in DTA which can flag the reality mesh so that it drapes the background map onto it. The flag can be turned on or off at will. We need to think about and implement some real API for it though as it currently uses some internal classes to accomplish it.

[alarkbentley]

@markschlosseratbentley sorry for the late answer, missed you replied.

So local copies of this data will be a bit tricky as the data is dynamically generated on request by our backend (original files are not 3dtiles/wms). I assume we don't have a easy way to export this @mborjesson-bentley ?

But the setup is very simple, a RealityMesh with a WMS draped on same location. Do we not have existing examples of this in iTwin.js sandbox or somewhere?

@MarcNeely your update sounds promising! Any screenshot or demo?

[markschlosseratbentley]

But the setup is very simple, a RealityMesh with a WMS draped on same location. Do we not have existing examples of this in iTwin.js sandbox or somewhere?

The challenge for this use case is getting a reality dataset and a map layer that share the same location.

[alarkbentley]

@markschlosseratbentley I guess you could try just dig out the URLs from network tab.
You only need to update the SAS tokens as they will expire

RealityMesh:
https://stratos-beta-eu.opencitiesplanner.bentley.com/a08091ae-bed1-498f-8b1c-e1bbc6ea7232/9d5dc01c-9c80-4d5d-a392-7f46baa6131c/241030_024805_1730256485690/services/3dtiles/default/tileset.json?qs_st=2024-11-21T00%3A34%3A03.579359005Z&qs_ver=201113&qs_sign=a7ICP_qxblm05kyTM4w8A1Lq4LtXZzl7o7sCVAmE7lQ-&qs_res=f&qs_prot=http%2Chttps&qs_end=2024-11-21T12%3A34%3A03.579321795Z

WMS:
https://stratos-beta-eu.opencitiesplanner.bentley.com/a08091ae-bed1-498f-8b1c-e1bbc6ea7232/ab42c147-215b-49cf-a389-fc4a55ffb2aa/241031_055543_1730354143970/services/wms?request=GetCapabilities&service=WMS&qs_st=2024-11-21T00%3A33%3A24.550305001Z&qs_ver=201113&qs_sign=QR8P6..tNDAu2KFe5N_QWZYvdpSWmSZV2whXXBDV9Xk-&qs_res=f&qs_prot=http%2Chttps&qs_end=2024-11-21T12%3A33%3A24.550264445Z