Broken Scenes

[teufelweich]

I've noticed that there are scenes with pixels belonging to the class person. I took a closer look to theses scenes and discovered some broken renderings.

ai_004_009/images/scene_cam_01_final_preview contains completely broken images. Seems like there is a light missing in that scene. But even the textures don't make any sense. This scene confuses even me, as a human. 😆

frame.0001.diffuse_reflectance.jpg

frame.0001.tonemap.jpg

ai_038_007/images/scene_cam_00_final_preview is missing textures completely. The floating rectangles 👻 are labeled as persons.

frame.0014.tonemap.jpg

frame.0014.semantic.png

These are the first scenes I looked into. Both of them are included in the train split and are detrimental for training in my opinion.
Are you aware of such issues? How many more broken scenes could there be?

Despite this, thanks for your work.

[mikeroberts3000]

Hi! Great questions.

You're correct that {ai_004_009, cam_01} is broken. Thank you for flagging this problem! What's happening here is that the artist who originally created this scene accidentally placed a camera inside the bed. That's why the tonemap images are dark, but the but the diffuse_reflectance images still have vaguely plausible-looking textures.

In our computational pipeline, we assume that all artist-created cameras frame the scene in a meaningful way, and we use these artist-created cameras as a form of weak supervision to help us figure out how to generate more good views of each scene. But our assumption isn't always correct. Some artist-created cameras do not frame the scene in a meaningful way, e.g., {ai_004_009, cam_01}. In these cases, we manually filter the useless images from our training set. You can see many examples of this manual filtering step if you search for OUTSIDE VIEWING AREA (BAD INITIALIZATION) in our split file. But {ai_004_009, cam_01} must have slipped through the cracks, and I didn't filter it out, even though I should have. The good news is that this is a very occasional problem, and it is possible to obtain very strong sim-to-real results using Hypersim, despite occasionally encountering these weird images. If you find any other camera trajectories that are like this (i.e., the entire camera trajectory is pitch black or weird-looking), or if you obtain a measurable performance improvement on any computer vision task by excluding these trajectories, please do post here.

While we're on the subject of ai_004_009, are you sure that this scene contains objects that are labeled as people?

You are also correct that ai_038_007 is rendered without textures. When I examined the native source content for this scene (i.e., downloaded directly from a content marketplace), it does appear to contain textures because there are a bunch of JPG files that look like textures included with the scene. But for some reason, this particular scene is configured to render without textures, and I couldn't figure out how to switch texture rendering back on. Despite its obvious limitations, I decided to include this scene because I conjectured that a neural network may still be able to learn something useful about the visual world from this scene. Again, the good news is that this is a very occasional problem that affects only two or three scenes. If you obtain a measurable performance improvement on any computer vision task by excluding these scenes, please do post here.

[teufelweich]

That makes sense. Thanks for the explanation.
Shouldn't be that hard to find pitch black images, the sum of all RGB pixels in {ai_004_009, cam_01} is 0. But I guess you already did some kind of filtering like that. Anyway I'm currently running a simple search for more pitch black images. Will report here.

Regarding the person label: You're right, this images has no person label, it only contains ['floor', 'bed', 'bookshelf', 'pillow']. I don't know why my script spit out that image, maybe it was an off by one error.

As of ai_038_007 I'm not sure how useful this scene really is. Big portion of this scene is void/unlabeled, the ceiling consisting of glass windows is labeled as wall, though a ceiling label is used in the dataset. But In relation to the enormous size of the dataset the influence of this scene may be neglectable, unless it is present in the last batch of an epoch. Do you have a list, which scenes also suffer from this problem? Or is the information about rendering issues stored in the metadata?

I've also took a look on the depth distribution after correcting it with the code from #9 :

⌀ is the mean depth and σ it's standard deviation (excluding depths >= 2**16 mm because I'm using dtype=uint16)

My explanation for the large spikes in val and test split are scenes, where the camera faces a wall perpendicular to it. Do you think this could be a sign for more broken scenes?

[mikeroberts3000]

I agree that ai_038_007 isn't very useful for segmentation tasks, but Hypersim can also be used for other computer vision tasks beyond segmentation.

Unfortunately I don't have a list of scenes affected by this texture rendering issue, but I know there are at most two or three such scenes. You'll be able to find the others by looking at the first image for each scene. There are only 461 scenes, so you won't need to do too much manual inspection.

Those depth distributions look reasonable to me. Your interpretation of the spikes also sounds reasonable. There must be occasional frames where the camera is exactly aligned with a wall. I'm sure this phenomenon occasionally happens in the training set too, but perhaps it's less noticeable in your plots because the training set is bigger.

I think this phenomenon can arise naturally when an artist-placed camera happens to be axis-aligned, and the walls in the scene also happen to be axis-aligned. And it is always possible (but unlikely) that one of our randomly generated views aligns perfectly with a wall. So this phenomenon is not necessarily indicative of a broken scene or camera trajectory or frame. That being said, it seems plausible to me that a perfectly axis-aligned camera is more likely to have been accidentally spawned by the artist, and less likely to have been carefully placed by the artist to frame the scene in a particular way. Therefore, when a camera is perfectly axis-aligned, perhaps it is a hint that the camera does not convey artistic intent. Of course, this reasoning only holds if the walls in the scene are also axis-aligned, but they usually are.

[teufelweich]

My simple search found another pitch black scene in the training split: (ai_003_001, cam_00)
In the test and val split I've found none.

Thats true, I completely forgot that there are other tasks apart from the ones I'm working on 😅

That's a good explanation.
Maybe I could search for these monotone depth images with a heuristic like np.std(depth) < 1m. A straight faced wall without objects in the frame would likely render it impossible to estimate the depth of it. I should try a histogram of mean and std of every image.

[mikeroberts3000]

Good catch! So the pitch black camera trajectories we've found so far are as follows:

ai_003_001, cam_00
ai_004_009, cam_01

While you were searching for pitch black camera trajectories, I manually inspected the scenes to look for that texture rendering problem. The scenes I found with the texture rendering problem are as follows:

ai_015_006
ai_038_007
ai_046_001
ai_046_009
ai_048_004 (all the objects in this scene are assigned a seemingly random bright color)
ai_053_005

I'll update these lists if we find any more. Thanks for helping to dig into these issues 😄

[teufelweich]

We've found more broken scenes. This time with black segmentation images or depths.

        'ai_012_007': '*',  # train (no depth)
        'ai_013_001': '*',  # train (no depth)
        'ai_023_008': 'cam_01',  # train (no labels)
        'ai_026_020': '*',  # train (no labels)
        'ai_023_009': '*',  # train (no labels)
        'ai_038_007': '*',  # train
        'ai_023_004': 'cam_01',  # train (no labels)
        'ai_023_006': '*',  # train (no labels)
        'ai_026_013': '*',  # train (no labels)
        'ai_026_018': '*',  # train (no labels)
        # 'ai_044_004': 'cam_01',  # test (no labels)

[mikeroberts3000]

Good catch, thank you!