Issues with Data

[mlepori1]

Stumbled upon this while training the network.

[batson]

Check the range of values in each image when you see something weird like this. I would modify plot_tensors to have a flag allowing you to clamp images to zero below and also to plot all four with the same max intensity. Grey in one image may be black in another because matplotlib by default scales each image so min is black and max is white.

…

On Monday, July 15, 2019, mlepori1 ***@***.***> wrote: [image: Screen Shot 2019-07-08 at 11 58 50 PM] <https://user-images.githubusercontent.com/25048682/61264280-95dd2b00-a751-11e9-91a8-12d30d6ea8e5.png> Stumbled upon this while training the network. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=AAJT42VTIWJC2R4KAPMPIPDP7U7ONA5CNFSM4ID4XQR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G7LWNLA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJT42WS7SQSLVQVWFYIB63P7U7ONANCNFSM4ID4XQRQ> .

[bnord]

We may also want to be flagging these for removal before doing anything with them.
@mlepori1 are these nan's in the middle of the image?

[kadrlica]

@mlepori1 Are these being caused by the edge images that I documented in my note on slack? Are you applying the flag selection I described there? I'm guessing the gray region is np.nan values outside the edge of the exposure (you can see some real data on the left side).

[aruba19th]

I think now it's quite clear that the abnormally high pixel values in that thin line are causing such visualization in plots. I tried to exclude by examining the max pixel value. But there are certainly some normal-looking images that have extremely high maximum pixel value. I wonder if we should exclude these ones as well. Next, I will try to see their connection with flagging and try some new methods.

[kadrlica]

I don't think you can exclude images based on the max pixel values. There are several astronomical/detector artifacts that will create small numbers of very bright pixels. Throwing out an entire image due to small artifacts is unacceptable for the science. We may need to also send you masks...

[aruba19th]

Well, the original plan was to pick out the suspicious ones with max value and plot them. Then exclude them manually. But since there are so many of them, I am indeed thinking of other approaches.