How does Hail treat missing calls?

[iris-garden]

Note

The following post was exported from discuss.hail.is, a forum for asking questions about Hail which has since been deprecated.

(Oct 02, 2023 at 21:49) akhattab said:

Hello, I’m trying to understand how Hail treats the missing calls.
When using mt.GT.show I could see the missing calls as NA:

But when using hl.is_missing, they are still NA instead of True:

Same when using is_defined:

My question is, why are these calls ‘ignored’?
Thank you!

[iris-garden]

Note

The following post was exported from discuss.hail.is, a forum for asking questions about Hail which has since been deprecated.

(Oct 04, 2023 at 14:20) danking said:

Hey akhattab ,

Yeah, this is an unfortunate and confusing part of the Hail interface, particularly as it relates to show. Those NA values are not missing but “filtered”. Filtered entries are produced with filter_entries. This is the entry analogue to row or column filtering, but unlike row or column filtering, we can’t simply remove the entire row or column (because there are other not filtered entries at the same row or column).

You can convert filtered entries to missing entries with mt.unfilter_entries(). You can compute statistics about the number of filtered entires with mt.compute_entry_filter_stats.

There is no way to test for the filtered status of a particular field, because its not the field that is filtered, it is the entry of the matrix that is filtered.

By convention:

• “Filtered” entries are like filtered rows or columns: we want to pretend that we didn’t even observe them. They’re not part of our dataset anymore.
• “Missing” values are data we measured but we’re uncertain of its value. We want to consider them as a part of our analysis but as a separate third category of thing: something we know exists but we can’t observe directly.