Feature: Ghost error & error pcts (#2) #624
Conversation
|
This pull request has been linked to Shortcut Story #5478: [DQ] Add ghost errors to pred polygons. |
Codecov Report
@@ Coverage Diff @@
## main #624 +/- ##
==========================================
- Coverage 84.64% 84.50% -0.14%
==========================================
Files 163 164 +1
Lines 12900 12923 +23
==========================================
+ Hits 10919 10921 +2
- Misses 1981 2002 +21
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| @@ -18,6 +18,7 @@ class SemSegCols(str, Enum): | |||
| class ErrorType(str, Enum): | |||
| classification = "classification" | |||
| undetected = "undetected" | |||
| ghost = "ghost" | |||
| def polygon_accuracy( | ||
| preds: np.ndarray, gold_mask: np.ndarray, correct_class: int | ||
| ) -> Tuple[float, Optional[int]]: | ||
| """Calculates the accuracy of one ground truth polygon | ||
| accuracy = (number of correct pixels) / (number of pixels in polygon) | ||
|
|
||
| :param preds: argmax of the prediction probabilities | ||
| shape = (height, width) | ||
| :param gold_masks: ground truth masks | ||
| shape = height, width) | ||
| :param correct_class: the correct class of the polygon | ||
|
|
||
| returns: pixel accuracy of the predictions | ||
| """ | ||
| relevant_region = gold_mask != BACKGROUND_CLASS |
There was a problem hiding this comment.
just moving this fn a little lower
| @@ -146,6 +148,7 @@ def get_polygon_data( | |||
| golds.append(polygon.label_idx) | |||
| data_error_potentials.append(polygon.data_error_potential) | |||
| errors.append(polygon.error_type.value) | |||
| error_pcts.append(polygon.error_pct) | |||
There was a problem hiding this comment.
Just to confirm: gold won't have any overlap with preds, right? Else we'll see duplicates for the same polygon.
| @@ -18,6 +18,7 @@ class SemSegCols(str, Enum): | |||
| class ErrorType(str, Enum): | |||
| classification = "classification" | |||
| undetected = "undetected" | |||
| ghost = "ghost" | |||
There was a problem hiding this comment.
Is this how these are errors are referred to in research?
There was a problem hiding this comment.
Good question. I was wondering the same - check the #cv-taskforce slack channel i just started a discussion there!
Also - in a follow up PR i am renaming this to "background" and renaming "undetected" to "missed" for consistency with OD
In DQ We are testing live for this sprint to get a run in sandbox 😅 It's not being used by customers and isolated code, so for now no risk. And then dedicating time after for robust testing |
https://app.shortcut.com/galileo/story/5478/dq-add-ghost-errors-to-pred-polygons
https://app.shortcut.com/galileo/story/5481/dq-add-error-pct-to-polygons-so-that-we-can-threshold-errors-on-the-api-side