Clarifications on how target labels are used during training (especially in RPN loss)

[FlorentF9]

Hi,

Thanks for this repo! I have re-implemented your method in detectron2 based on the paper and this code, and currently trying to reproduce the results. For now, they are a bit lower than reported in your paper (about 38 AP50, vs 44 in the paper). I'm trying to figure out where is the difference.

I have trouble understanding how you manage the target labels during the training, especially for the RPN loss. As far as I understand:

1. In the dataset builder, you have added a "is_source" flag to source data samples.
2. This flag is later used to generate "domain_labels" masks for the proposal, with a 1 for source and a 0 for target samples.
3. In the ROI Box Head loss computation (detector_losses), the target class/regression targets seem to be effectively masked out at line 193 in maskrcnn_benchmark/modeling/roi_heads/box_head/loss.py:

domain_masks = cat([proposal.get_field("domain_labels") for proposal in proposals], dim=0)

class_logits = class_logits[domain_masks, :]
box_regression = box_regression[domain_masks, :]
labels = labels[domain_masks]
regression_targets = regression_targets[domain_masks, :]

4. However, I don't understand what happens in the RPN loss (proposal_losses). By looking at the code, in line 114 in maskrcnn_benchmark/modeling/rpn/loss.py:

masks = torch.cat(masks, dim=0)

this variable masks is not being used at all in the function. Does it mean that the RPN is also being trained on the target labels ?

It would be great if you could spend some time to clarify this point!
Cheers