How to write a multi-label object detection network?

[GuanRunwei]

Hello, I am a student in Uni and I am trying to write a multi-label object detection network. As we all know, an object tends to have many labels in the real world(e.g. a car may have 2 labels: SUV, black), so how to design and develop a multi-label network or system?
From my perspective, the ideal architecture of the detection head is multiple sub-heads for different category classification and one sub-head for bounding box regression. In addition, how to alleviate the conflict and ambiguous of the feature extracted by backbone is also chanllenging for me. Anyone interested in this project could join me.

?

[abhigoku10]

@GuanRunwei hi i think you looking to the problem statement similar to my requirement , i have reference here which others have done for MaskRCCN https://github.com/itmessager/Figma_RCNN
i would be interested to collaborate

[GuanRunwei]

Yeah, as I think about the network architecture for several days, maybe 2-stage is more suitable for multi-label object detection. If you are interested in my project, maybe we can develop together and my email is Runwei.Guan@liverpool.ac.uk, we could communicate with each other for further requirement.

[GuanRunwei]

I looked into the link you put and it is a tensorflow-based network, I think we could write a pytorch-based.

[abhigoku10]

That was just an example that i shared where with two-stage detectors it would be possible, but the challenge is to do it for a single stage network where the following things come into pictures

1. Annotation data: for your example, we can have annotation with a single bounding box value and multiple labels ie xywh man , xywh thief
2. Localisation : how will you make the architecture understand the correlation of single regession box with multiple labels ie xywh should be regress and class is man, thief and policeman and it has select man and thief based on the input image
3. Loss function : what kind of loss cann you use will you weightedbce or nn.multiclass loss

[lucasjinreal]

An easy way is to make a one-hot to all categories. for example, if you have 2 father class and 3 child class of each one, you can manipulate them into a whole 6 categories.

However, this is only applicable to onehot scenarios. If your labels is unmutual, you can directly output multilabel and mark every possible output as positive.

[GuanRunwei]

Sorry, I am confused about "If your labels is unmutual, you can directly output multilabel and mark every possible output as positive.", could you explained it more detailedly? Thank you very much!

[GuanRunwei]

Could YOLOX-based network realize my purpose that one bounding box with multiple labels?

[abhigoku10]

@jinfagang can you please share an example of your explanation /?

[GuanRunwei]

He means multi-hot. one-hot: [[0, 0, 1, 0], [0, 0, 0, 1]] - > multi-hot: [0, 0, 1, 1]