Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAJOR - Adapt code to use original vitpose checkpoints and multidataset #18

Merged
merged 21 commits into from
Sep 23, 2023

Conversation

JunkyByte
Copy link
Owner

@JunkyByte JunkyByte commented Sep 21, 2023

This is the initial draft PR. Initial reference #8.

idea

Adapt the code to be able to run inference using all the original vitpose models provided by
https://github.com/ViTAE-Transformer/ViTPose
To do so model_split.py is used to separate the multidataset checkpoints into single dataset for inference.

This should allow to run inference using all the original models on the datasets:

  • coco
  • coco + feet (coco_25) (my finetuned checkpoint as in old version)
  • mpii
  • ap10k (animals!)
  • apt36k (animals!)
  • aic

As we also add the support for animal pose estimation we need to adapt the detection phase to non human categories. To do so right now the idea is to simply specify the dataset or coco class to detect, these are the available ones:

  0: person
  14: bird  # NOT WORKING IN APT36k and AP10k, will be removed.
  15: cat
  16: dog
  17: horse
  18: sheep
  19: cow
  20: elephant
  21: bear
  22: zebra
  23: giraffe

I don't think all of them will work during pose estimation but this is the initial idea.
Another solution might be to find a custom trained version specifically for animal detection, I found https://github.com/SaiSwarup27/Animal-Intrusion-Detection which might be promising and uses yolov5 as well. (this will be addressed later)
It might be interesting to allow to specify an external detection process, especially when using VitInference directly in a script.

Some changes to json saving / inference pipeline and configuration are required.
Also the skeletons must be added one by one to the json_dict object to setup visualization.

For animals it still feels like integrating a different model might lead to better results.

  • Configs (WIP)
  • Inference pipeline (WIP)
  • json saving
  • visualization
  • setup train.py to work with new config system
  • provide all the new models (also in trt / onnx)

…l. Generated new checkpoints with corrected model_split, other minor changes.
@JunkyByte
Copy link
Owner Author

The configs present a lot of repeated code, but still they are in better shape than previously. I don't feel they are that important, they are easy to clone now if one wants to experiment fine tuning.

@JunkyByte JunkyByte merged commit d3c5d5c into main Sep 23, 2023
@JunkyByte JunkyByte deleted the dev branch September 23, 2023 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant