[WIP]Add ViTPose to Transformers

[shauray8]

What does this PR do?

Adds ViTPose to Huggingface/Transformers

Code and weights: https://github.com/ViTAE-Transformer/ViTPose
Paper: https://arxiv.org/abs/2204.12484

Fixes #24915

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@amyeroberts

[shauray8]

@amyeroberts weights for ViTPose are in PyTorch checkpoint files, should I convert them into .bin or something else?

[amyeroberts]

@shauray8 You will want to write a conversion script, which translates the names of the weights in the state dict to their equivalent in transformers. These weights will then be loaded into the transformers model. Then we save the transformers model using model.save_pretrained(...) which will save out the weights in the desired format (safetensors) as well as all other necessary files such as the model config.

For this model, because the encoder is ViT, the translation of these weights will follow a similar pattern to the conversion script here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/convert_vit_timm_to_pytorch.py

Similarly, the encoder structure can be implemented directly by using #Copied from statements to copy the ViT architecture in the modeling file.

If you haven't seen it already, I'd also suggest looking through this doc as it covers in detail all the steps for adding a model to transformers.

[shauray8]

Thank you for the information and the link to the conversion script for ViT.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[amyeroberts]

@shauray8 Are you still actively working on this model addition? If you don't have the capacity to address it at the moment, we could open up to someone else in the community to take over. Let us know if you have any questions about the implementation.

[shauray8]

Hi, @amyeroberts I'm sorry for not keeping you informed on this, it was a little time-consuming transferring everything from mmcv especially, I think I've done most of it correctly, But it does not work as it should, the only thing I can think of is I have not transferred the weights correctly (I might be totally wrong though), If that's the case maybe I can still find some time and fix it.

[amyeroberts]

@shauray8 Let us know if you need help in getting it across the line.

When you say that it's not working well - what tests are you running to confirm this? If you pass the same image to the mmcv implementation and this one, do you get the same outputs? If not, how about the same activations in the first layer? Debugging like this will help you pinpoint where the issue might be.

[shauray8]

Yes, I did compare to the results from the original code, that's the only test I did, I'll try debugging it step by step as you said and let you know If I need any help, maybe I'll try doing at faster this time

[shauray8]

@amyeroberts While debugging I couldn't find any non-trivial differences, opening it for the community again.