Adding hiera

[Namangarg110]

What does this PR do?

Adds Hiera model from Meta as suggested in #28993

GitHub Repo: https://github.com/facebookresearch/hiera/
arXiv: https://arxiv.org/abs/2306.00989

Model License recently changed to Apache

This PR is to merge the work done by @EduardoPach and me in #29339 and #29945 respectively.

c.c. @amyeroberts

[EduardoPach]

Hey @amyeroberts I've updated the model cards for all hiera checkpoints here could you help out to transfer them to facebook org?

c.c. @amyeroberts

[amyeroberts]

@EduardoPach Sure! Just to confirm, should all of these checkpoints be moved?

[EduardoPach]

@EduardoPach Sure! Just to confirm, should all of these checkpoints be moved?

Yeap, we have one for each (HieraModel, HieraForPreTraining, HieraForImageClassification) and 6 different sizes

[amyeroberts]

Hi @EduardoPach so, it seems that @NielsRogge was handling this and was in contact with people from meta regarding the weights transfers. Niels is coming back from holiday I think next week, so let's wait for him to confirm!

[EduardoPach]

@amyeroberts, @NielsRogge doesn't have permission to move the checkpoints from my account to the facebook orq and I was talking to @dbolya and he is only a part of facebookresearch org and therefore can't add Niels or me as contributors to the org. Could someone with the appropriate permissions just transfer the checkpoints from my account to the Facebook org? (or perhaps facebookresearch idk)

[Namangarg110]

@amyeroberts Any updates regarding the same ?

[NielsRogge]

Hi both, finally got to transfer the checkpoints :) they're now all here: https://huggingface.co/models?other=hiera&sort=created

[Namangarg110]

Thankyou @NielsRogge. @amyeroberts, is there anything else required from our side ?

[amyeroberts]

@EduardoPach @Namangarg110 Two final updates I can see needed before merge:

• Updating checkpoints for tests
• Removing imports in model_name/__init__.py for objects which don't exist

[amyeroberts]

Checkpoints here need to be updated too

[EduardoPach]

190c662

[EduardoPach]

@amyeroberts fixed the checkpoints and cleaned the __init__.py the failing test seems unrelated to Hiera

[amyeroberts]

@EduardoPach Great! The last thing is to have a run of the slow tests before merge. Could you push an empty commit with: git commit --allow-empty -m "[run_slow] hiera"? This will trigger a run of the slow integration tests for the model. I'll need to approve the workflow after the commit. Once it's all green we're good to merge!

[Namangarg110]

Thank you for your response, @amyeroberts. I have pushed the empty commit, and all tests are green.

[amyeroberts]

@Namangarg110 It seems there's some slow tests which fail and need to be addressed before merge:

Single-GPU tests

FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_attention_outputs - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_batching_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_determinism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_hidden_states_output - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_model_from_pretrained - OSError: We couldn't connect to 'https://huggingface.co/' to load this file, couldn't find it in the cached files and it looks like facebook/hiera-tiny-224-hf is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_model_outputs_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_save_load - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx_output_loss - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant_false - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_for_pretraining - AssertionError: False is not true
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_fp16 - OSError: facebook/hiera-tiny-224-hf does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.

Multi-GPU tests

FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_attention_outputs - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_batching_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_determinism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_hidden_states_output - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_model_outputs_equivalence - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_save_load - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_torch_fx_output_loss - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelTest::test_training_gradient_checkpointing_use_reentrant_false - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA_gather)
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_for_pretraining - AssertionError: False is not true
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_fp16 - ValueError: HieraModel does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_image_classification_head - RuntimeError: Float did not match Half
FAILED tests/models/hiera/test_modeling_hiera.py::HieraModelIntegrationTest::test_inference_interpolate_pos_encoding - RuntimeError: Input type (float) and bias type (c10::Half) should be the same
FAILED tests/models/hiera/test_modeling_hiera.py::HieraBackboneTest::test_backbone_outputs - RuntimeError: Input type (float) and bias type (c10::Half) should be the same
FAILED tests/models/hiera/test_modeling_hiera.py::HieraBackboneTest::test_backbone_stage_selection - RuntimeError: Input type (float) and bias type (c10::Half) should be the same
FAILED tests/models/hiera/test_modeling_hiera.py::HieraBackboneTest::test_create_from_modified_config - RuntimeError: Input type (float) and bias type (c10::Half) should be the same

[EduardoPach]

c.c. @amyeroberts

[amyeroberts]

Thanks for addressing all the slow tests - we're good to merge!