Fix loading models with mismatched sizes

[qubvel]

What does this PR do?

Fix loading model with mismatched sizes, the bug was introduced since:

• Update form pretrained to make TP a first class citizen #36335

Reported in

• Add D-FINE Model into Transformers #36261

Fixes: #36464

Minimal reproducing example:

from transformers import AutoModelForObjectDetection

model = AutoModelForObjectDetection.from_pretrained(
    "PekingU/rtdetr_r18vd",
    id2label={0: "person"},
    label2id={"person": 0},
    ignore_mismatched_sizes=True
)

cc @ArthurZucker

[qubvel]

strict=False does not allow loading weights with mismatched sizes, filter those keys first

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[muellerzr]

Thanks! This is a much better solution than the try/catch I initially was doing.

[ArthurZucker]

Super nice fix merging to get this asap