Fix torch.fx symbolic tracing for LLama

[michaelbenayoun]

What does this PR do?

As per title.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Thanks for fixing!
Should work! Let's fix other models as well

[ArthurZucker]

I guess passing cache_position[-1] doesn't work? The only issue is that past_seen_tokens s gonna be deprecated in favor of cache positions, but that works since with static cache we use the max positions embeddings

[ArthurZucker]

Last thing, can you run the slow tests RUN_SLOW=1 pytest tests/models/llama on cuda?

[ArthurZucker]

====================================== short test summary info =======================================
SKIPPED [1] ../miniconda3/envs/py39/lib/python3.9/unittest/case.py:117: TODO @gante fix this for Llama
SKIPPED [1] tests/test_pipeline_mixin.py:327: LlamaModelTest::test_pipeline_audio_classification is skipped: `audio-classification` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:331: LlamaModelTest::test_pipeline_automatic_speech_recognition is skipped: `automatic-speech-recognition` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:335: LlamaModelTest::test_pipeline_conversational is skipped: `conversational` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:339: LlamaModelTest::test_pipeline_depth_estimation is skipped: `depth-estimation` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:346: test requires PyTesseract
SKIPPED [1] tests/test_pipeline_mixin.py:357: LlamaModelTest::test_pipeline_fill_mask is skipped: `fill-mask` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:361: LlamaModelTest::test_pipeline_image_classification is skipped: `image-classification` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:379: LlamaModelTest::test_pipeline_image_feature_extraction is skipped: `image-feature-extraction` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:367: LlamaModelTest::test_pipeline_image_segmentation is skipped: `image-segmentation` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:374: LlamaModelTest::test_pipeline_image_to_text is skipped: `image-to-text` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:386: `run_pipeline_test` is currently not implemented.
SKIPPED [1] tests/test_pipeline_mixin.py:393: LlamaModelTest::test_pipeline_object_detection is skipped: `object-detection` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:404: LlamaModelTest::test_pipeline_summarization is skipped: `summarization` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:408: LlamaModelTest::test_pipeline_table_question_answering is skipped: `table-question-answering` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:412: LlamaModelTest::test_pipeline_text2text_generation is skipped: `text2text-generation` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:425: LlamaModelTest::test_pipeline_text_to_audio is skipped: `text-to-audio` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:430: LlamaModelTest::test_pipeline_token_classification is skipped: `token-classification` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:434: LlamaModelTest::test_pipeline_translation is skipped: `translation` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:438: LlamaModelTest::test_pipeline_video_classification is skipped: `video-classification` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:445: LlamaModelTest::test_pipeline_visual_question_answering is skipped: `visual-question-answering` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:455: LlamaModelTest::test_pipeline_zero_shot_audio_classification is skipped: `zero-shot-audio-classification` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:460: LlamaModelTest::test_pipeline_zero_shot_image_classification is skipped: `zero-shot-image-classification` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/test_pipeline_mixin.py:465: LlamaModelTest::test_pipeline_zero_shot_object_detection is skipped: `zero-shot-object-detection` is not in `self.pipeline_model_mapping` for `LlamaModelTest`.
SKIPPED [1] tests/models/llama/test_modeling_llama.py:371: Llama buffers include complex numbers, which breaks this test
SKIPPED [1] tests/models/llama/test_modeling_llama.py:657: Model is curently gated
SKIPPED [1] tests/models/llama/test_modeling_llama.py:615: Logits are not exactly the same, once we fix the instabalities somehow, will update!
SKIPPED [1] tests/models/llama/test_modeling_llama.py:628: Logits are not exactly the same, once we fix the instabalities somehow, will update!
SKIPPED [1] tests/models/llama/test_modeling_llama.py:641: Logits are not exactly the same, once we fix the instabalities somehow, will update! Also it is gonna be a `too_slow` test
SKIPPED [1] tests/models/llama/test_modeling_llama.py:602: Logits are not exactly the same, once we fix the instabalities somehow, will update!
================= 1 failed, 120 passed, 30 skipped, 23 warnings in 143.75s (0:02:23) =================

LGTM thanks for taking the time to fix it