ORTOptimizer support ORTModelForCausalLM

[fxmarty]

As per title.

Notes:

• longt5 results in segmentation fault with O4.
• gpt2 / gptj / etc with cache and O4 (fp16) results in:

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Concat node. Name:'/transformer/h.0/attn/Concat_3' Status Message: /onnxruntime_src/onnxruntime/core/framework/op_kernel.cc:83 virtual OrtValue* onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape&) status.IsOK() was false. Shape mismatch attempting to re-use buffer. {1,4,11,8} != {1,4,12,8}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model.

Did you already witness this @echarlaix @michaelbenayoun @JingyaHuang ?

Before submitting

• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[michaelbenayoun]

Might be linked to the issue with shape inference.

[michaelbenayoun]

LGTM

[fxmarty]

@michaelbenayoun ONNX Runtime have some scripts where they use half precision, I'm not sure if it is with past key values though. I'll have a look.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.