changing dimensions of batch size, kv cache and num_input_heads #793
Conversation
…genai into ly/handling_tensors_access_in_genai
…genai into ly/handling_tensors_access_in_genai
…com/lamiayous/openvino.genai into ly/handling_tensors_access_in_genai
…genai into ly/handling_tensors_access_in_genai
|
jenkins_build |
|
jenkins_build |
|
Failing checks are caused by broken master. Ignore them until #807 is merged. |
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
| KVAxesPosition axes; | ||
| if (model_type == "chatglm") { | ||
| axes.batch = 1u; | ||
| axes.seq_len = 0u; |
There was a problem hiding this comment.
why this approach https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/prompt_lookup_decoding_lm/prompt_lookup_decoding_lm.cpp#L13 is not used? it seems generic and does not required model type
There was a problem hiding this comment.
@ilya-lavrenov Yes, you're right. The only drawback is that this function cannot be used for .blob's case as they will be stateless + static.
There was a problem hiding this comment.
how is it supposed to pass compiled blob via current LLMPipeline API ?
There was a problem hiding this comment.
It's on review so far: #811
There was a problem hiding this comment.
The current solution aim to handle qwen / chatglm cases that are crucial for now.
But in general, I'd prefer using your approach but somewhere inside StatefulToStateless transformation, so that it could save necessary metadata that will be available from both xml / blob formats.
…com/lamiayous/openvino.genai into ly/handling_tensors_access_in_genai
|
build_jenkins |
Once KV-cache tensors are exposed from the stateful model, they should be reshaped to have static size. Current implementation of reshape function assumes that KV-cache dimension is always equal to 2 and batch dimension always equal to 0. For chatglm and Qwen this is not the case. This PR identifies the KV-cache and batch dimensions by reading the models config.json file