RoPE: fix back, CUDA support for back + noncont. #11240
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the backward pass for RoPE. On master
test-backend-ops grad
is failing on a related assert. The backwards pass can be constructed relatively simply by just creating a tensor for the forward pass and then changing the op fromGGML_ROPE
toGGML_ROPE_BACK
. One could maybe setggml_tensor.op_params
instead of the op but I don't think that would reduce the overall complexity.This PR also adds CUDA support for the RoPE backwards pass and for non-contiguous inputs. The latter is needed for the backwards pass of the KV cache. I also added
__restrict__
andconst
where applicable and simplified the templating a bit.Implicitly
test-backend-ops
is already testingGGML_OP_ROPE_BACK
via gradients, I also added an explicit test.