Add prefix cache aware routing #641
Conversation
| end = len(unMatchedTokens) | ||
| } | ||
| chunk := unMatchedTokens[i:end] | ||
| prefixHash := xxhash.Sum64(IntArrayToByteArray(chunk)) |
There was a problem hiding this comment.
here. it just consider the current block size? just like to confirm this is not 100% same as vLLM's solution right? their 1st block hash is part of 2nd hash input
There was a problem hiding this comment.
Yes, hash only for current block. No link list kind of behavior that vllm has. For our usecase we do not need that link list behavior.
There was a problem hiding this comment.
if that case, we need to do up to O(n) calculations? n=number of blocks
There was a problem hiding this comment.
Yes. If you are referring to that link list behavior could prevent O(n) calculations then it wont be the case. Total computations stays the same.
There was a problem hiding this comment.
it's a little bit different. LinkedList you can do binary search etc for optimization. In this way, we can only do O(n).
There was a problem hiding this comment.
Let me look into this, but for our usecase we need to evaluate all blocks to ensure a 50%+ hit rate.
No description provided.