bug: Free the allocated tokens in the batch

[irbull]

The llama_batch_init allocates memory for a fixed number of tokens. However, the llama_batch_free only frees memory for the number of tokens that were added to the batch.

This change-set tracks the size of the batch structure and frees the all the allocated memory.

[ggerganov]

Thanks for catching that

Instead of adding n_alloc_tokens, should we make seq_id to be null-terminated? This way we'll avoid changing the public struct and introducing what is essentially private data.

In the long run, llama_batch would probably have to be just forward declared and accessed via an interface.

[irbull]

I've updated the commit to use a null terminated array. I added an extra element to the seq_id array and assigned the last value to nullptr. During the free, I loop through until we find the nullptr element.

I agree, not updating the public API is a +1. I did change the name of the first parameter of llama_batch_init to n_alloc_tokens as n_tokens is a very overloaded term in Llama.cpp, and I thought this made it clear that this isn't the number of tokens in the batch.

[ggerganov]

Ok makes sense. Just a tiny nit: change the name to n_tokens_alloc

[irbull]

Ok makes sense. Just a tiny nit: change the name to n_tokens_alloc

+1, sounds good. Done!

[irbull]

Is there typically build errors? This seems to be a timeout of some sort (after 5h) on the Mac build running. cmake on my M1 seems to work, but I'm sure the server setup is different. I'm not sure if this is a transient error, or if this is an indication of a real problem.

[ggerganov]

I think it's the Github Actions acting up - should be ok