Tokenization with truncation, offset support and tests

[fialhocoelho]

• Support for Truncation and Offset: The tokenize() function has been edited to include support for truncation and offset.
• New Tests: Tests have been added to evaluate the tokenization output, focusing on truncation and offset functionality to ensure replicability.
• Makefile Addition: A Makefile has been added to facilitate the generation of necessary files for gRPC import for vllm/tests/test_server_tokenize_truncate.py
• Direct Use of tokenizer.encode_plus(): The tokenizer.encode_plus() function is used directly to avoid modifying the tokenizer_group.encode_async function, which is part of the upstream code/project. This ensures compatibility and maintains the integrity of the upstream codebase. This implementation won't affect performance because tokenizer_group.encode_async doesn't run Tokenizer.encode in a background thread.

[maxdebayser]

@njhill, here the code is calling encode_plus directly because adding an async wrapper would require changing the vLLM code. But I don't think it has a performance impact because the actual encoding in encode_async not offloaded to a thread pool or anything:

    async def encode_async(
            self,
            prompt: str,
            request_id: Optional[str] = None,
            lora_request: Optional[LoRARequest] = None) -> List[int]:
        tokenizer = await self.get_lora_tokenizer_async(lora_request)
        ret = tokenizer.encode(prompt)
        self._raise_if_input_too_long(ret, lora_request)
        return ret

[joerunde]

for 515

[joerunde]

@prashantgupta24 can you help coordinate getting this into the tgis adapter?

[prashantgupta24]

Suggested change

	service_metrics.observe_tokenization_request(request)
	service_metrics.count_tokenization_request(request)

[prashantgupta24]

I'm going to be making this change in the adapter repo

[njhill]

Closing since this was ported to the vllm-tgis-adapter repo. Thanks @fialhocoelho.