server: allow to override threads server pool with --threads-http

[phymbert]

Motivation

The default httplib threads pool is set to std::thread::hardware_concurrency()
which is NOT always adapted for completions.
As far, completions tasks can be deferred, http server threads pool
can quickly be full by reaching the number of available host cpu cores
while actually continuous batching can accept more requests on the GPU device.

This fix allows the user to choose the balanced number of threads in the http server pool.

Changes

Introduce --threads-http N to override the default configuration.

[phymbert]

@ggerganov Hi, as I got one approval, should I wait for yours ? thanks

[ggerganov]

If a PR touches just the server code, you don't need to wait for my approval.

Btw, isn't it better to reuse the --parallel parameter instead of introducing a new one?

[phymbert]

If a PR touches just the server code, you don't need to wait for my approval.

Noted thanks.

Btw, isn't it better to reuse the --parallel parameter instead of introducing a new one?

No there are different meaning, --parallel for number of sequences/slots on the llama backend, --threads-http for number of concurrent http requests on httplib (which can generate deferred tasks if all slots are processing).