add request routers - least kv cache, least expected latency#543

Merged

Aspirin96 merged 17 commits intomainfrom binbin/router

Jan 3, 2025

+347-4

Commits on Dec 16, 2024

Add random adapter scheduler

ChenBinbin1996
authored and
brosoul
committed
Add leastExpectedLatency request router

ChenBinbin1996
authored and
brosoul
committed
Add least latency scheduler

ChenBinbin1996
authored and
brosoul
committed
Add least kv cache router

ChenBinbin1996
authored and
brosoul
committed
Add bin packing scheduler (first-fit as examole)

chenzuzhi22
authored and
brosoul
committed
Add least utilization scheduler (RPM, TPM, kv_cache, busy_time as utilization)

chenzuzhi22
authored and
brosoul
committed
Add least busy time (or least gpu utilization) router

chenzuzhi22
authored and
brosoul
committed
Add weighted round robin router

chenzuzhi22
authored and
brosoul
committed
Add metrics that scheduling needed (#486 )
brosoul
committed
适配metrics接口

ChenBinbin1996
authored and
brosoul
committed
add .vscode to gitignore

ChenBinbin1996
authored and
brosoul
committed
fix mock cpu_cache_usage_perc metrics
brosoul
committed
feat: add least kv cache into route strategy
brosoul
committed

Commits on Dec 27, 2024

Commits on Jan 3, 2025

Merge branch 'main' into binbin/router
ChenBinbin1996
committed