[RFC]: Cost-efficient LLM Serving with GPU Heterogeneity

[Jeffwan]

Summary

As the demand for large-scale model serving increases, ensuring consistent GPU availability has become a challenge, particularly within regions where identical GPU types are often unavailable due to capacity constraints. This shortage necessitates the use of heterogeneous GPUs to support the same model deployment, such as mixing L40 GPUs with A800s. Additionally, for latency-insensitive applications, users may seek to incorporate lower-cost, lower-performance GPUs to reduce expenses. However, this approach introduces complexities in determining the optimal GPU combinations and routing strategies to balance performance and cost.

Key Features:

1. Model Profiling for Performance Estimation: A profiling framework to predict model performance across various GPU types, enabling informed decisions about resource allocation.
2. Custom Autoscaler for Heterogeneous Scaling: A tailored autoscaler that manages scaling across deployments and GPU types, ensuring efficient utilization of heterogeneous resources.
3. Heterogeneous Load-Balancing Algorithm: A load-balancing strategy that optimally distributes requests across GPUs with differing computational capabilities, maximizing efficiency while balancing requests.

Motivation

No response

Proposed Change

No response

Alternatives Considered

No response