optimize benchmark scripts for autoscaler, add more logs #356
Conversation
benchmarks/autoscaling/benchmark.py
Outdated
| ) | ||
| def setup_logging(log_filename, level=logging.INFO): | ||
| """ | ||
| 设置全局日志配置，日志将被写入指定的文件，并同时输出到控制台。 |
There was a problem hiding this comment.
let's remove non English characters in github repo. the policy is a different bit different from our internal repo
There was a problem hiding this comment.
Corrected it. thanks.
| @@ -3,6 +3,7 @@ | |||
| import os | |||
| import random | |||
| import time | |||
| from datetime import datetime | |||
There was a problem hiding this comment.
@happyandslow Can you help review the benchmark changes?
benchmarks/autoscaling/kpa.yaml
Outdated
| @@ -11,8 +11,8 @@ spec: | |||
| apiVersion: apps/v1 | |||
| kind: Deployment | |||
| name: aibrix-model-deepseek-coder-7b-instruct | |||
| minReplicas: 1 | |||
| minReplicas: 2 | |||
There was a problem hiding this comment.
why do we change to 2 here?
There was a problem hiding this comment.
@Jeffwan
it's to align with hpa.
When our experiment started, the initial replicas of the llm service were set to 2, and we would wait for 30 seconds before adding traffic. We found that KPA would quickly reduce the replica to 1 as beginning, but HPA would not immediately scale down even without traffic. This would lead to an unfair experiment.
Should we uniformly set the min-replica of HPA.yaml and KPA.yaml to 2? I will modify the hpa.yaml to keep it consistent with the kpa.yaml.
There was a problem hiding this comment.
talked with @kr11 offline on this. We determined to change to 1.
| @@ -202,6 +202,14 @@ func (k *KpaAutoscaler) Scale(originalReadyPodsCount int, metricKey metrics.Name | |||
|
|
|||
| isOverPanicThreshold := dppc/readyPodsCount >= spec.PanicThreshold | |||
|
|
|||
| klog.InfoS("--- KPA Details", "readyPodsCount", readyPodsCount, | |||
There was a problem hiding this comment.
is this a debug message, if so, let's use V(4)
There was a problem hiding this comment.
is this a debug message, if so, let's use
V(4)
I have fixed it together with two other debug klogs.
* optimize workload scripts and result output * add more logs. resync period: 30->10, fix kpa.yaml * fix lint * add right klog level. unify min-replica in hpa.yaml and hpa.yaml * unify hpa and kpa yaml min-replica to 1
Pull Request Description
After optimizing some KPA configurations, KPA shows faster upscaling and better replica decisions, resulting in lower latency and higher throughput.
The main changes are:
Related Issues
Resolves: #[Insert issue number(s)]
Important: Before submitting, please complete the description above and review the checklist below.
Contribution Guidelines (Expand for Details)
We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:
Pull Request Title Format
Your PR title should start with one of these prefixes to indicate the nature of the change:
[Bug]: Corrections to existing functionality[CI]: Changes to build process or CI pipeline[Docs]: Updates or additions to documentation[API]: Modifications to aibrix's API or interface[CLI]: Changes or additions to the Command Line Interface[Misc]: For changes not covered above (use sparingly)Note: For changes spanning multiple categories, use multiple prefixes in order of importance.
Submission Checklist
By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.