Enhance the model adapter stability to alpha grade #136

Jeffwan · 2024-09-06T21:50:48Z

🚀 Feature Description and Motivation

model adapter controller lacks off enough tests coverage
Some behaviors like the spec.baseModel and secretRef seems not that smooth yet -> we need Model abstraction later
per lora scheduler name may not work as well, it could be the controller level configuration? -> similar like pod, we should allow it
status and conditions need to be double checked.
Limit the maximum rank supported by base model replicas, otherwise, it may experience the OOM later -> controlled by scheduler at this moment

Use Case

No response

Proposed Solution

No response

Jeffwan · 2024-09-23T20:42:54Z

Few follow-ups

if the schedule restarts, how does it know the lora rank size? -> retrieve from the model list?
if we'd like to support replicas, then what's the primary change, how to guarantee there's no duplicate creations issues etc

Jeffwan · 2024-09-27T19:02:09Z

Stability Improvement

Scenario 0: Any status issue or unloaded issues.

unloaded issues are normally related to pod restart etc and the lora information is missing. Now Pod watch has been corrected and all related issues should be resolved.

Status issue should be resolved for most of major workflow. This needs more improvement later

Scenario 1: Delete base model, then delete lora

Still running, this is definitely a bug, it should enqueue the right model adapter objects. then it could update the status.

Scenarios 2: Create adapter, Delete adapter, Create adapter.

#239 reproducible

Scenarios 3: Failed to delete model adapter which take long time

Deletion doesn't get stucked, it's working fine.

Jeffwan · 2024-09-29T17:22:14Z

Scenarios 3: partial pods in terminating status and apply the the adapter

2024-09-29T17:21:47Z	INFO	Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler	{"controller": "model-adapter-controller", "controllerGroup": "model.aibrix.ai", "controllerKind": "ModelAdapter", "ModelAdapter": {"name":"lora-1","namespace":"aibrix-system"}, "namespace": "aibrix-system", "name": "lora-1", "reconcileID": "3c12e51b-5dcf-4757-93e9-ebcd3bd094a4"}
2024-09-29T17:21:47Z	ERROR	Reconciler error	{"controller": "model-adapter-controller", "controllerGroup": "model.aibrix.ai", "controllerKind": "ModelAdapter", "ModelAdapter": {"name":"lora-1","namespace":"aibrix-system"}, "namespace": "aibrix-system", "name": "lora-1", "reconcileID": "3c12e51b-5dcf-4757-93e9-ebcd3bd094a4", "error": "pod does not exist in the cache: llama2-70b-8fd6c849b-hkwnl"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227

Jeffwan · 2024-09-29T17:25:06Z

Misc 1: model adapter didn't filter out unrelated pods (resolved) #255

I0929 17:12:28.773609       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="envoy-gateway-system/envoy-gateway-8484c95484-hkhr5"
I0929 17:12:28.773829       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/vpnkit-controller"
I0929 17:12:28.773852       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="envoy-gateway-system/envoy-aibrix-system-aibrix-eg-903790dc-865ff88d5f-5bvxn"
I0929 17:12:28.773871       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="aibrix-system/aibrix-kuberay-operator-7648cb948-59tcq"
I0929 17:12:28.773886       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/coredns-76f75df574-k5n5b"
I0929 17:12:28.773908       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/kube-apiserver-docker-desktop"
I0929 17:12:28.773944       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/kube-controller-manager-docker-desktop"
I0929 17:12:28.773962       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/kube-scheduler-docker-desktop"
I0929 17:12:28.773983       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="aibrix-system/aibrix-controller-manager-56576666d6-xq5cp"
I0929 17:12:28.773998       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/coredns-76f75df574-9jrxk"
I0929 17:12:28.774013       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/etcd-docker-desktop"
I0929 17:12:28.774026       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/storage-provisioner"
I0929 17:12:28.774042       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="aibrix-system/aibrix-gateway-users-6c5dc67c96-w25zp"
I0929 17:12:28.774055       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="aibrix-system/aibrix-redis-master-84769768cb-zmsm8"
I0929 17:12:28.774068       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="kube-system/kube-proxy-fkgpd"
I0929 17:12:28.774085       1 modeladapter_controller.go:176] "ModelAdapter resource not found. Ignoring since object mush be deleted" modelAdapter="aibrix-system/aibrix-gateway-plugins-9c68c4bb7-ztfs7"

unrelated pods should be filtered out. This is because the model adapter controller Watches(&corev1.Pod{}, &handler.EnqueueRequestForObject{}) but did not filter the related ones, for example with the ownership type or labels. Adding label predicates resolves the issue but we have to use a specific label in future to indicate the pod and lora relationship.

Jeffwan · 2024-09-29T17:27:41Z

Misc 2: base model pod event can not trigger adapter reconciliation

Since the pod is not owned by the modelAdapter, we can not that easily trigger the adapter reconciliation. It should check reverse lookup adapter list. which one status has this pod name, then put that model adapter in the queue

&handler.EnqueueRequestForObject{} was not correctly used here, ForObject will put pod in the queue, ForOwner will put owner in the queue but we can not set owner, FromMapFunc make sense to support arbitrary collection of objects.

https://book-v1.book.kubebuilder.io/basics/simple_controller

Jeffwan · 2024-09-30T05:32:09Z

Misc 3: Create lora always experience the object outdated issue

24-09-30T05:31:20Z	ERROR	Reconciler error	{"controller": "model-adapter-controller", "controllerGroup": "model.aibrix.ai", "controllerKind": "ModelAdapter", "ModelAdapter": {"name":"lora-1","namespace":"aibrix-system"}, "namespace": "aibrix-system", "name": "lora-1", "reconcileID": "04f54bb5-9ff3-4f98-8d2a-de18f65a7745", "error": "Operation cannot be fulfilled on modeladapters.model.aibrix.ai \"lora-1\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227

Jeffwan · 2024-09-30T05:37:40Z

Testing after base model pod is deleted.

t0: still same as usual
t1: ip not found, pod in terminating status
t2: eventually failed

conclusion: It's the reconciliation loop issue, we check whether the selectPod is in termination, if so, we should take corresponding actions

Jeffwan · 2024-09-30T07:36:49Z

new pod coming and pending adapter is scheduled but timestamp is not updated.

Jeffwan added kind/enhancement New feature or request area/lora priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Sep 6, 2024

Jeffwan added this to the v0.1.0-rc.2 milestone Sep 6, 2024

Jeffwan self-assigned this Sep 6, 2024

Jeffwan modified the milestones: v0.1.0-rc.2, v0.1.0-rc.3 Sep 24, 2024

Jeffwan mentioned this issue Oct 1, 2024

Improve model adapter reconcile workflow stability #260

Merged

Jeffwan changed the title ~~Enhance the model adapter stability to production grade~~ Enhance the model adapter stability to alpha grade Oct 1, 2024

Jeffwan closed this as completed in #260 Oct 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance the model adapter stability to alpha grade #136

Enhance the model adapter stability to alpha grade #136

Jeffwan commented Sep 6, 2024 •

edited

Loading

Jeffwan commented Sep 23, 2024

Jeffwan commented Sep 27, 2024 •

edited

Loading

Jeffwan commented Sep 29, 2024 •

edited

Loading

Jeffwan commented Sep 29, 2024 •

edited

Loading

Jeffwan commented Sep 29, 2024 •

edited

Loading

Jeffwan commented Sep 30, 2024

Jeffwan commented Sep 30, 2024 •

edited

Loading

Jeffwan commented Sep 30, 2024 •

edited

Loading

Enhance the model adapter stability to alpha grade #136

Enhance the model adapter stability to alpha grade #136

Comments

Jeffwan commented Sep 6, 2024 • edited Loading

🚀 Feature Description and Motivation

Use Case

Proposed Solution

Jeffwan commented Sep 23, 2024

Jeffwan commented Sep 27, 2024 • edited Loading

Stability Improvement

Scenario 0: Any status issue or unloaded issues.

Scenario 1: Delete base model, then delete lora

Scenarios 2: Create adapter, Delete adapter, Create adapter.

Scenarios 3: Failed to delete model adapter which take long time

Jeffwan commented Sep 29, 2024 • edited Loading

Scenarios 3: partial pods in terminating status and apply the the adapter

Jeffwan commented Sep 29, 2024 • edited Loading

Misc 1: model adapter didn't filter out unrelated pods (resolved) #255

Jeffwan commented Sep 29, 2024 • edited Loading

Misc 2: base model pod event can not trigger adapter reconciliation

Jeffwan commented Sep 30, 2024

Misc 3: Create lora always experience the object outdated issue

Jeffwan commented Sep 30, 2024 • edited Loading

Testing after base model pod is deleted.

Jeffwan commented Sep 30, 2024 • edited Loading

new pod coming and pending adapter is scheduled but timestamp is not updated.

Jeffwan commented Sep 6, 2024 •

edited

Loading

Jeffwan commented Sep 27, 2024 •

edited

Loading

Jeffwan commented Sep 29, 2024 •

edited

Loading

Jeffwan commented Sep 29, 2024 •

edited

Loading

Jeffwan commented Sep 29, 2024 •

edited

Loading

Jeffwan commented Sep 30, 2024 •

edited

Loading

Jeffwan commented Sep 30, 2024 •

edited

Loading