Handle the model unload requests in model adapter controller

[Jeffwan]

🚀 Feature Description and Motivation

After the model adapter is deleted, we should unload the model adapter from the pod. Otherwise, it will wait for the GC to evict the Lora adapter. We definitely prefer eager way over lazy way here.

We need to response the deletion event and send request to the corresponding pod. There's no way to update the model adapter then.

BTW, we could also leverage finalizer to make it. .metadata.deletionTimestamp will be marked first and the finalizer can help unload the model.

Use Case

No response

Proposed Solution

No response

Alternatives Considered

No response

Additional Context

No response

[Jeffwan]

Some edge cases related to unload

1. Pod deletion -> we should update the lora crd status
2. CRD deletion -> finalizer -> clean up the lora on the pods

[varungup90]

I can take this task, if no one is working on this actively.

[Jeffwan]

@varungup90 I am working some improvements for model adapter. Please start this one later to avoid merging conflicts