Seems Httproute not in aibrix-system won't work

[Jeffwan]

🐛 Describe the bug

Steps to Reproduce

No response

Expected behavior

No response

Environment

No response

[Jeffwan]

gateway part should have namespace fix to aibrix-system, otherwise, it think the namespace should be same as HTTPRoute

[varungup90]

Adding a fix to create referance grant to connect httproute (in aibrix-system namespace) and service if in another namespace. Right now httproute is created in same namespace as service but I am thinking, since it is gateway related object lets create it in our namespace (aibrix-system)

[varungup90]

#347

[Jeffwan]

1. I double checked v0.1.0-rc.4. I update the namespace with aibrix-system and it doesn't work, I may add some other grants etc in my earlier test. this is expected and approve the grant is needed here.
   

2. However, I build nightly image from main branch and notice the problem is gone.. this is weird. could you help reproduce it?

[varungup90]

For 2, on describing httproute there is no status. I think, it is using least-request as routing algorithm which does not need httproute.

Can you check env variable for gateway deployment and use -v option in curl request to print response header (it will have target-pod)

[Jeffwan]

@varungup90 Got you. I listed the details. My point is nightly build is supposed not to work with empty strategy header.. why does it work now? It should not bypass httproute and use a pod direct strategy as backup?

curl -v http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer any_key" \
  -d '{
     "model": "llama2-70b",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8000...
* Connected to localhost (::1) port 8000
> POST /v1/chat/completions HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer any_key
> Content-Length: 129
>
* upload completely sent off: 129 bytes
< HTTP/1.1 200 OK
< server: Werkzeug/3.0.6 Python/3.9.20
< date: Mon, 11 Nov 2024 18:37:29 GMT
< content-type: application/json
< content-length: 305
< x-went-into-resp-headers: true
< target-pod: 10.1.6.61:8000
<
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"\n\nThis is a test fromllama2-70b!","role":"assistant"}}],"created":1677858242,"id":"chatcmpl-abc123","model":"llama2-70b","object":"chat.completion","usage":{"completion_tokens":36,"prompt_tokens":61,"total_tokens":97}}
* Connection #0 to host localhost left intact

[Jeffwan]

Synced with Varun offline, I forgot the env override on the deployment. that's why it bypass the httproute and start to work.

[Jeffwan]

We determine to leave httproute in aibrix-system at this moment