error reading response body: unexpected EOF investigation

[Jeffwan]

🐛 Describe the bug

this is kind of annoying. What I worried is it could be something wrong in the gateway plugins. Since this is data plane components, we have to figure out the root problem

Steps to Reproduce

#!/bin/bash

# Number of times to run the test
NUM_RUNS=10

# Counter
for i in $(seq 1 $NUM_RUNS); do
    echo "Running test iteration $i..."
    go test test/e2e/*
    if [ $? -ne 0 ]; then
        echo "Test failed on iteration $i"
        exit 1
    fi
done

echo "All tests passed successfully!"

there's chance to encounter the problem

Expected behavior

It should not raise any issues

Environment

nightly

[Jeffwan]

[Jeffwan]

I0214 17:35:42.153380       1 gateway.go:218]
I0214 17:35:42.153396       1 gateway.go:219] "-- In RequestHeaders processing ..." requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49"
I0214 17:35:42.153603       1 gateway.go:284] "-- In RequestBody processing ..." requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49"
I0214 17:35:42.183140       1 prefix_cache.go:99] "prefix cache route" message="[{\"content\":[{\"text\":\"this is 478 message\",\"type\":\"text\"}],\"role\":\"user\"}]" tokens=[58,5018,1834,67682,1342,3332,576,374,220,22086,1984,2247,1337,3332,1342,9388,29603,5898,3332,882,9388,60] matched_tokens=[] unmatched_tokens=[58,5018,1834,67682,1342,3332,576,374,220,22086,1984,2247,1337,3332,1342,9388,29603,5898,3332,882,9388,60] matched_pods=null ready_pods=["10.1.9.149","10.1.9.177","10.1.9.178","10.1.9.176"] target_pod="10.1.9.149"
I0214 17:35:42.183170       1 gateway.go:385] "request start" requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49" model="llama2-7b" routingStrategy="prefix-cache" targetPodIP="10.1.9.149:8000"
I0214 17:35:42.186325       1 gateway.go:404] "-- In ResponseHeaders processing ..." requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49"
I0214 17:35:42.186809       1 gateway.go:436] "-- In ResponseBody processing ..." requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49"
I0214 17:35:42.186877       1 gateway.go:550] request end, requestID: 1bbb99ff-4b76-46ab-b2b7-6f9623dbed49 - targetPod: 10.1.9.149:8000
I0214 17:35:42.186905       1 gateway.go:436] "-- In ResponseBody processing ..." requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49"
E0214 17:35:42.186935       1 gateway.go:476] "error to unmarshal response" err="unexpected end of JSON input" requestID="1bbb99ff-4b76-46ab-b2b7-6f9623dbed49" responseBody=""

this is a complete story, It is supposed to be end after request end message. However, it triggers another In ResponseBody processing which is unexpected and the responseBody is empty. that's the problem. We need to figure out why it happens

[Jeffwan]

more investigation

1. earlier, I thought it would be always last request but notice not the case after running more expeirments. Just randomly failed.
2. above log problem is very obvious. I will check the header whether it's chunked... not fully buffered.

[Jeffwan]

b.ResponseBody.EndOfStream return true.

[Jeffwan]

Option 1. I will increase the buffer size for short term.
Option 2. Use a responseBuffer and handle the end_of_stream case

[Jeffwan]

Seems option 1 can not fully resolve it. I changed to option 2.