sendfd doesn't send fd on Forwarder/Endpoint iteration

[Bolodya1997]

Expected Behavior

sendfd always should send fd to the next hop.

Current Behavior

sendf doesn't send fd in case of Forwarder/Endpoint iteration

Steps to Reproduce

Run multiforwarder tests.

Failure Logs

https://github.com/networkservicemesh/integration-k8s-packet/suites/4099955253/artifacts/104589184

[Bolodya1997]

Cause

There are few different reasons for this issue:

roundrobin and interpose doesn't clone Request on iteration

1. roundrobin/interpose selects new Endpoint/Forwarder.
2. sendfd replaces file:// URL with inode:// URL in Request and sends fd to the Forwarder.
3. Request fails.
4. rounrobin/interpose selects new Endpoint/Forwarder.
5. sendfd does nothing.

dial stands before sendfd client and first Closes old URL

1. sendfd adds grpc.CallOption for sending fd on first Request/Close.
2. dial makes Close to the old URL -> fd is sent to the old URL.
3. dial makes Request to the new URL -> no fd sent.

So we need to:

1. Clone Request in roundrobin, interpose.
2. Put dial before additionalFunctionality.

But here we have one more issue:

sendfd client adds grpc.CallOption to the existing opts list instead of creating a new one

1. dial makes Close to the old URL
2. sendfd adds grpc.CallOption for sending fd to the existing opts list.
3. Close happens to the old URL -> fd is sent to the old URL, grpc.CallOption captures FDTransceiver.
4. dial makes Request to the new URL.
5. sendfd fetches existing grpc.CallOption from the opts list.
6. Request happens to the new URL -> fd is again sent to the old URL because of captured FDTransceiver.

Solution

1. Clone Request in roundrobin, interpose.
2. Put dial before additionalFunctionality.
3. Copy opts in sendfd.