Why are file transfers so slow

[matkr265]

I have replaced my own proxy middleware with YARP, and now I am experiencing very slow file transfers. Downloading 50 MB takes 120 seconds but when I perform the same test with my old middleware it takes 0.5 seconds.

Is it possible that I have configured YARP incorrectly to induce this issue?

YARP configured as a forwarder in code:

public void Configure( IApplicationBuilder app, IWebHostEnvironment env, IHttpForwarder _forwarder )
{
      httpClient = new HttpMessageInvoker( new SocketsHttpHandler
                                           {
                                               UseProxy = false,
                                               AllowAutoRedirect = false,
                                               AutomaticDecompression = DecompressionMethods.None,
                                               UseCookies = false
                                           } );

      requestOptions = new ForwarderRequestConfig { ActivityTimeout = TimeSpan.FromSeconds( 100 ) };
            
      app.Run( async context => { 
            /* Abbreviated code */
            var error = await forwarder.SendAsync( httpContext, resolvedUri.ToString(), httpClient, requestOptions );
      } );
}

My use case:

net472 desktop client (auto update feature) 
      <- HTTP/1.1 ->
netcoreapp3.1 YARP server 
      <- HTTP/1.1 ->
 net472 API (hosts the new application binaries)

Environment:

• Client / Server OS: Windows 10 / Windows Server 2019
• Client: C# / WPF/ net472
• API: C# / net472
• YARP: C# / netcoreapp3.1
• YARP version: 1.0.0
• HTTP 1.1 between all endpoints

Any and all ideas are greatly appreciated, thanks in advance!

[MihaZupan]

Can you please provide a few more details:

• Is the desktop app (client) also .NET?
• Which version of .NET are the client & YARP running on?
• Which YARP version you are using (1.0 / 1.1)?
• Which HTTP versions are being used between the client and proxy, or between the proxy and downstream server?

Before .NET 6, there is a known issue where using HTTP/2 will result in slow data transfers on higher-latency connections.

[matkr265]

I have updated the post with the requested information.

• Yes the client is also .NET
• YARP is running on netcoreapp3.1 and the client and api on net472
• I am using YARP 1.0
• HTTP 1.1 is used between client and YARP as well as YARP and API

[karelz]

@matkr265 can you please double check it is indeed HTTP/1.1 and not HTTP/2?
If it is HTTP/1.1, we would be interested in a repro. Are you able to provide it? (i.e. try to reproduce it locally yourself, then try to transfer it to us)

[matkr265]

@karelz I used wireshark to verify in production, but my local test project (YARP 1.1.0) shows the following output below. I will try to produce a repro after I have fixed a work around in production (i.e. move the file download so it does not pass YARP).

Yarp.ReverseProxy.Forwarder.HttpForwarder:
   Information:
      Proxying to http://localhost/internalapi/test/k/1.0 HTTP/2 RequestVersionOrLower no-streaming
Yarp.ReverseProxy.Forwarder.HttpForwarder:
   Information:
      Received HTTP/1.1 response 200.

I have been trying to figure out what is slowing things down and at the moment my hunch is that it is the URl format. Adding a dot in the url slows it down a bit.

• http://MySiteInAzureVm/api/test/k/1.0.0.456
  51 MB in 150 s, PowerShell script to YARP to API
• http://MySiteInAzureVm/api/test/k/100456 (no dots in url)
  51 MB in 20 s, PowerShell script to YARP to API

The slowdown is a less, but noticable in my local test application

• http://localhost:[80, 5000]/internalapi/test/k/1.0.0.456
  51 MB in 5,46 s, PowerShell script to, port 5000, YARP to API
  51 MB in 3,61 s, PowerShell script to, port 80, API

• http://localhost:[80, 5000]/internalapi/test/k/100456
  51 MB in 3,78 s, PowerShell script, port 5000, to YARP to API
  51 MB in 2,61 s, PowerShell script, port 80, to API

[karelz]

@matkr265 your results above are highly suspicious. Having or not having dots in URL's Path has hardly a chance to influence perf at all (the more in a measurable way and in your case with 7x throughput impact).
Something has to be horribly wrong on your machine(s). It is highly unlikely that this is problem in .NET or YARP.

I would recommend to:

• Try your tests on set of new machines.
• Simplify things as much as possible (incl. e.g. removing PowerShell from the scenario).
• Do multiple measurements -- in serious perf measurements we typically measure 10x, remove the 2 best and 2 worst measurements (or just 2 worst -- to avoid hitting some weird cases when things go super slow from weird unrelated reasons), then do average on the rest. Computing statistical variance would be also great to check on consistency of data.
• Or you can try to do deeper perf investigation -- e.g. check perf telemetry on YARP, server and client to see where the time is truly spend.

[matkr265]

I agree, something is very off. I worked around the issue, whatever it was, and have not hade time to dig into this.

I resolved my issue by placing the download function in its own middleware in our gateway, instead of routing it thru the YARP middleware to the API.

Thank you for your effort.

[karelz]

Not actionable for YARP at this moment. Marking as answered.