Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory load of redistributable download #342

Merged
merged 1 commit into from
Jan 22, 2025

Conversation

atifaziz
Copy link
Contributor

The use of HttpClient.GetAsync to download the Python redistributable in RedistributableLocator causes
the entire response content to be buffered to memory before being written to disk. Just stepping over the following line in the debugger:

using HttpResponseMessage response = await client.GetAsync(fileUrl);

shows the memory spike (over 2,600 objects and tens of MBs allocated):

If one examines the heap for byte arrays, we find a 41.6MB allocation done by HttpContent.LimitMemoryStream

The exact allocation size of 416,204,438 corresponds to the HTTP content length:

Screenshot 2025-01-18 150747

The PR fixes the problem by using HttpClient.GetStreamAsync, which checks for a success response code before returning a stream over the content (instead of entirely buffering it). The image below shows the memory usage drops dramatically, allocating just 284.4 KB after stepping over the HttpClient.GetStreamAsync call (ID 2) and something similar until the end of the DownloadFileToTempDirectoryAsync method (ID 3):

@tonybaloney tonybaloney merged commit de88333 into tonybaloney:main Jan 22, 2025
37 checks passed
@atifaziz atifaziz deleted the opt-redist-download-mem branch January 22, 2025 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants