Improve SFTP hook's directory transfer to use a single connection for multiple files #46582
+17
−7
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR improves SFTP hook's
store_directory
andretrieve_directory
functions to use a single connection when transferring a directory with multiple files.Previously, these functions relied on
store_file
andretrieve_file
functions. And with this PR, thestore_file
andretrieve_file
functions were modified to open and close sftp connection each time.This leads to the
store_directory
andretrieve_directory
functions to open and close too many connections repeatedly when there are many files in a directory, which causes significant overhead.To address this, I modified them to open a connection, transfer all files in a directory, and then close the connection afterward.
I also did a performance test in my local environment by transferring a directory containing 1,000 small files. This reduced the transfer time by approximately 8-9 seconds. The results are shown below.