Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve SFTP hook's directory transfer to use a single connection for multiple files #46582

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Dawnpool
Copy link
Contributor

@Dawnpool Dawnpool commented Feb 8, 2025

This PR improves SFTP hook's store_directory and retrieve_directory functions to use a single connection when transferring a directory with multiple files.

Previously, these functions relied on store_file and retrieve_file functions. And with this PR, the store_file and retrieve_file functions were modified to open and close sftp connection each time.
This leads to the store_directory and retrieve_directory functions to open and close too many connections repeatedly when there are many files in a directory, which causes significant overhead.

To address this, I modified them to open a connection, transfer all files in a directory, and then close the connection afterward.

I also did a performance test in my local environment by transferring a directory containing 1,000 small files. This reduced the transfer time by approximately 8-9 seconds. The results are shown below.

AS-IS TO-BE
store 47.18 sec 39.35 sec
retrieve 63.50 sec 54.73 sec

@Dawnpool Dawnpool changed the title Improve SFTP hook's directory transfer to use a single connection in multiple files Improve SFTP hook's directory transfer to use a single connection for multiple files Feb 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant