Add LinkFileTransfer #200
Add LinkFileTransfer #200
Conversation
076557e to
b7980d0
Compare
|
|
||
| .. code-block:: | ||
|
|
||
| /downloads/file1.dat -> /dataset/source/file1.dat |
There was a problem hiding this comment.
Should we be making symlinks or should we be making copies?
I guess when you are using a sftp file transfer, you make a copy. And then you are allowed to manipulate/modify the file you get as you wish.
If we make symlinks to read-only files, we won't then be able to do the same.
This would imply that using a different file-transfer would give you different results, even if the file you are targetting is the same. Does it seem inconsistent?
There was a problem hiding this comment.
Good point. It is a bad idea to modify the files regardless because that can ultimately lead to derived data that was produced from different files than it claims. E.g., reduced data that is uploaded to SciCat and linked to an input dataset even though it was produced from a different file. (The same happens with any storage solution that links back to the input.)
So given this, I think that enabling this behaviour is not worth the cost of copying files. Do you agree?
| con.download_files( | ||
| remote=[RemotePath(str(remote_dir / "text.txt"))], | ||
| local=[local_dir / "text.txt"], | ||
| ) |
There was a problem hiding this comment.
Do you also need a test for when the local directory does not exist?
There was a problem hiding this comment.
Not sure. The file transfers never create the directory, but Client does. Is this something we need to test?
|
Actually needs a test to check what happens when a file already exists where it tries to create a link. |
Fixes #38
First version of a file transfer that directly accesses files if they are available.