import/get: meta-issue

[dberenbaum]

This issue collects improvements needed for dvc import/get:

• Don't clone if repo is available locally (fixed in dep: repo: use dvcfs #9246)
• Pull directories from source: cloud versioning: dvc pull fails after version aware import-url for non version aware s3 remote #8872
• Choosing a remote get/import/list/etc command should accept a remote as argument #2466
• Configuring remote options (like AWS profile)
• api/cli: introduce remote_config #9656
• dvc remote/config --config myconfig

p2/research issues for performance improvements:

• Filter clones (limit what is fetched for monorepos)
• Use bare repos
• Cache clones (probably need clone: support creating mirrors scmrepo#204 first)
• Incremental updates: update: only download incremental changes #8808
• Cache data

[efiop]

For the record: Edited first (fixed) and third (need scmrepo mirrors) points.

[dberenbaum]

@aguschin Any details you can add about monorepo support?

[aguschin]

No @dberenbaum. I'll add something when I start implementing import/get I think.

[pmrowla]

With regard to clone caching, do we want it to be configurable? I'm not sure having the caching as default behavior is ideal.

Right now if you import/get with a specific branch or tag name, we make a shallow clone with only that specific branch or tag. If a user has a giant git repo (which seems likely in the monorepo case), it is probably going to be faster for DVC to always make a new temporary shallow clone that only contains the specific thing they want to import than it would be for us to mirror their entire monorepo once and then re-use that cached mirror for future imports.

It seems to me that it would make more sense to have the caching be optional (and non-default).

[efiop]

@pmrowla Though global caching might not imply mirroring. We have caching within the session already, maybe there is a way to just make that global as is so we cache those shallow clones in site_cache_dir. And maybe there is a way to be smart about unshallowing them when we need to.

[dberenbaum]

Moved caching clones to the bottom of the checklist. I think what @efiop suggested makes sense, but if it's overly complex and/or isn't likely to be a major performance improvement, we can skip this item.

[efiop]

For the record: all main tasks are done now.