remote: migrate: deduplicate objects between v2 and v3

[12michi34]

My situation is like this
a) dataFolderA containing fileA.bin fileB.bin, fileC.Bin and I added that via "dvc add dataFolderA" to the remote dvc via 2.0
b) then I changed fileB.bin and added that via "dvc add dataFolderB" to the remove via dvc 3.0

when investigating the remote(and cache) I can see the md5-renamed file for fileA.bin and fileC.bin in both files/md5// and /
it is the same exact md5 hash and the data for fileA.bin and fileC.bin are now twice in the remote (and cache)
(I am simplifying my case there are many fileA,fileB,fileC's involved)

How can I clean up the remote?. I know there exists a "dvc cache migrate" (have not tried it yet though) .
Kindest regards

[efiop]

@12michi34 Is this you on the forum as well https://discuss.dvc.org/t/help-with-upgrading-imported-via-dvc2-x-dvc-data-with-dvc3-0/1750/10? If not, then just linking for the record.

[efiop]

Regarding the question itself, there is no such feature right now. We've thought about it when implementing migrate, but didn't prioritize it. One obvious way to go here is to try to use hardlinks to link v2 and v3 files.

[0x2b3bfa0]

(Originated on Discord)

[pmrowla]

Just to clarify here, DVC already does support 2.x/3.x deduplication for local cache (via dvc cache migrate). Deduplication is only currently unsupported for remotes.

[dberenbaum]

After #9938, let's document how best to handle this -- migrate everything to 3.0 and then gc

[12michi34]

@efiop .. sorry about the late reply . Yes, this originated on discord. Somehow missed notification emails that there are new comments on this issue.

[dberenbaum]

Added some explanation to the migration guide in the docs. No more action is planned at the moment, so closing this one.