Add a way to check for duplicate dependencies

[gnzlbg]

There should be a simple way to check that there aren't any duplicate dependencies in the dependency graph, e.g., a cargo build --unique-deps / cargo run --unique-deps that errors if two versions of a dependency are anywhere in the dependency graph.

This would be useful for usage in CI.

What I currently do, is manually use cargo tree to pretty print the whole dependency tree, and manually inspect the dependencies. There should be a better way.

[est31]

@RalfJung has used Cargo.lock for doing this but the upcoming format doesn't lend itself as nicely to detection of dupes and it was suggested by to use tooling for this instead: #7070 (comment)

It would also be very helpful if you could specify a whitelist of crates that are allowed to be duplicate. Sometimes it requires major effort to deduplicate a dependency. I think servo might only switch to the builtin tool if you can specify a whitelist.

cc @Eijebong

[Eijebong]

Yeah, it's really really hard to go for 0 dupes. (try to have only one version of rand_core if you've got anything depending on rand or one version of winapi if you've got anything depending on mio...).

A whitelist is needed but at least makes people aware of the fact they're duping something. It's useless without a committed lockfile though and that's something to keep in mind.

[Aaron1011]

@gnzlbg: You can use cargo tree -d to only print duplicates.

[repi]

We created cargo-deny specifically to disallow duplicates by default and manage which duplicates to allow. This we run on CI to verify and uphold and after doing a cargo update or adding crates

[epage]

I'd love for us to look into moving some of cargo-denys functionality into cargo (or cargo-clippy) once we have #12235.