Crates.io index update failing due to conflict with ~/.gitconfig

[marcospb19]

EDIT: workaround solution at #8172 (comment).
EDIT: more precise explanation of this bug at #8172 (comment).

---

While trying to install delta, I got this error:

cargo build --release

Updating crates.io index
error: failed to get ansi_colours as a dependency of package git-delta v0.1.1 (/home/marcospb19/delta)
Caused by:
failed to fetch https://github.com/rust-lang/crates.io-index

Caused by:
failed to authenticate when downloading repository
attempted ssh-agent authentication, but none of the usernames git succeeded

Caused by:
error authenticating: no auth sock variable; class=Ssh (23)
make: *** [Makefile:2: build] Error 101

I know almost nothing about Rust or Cargo, sorry if this issue isn't very clear.
Why would cargo use ssh to make this update?

Notes

Cargo and Rust --version:
1.43.0

OS:
Arch Linux

[alexcrichton]

Oh dear, this looks bad! It may be a bug elsewhere though that is triggering the apparent need for authentication when auth shouldn't be needed. Are you able to clone that url via the git CLI itself? If not, can you try running with CARGO_LOG=verbose and see if that provides anything helpful?

[marcospb19]

Sorry for the delay, I'll try your instructions now.

[marcospb19]

So I reinstalled my system and now it is working as expected.

I'm sorry but now I'm not able to debug and reproduce the error anymore, but thanks for your help.

[marcospb19]

Reopening the issue with more precise information.

The issue is with ~/.gitconfig's [url ...] -> insteadOf configuration.

What is ~/.gitconfig?.

~/.gitconfig is the file that holds git configuration, for example, git config --global user.name "João" adds this line to the file.

[user]
        name = João

url and insteadOf

Is a configuration that replaces an URL format with another.

Here is how I use it (conflicts with cargo!!!!)

~/.gitconfig:

[url "git@github.com:"]
        insteadOf = https://github.com/

When I clone a github repository by https URL, git formats it to SSH

Typing
git clone https://github.com/rust-lang/cargo
Will translate to
git clone git@github.com:rust-lang/cargo

---

So when cargo tries Updating crates.io index, if the conflicting config is present, this error appears

This probably means that cargo is calling git and expecting it to clone the HTTPS URL, but git is actually cloning to the SSH URL.

---

How to reproduce:

rm ~/.cargo/registry/index -fr
echo '[url "git@github.com:"]
        insteadOf = https://github.com/' >> ~/.gitconfig
cargo search a

---

Fixing it?

Fixing this can be very tricky, as there is not only --global creating ~/.gitconfig, but also --system creating /etc/gitconfig (I think that --local won't effect).

Ignoring ~/.gitconfig is possible by unsetting the env var "HOME" when calling git clone.

HOME= git clone <URL> does the job

This worked for me when invoking cargo.

HOME= cargo search a (note that the space after = means that HOME is unset only for that command)

[marcospb19]

After some testing, I think that hiding HOME leads to another errors, as Rust uses the environment variable.

[ehuss]

If you are using SSH authentication, there is some guidance here for how to configure that with Cargo. Cargo doesn't ever ask for a password, so you have to make sure authentication is configured before running Cargo. Additionally, Cargo's built-in SSH library does not use OpenSSH's configuration files.

If you have more advanced authentication requirements, I recommend using net.git-fetch-with-cli.

[marcospb19]

@ehuss

I don't want to use SSH with cargo!, the SSH config present in my .gitconfig file is meant for cloning github repositories.

I found this insteadOf feature after a Google search because I almost always prefer SSH over HTTPS when working on projects, when using cargo I just rename my .gitconfig file to something else, and then I rename it back.

[joshtriplett]

@marcospb19 I have a similar setup, for the same reason, but using pushInsteadOf instead; that way, things that clone/pull from github will not use SSH, but trying to push to github (which cargo never does) will use SSH.

Here's the configuration you want:

[url "ssh://git@github.com/"]
        pushInsteadOf = "https://github.com/"
        pushInsteadOf = "git://github.com/"

That should solve your problem.

[joshtriplett]

On Cargo's side, we could have an extra check that asks git2-rs what the repository URL is after it has been mapped via git configuration, and if it's going to use SSH protocol, we can print an additional warning explaining the limitations of using SSH repositories, and suggesting the use of pushInsteadOf or net.get-fetch-with-cli.

[marcospb19]

@joshtriplett thanks for the response, looks like I don't need to clone the url with SSH protocol, this should work.

But still, cargo should check this, it is an existing conflict with valid git user configuration.

[marcospb19]

Works for not having to type password on git pushes, but not on git clones if the repository is private.

[joshtriplett]

@marcospb19 It's possible to configure git to handle authentication for private github repos over https. But if the repository is private, I'd suggest just using an ssh URL to clone it.

[marcospb19]

Sometimes I'm browsing my private repositories and I just copy the URL at my browser directly to the terminal, at others I type the HTTP address on the terminal because I memorized it.

This is a VALID feature of Git, but Cargo don't consider it, so now I have to always go to the repository link and copy the SSH address that GitHub gives me.

With all the respect, I'm not a huge fan of the way you recommend avoiding using a Git feature instead of debating how can we solve the issue, if it is solvable and why.

Is there a reason why I should not use this Git feature besides the fact that Cargo has a bug?

If this is unsolvable, then I might just accept it and close this issue, at least people that have the same problem can find this issue for reference on how to solve it, because there's actually no clear hints of how to. But I don't think that this is unsolvable (at least a message error advertising the possible reason of the error).

[ehuss]

@marcospb19 Can you clarify what the issue is with setting net.git-fetch-with-cli? Do you continue to have authentication problems? If so, what does the error look like? If you have GitHub SSH authentication configured in such a way that you can git clone GitHub repos from your command-line, Cargo should also work.

[marcospb19]

If you are using SSH authentication, there is some guidance here for how to configure that with Cargo. Cargo doesn't ever ask for a password, so you have to make sure authentication is configured before running Cargo. Additionally, Cargo's built-in SSH library does not use OpenSSH's configuration files.

@marcospb19 Can you clarify what the issue is with setting net.git-fetch-with-cli?

Do you continue to have authentication problems? If so, what does the error look like?

@ehuss, I'm sorry but I have no idea what you're talking about, I'm not trying to authenticate nor configuring anything with cargo and I don't know what is net.git-fetch-with-cli.

I just tried running this command on my terminal:

$ cargo search a

And problems occurred as I explained.

[ehuss]

Ah, sorry about the confusion. Cargo needs to communicate with GitHub for the index, and the insteadOf git configuration is telling Cargo to use SSH for that purpose. However, Cargo's SSH behavior doesn't exactly match that of the git command-line client (it typically needs ssh-agent to be running). The net.git-fetch-with-cli Cargo configuration value tells Cargo to use the git command-line client for git interaction instead of its built-in git library. You can set that up by creating a file called ~/.cargo/config and add the text:

[net]
git-fetch-with-cli = true

Assuming the git CLI is able to communicate with GitHub over SSH, then this should work.

[marcospb19]

Thanks for your response, there's something I did not understood.

Assuming the git CLI is able to communicate with GitHub over SSH, then this should work.

Why would I want this to happen?

[ehuss]

If you want to have a global git configuration that tells git to always use SSH, then there isn't any choice. All tools using git will use that configuration.

I think, if for whatever reason you really need to use https for Cargo's index, I think an alternate solution would be to use this in gitconfig:

[url "git@github.com:"]
        insteadOf = https://github.com/

[url "https://github.com/rust-lang/crates.io-index"]
        insteadOf = https://github.com/rust-lang/crates.io-index

I'm not 100% certain that always works, but I did a quick test and it seems ok.

[vors]

It seems to me that a reasonable way to solve it is by bypassing the default git config by using the libgit2 machinery that was added here:

Issue
libgit2/libgit2#4658

PR
libgit2/libgit2#4667

We probably need to add some plumbing in git2-rs in order to take advantage of GIT_REMOTE_CREATE_OPTIONS_INIT

But in theory after that is done, cargo can finally break free from the dependency on the user's defined default git config.

[werdahias]

I have the same issue on debian testing with rustup. Stable rustc and cargo installed. I get the following error message:

 Updating crates.io index
error: failed to get `anyhow` as a dependency of package `popsicle_cli v1.3.0 (/home/user/popsicle/cli)`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  an unknown git error occurred; code=NotFound (-3)

[rjhornsby]

The suggested fix from @ehuss worked great. Thanks for that.

That said, coming from many other languages or tools (python, ruby, go, etc) and being brand new to rust, this behavior - the error - is completely unexpected. While the error message is more clear having the context of this thread in mind, it certainly was not so when trying to decipher why everything else that uses git authenticates fine, but suddenly I've got something that's saying the auth is missing, or more generally wrong? While staring at the error, contemplating everything that does work re: git, it's very confusing why rust/cargo does not. What broke?

While there's probably enough verbiage in the error message, without the additional context that cargo has read something from .gitconfig it doesn't like, a suggestion of how it might be solved, or a link to a doc that explains more about the problem, it's difficult to expect the user to figure this out.

Ideally, this could be fixed so cargo behaved more like other consumers of .gitconfig rather than requiring a special rule just for it. Barring that, somehow making the messaging more clear so that it doesn't take finding and trying to understand a github issue thread might help usability and lower the barrier to entry a bit.

Maybe something simple like this when running cargo without @ehuss's override:

WARNING: your ~/.gitconfig file uses insteadOf to rewrite https://github.com/ to ssh://git@github.com/.

This is likely to cause cargo to have authentication issues talking to git. See _some url_ for more information.

[rkulla]

@ehuss's fix works great for me. I agree with @rjhornsby that at the very least a better message should be printed about the use of insteadOf in gitconfig.

Perhaps adding a section on it to The Cargo Book and linking to it, along with the net.git-fetch-with-cli link, since the actual 'Caused by' was different. Maybe say "Possible causes: ..." instead of caused by?