Cannot retrieve commits from a specific repository

[jemaf]

GitPython is failing to retrieve commits from a specific repository I am currently analyzing: https://github.com/jbrowncfa/Cryptobomb.

Exception's stack trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/git/repo/base.py", line 524, in iter_commits
    rev = self.head.commit
  File "/usr/local/lib/python3.6/site-packages/git/refs/symbolic.py", line 197, in _get_commit
    obj = self._get_object()
  File "/usr/local/lib/python3.6/site-packages/git/refs/symbolic.py", line 190, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/usr/local/lib/python3.6/site-packages/git/objects/base.py", line 64, in new_from_sha
    oinfo = repo.odb.info(sha1)
  File "/usr/local/lib/python3.6/site-packages/git/db.py", line 37, in info
    hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
  File "/usr/local/lib/python3.6/site-packages/git/cmd.py", line 1068, in get_object_header
    return self.__get_object_header(cmd, ref)
  File "/usr/local/lib/python3.6/site-packages/git/cmd.py", line 1057, in __get_object_header
    return self._parse_object_header(cmd.stdout.readline())
  File "/usr/local/lib/python3.6/site-packages/git/cmd.py", line 1019, in _parse_object_header
    raise ValueError("SHA could not be resolved, git returned: %r" % (header_line.strip()))
ValueError: SHA could not be resolved, git returned: b''

Reproducible example:

$ git clone https://github.com/jbrowncfa/Cryptobomb

from git import Repo
repo = Repo("./Cryptobomb")
for commit in repo.iter_commits():
    print(commit.hexsha) # error

Here is the Dockerfile I'm using:

FROM python:3.6

ENV GIT_DISCOVERY_ACROSS_FILESYSTEM=1
RUN apt-get update
RUN apt-get install -y git 

Please let me know if you guys need any more info.

[Byron]

Thanks a lot, I was able to reproduce the issue, and here is how it looks for me:

➜  GitPython git:(master) ✗ python3 reproduce.py
Traceback (most recent call last):
  File "reproduce.py", line 3, in <module>
    for commit in repo.iter_commits():
  File "/Users/byron/dev/GitPython/git/repo/base.py", line 524, in iter_commits
    rev = self.head.commit
  File "/Users/byron/dev/GitPython/git/refs/symbolic.py", line 197, in _get_commit
    obj = self._get_object()
  File "/Users/byron/dev/GitPython/git/refs/symbolic.py", line 190, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/Users/byron/dev/GitPython/git/objects/base.py", line 64, in new_from_sha
    oinfo = repo.odb.info(sha1)
  File "/Users/byron/dev/GitPython/git/db.py", line 37, in info
    hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
  File "/Users/byron/dev/GitPython/git/cmd.py", line 1068, in get_object_header
    return self.__get_object_header(cmd, ref)
  File "/Users/byron/dev/GitPython/git/cmd.py", line 1057, in __get_object_header
    return self._parse_object_header(cmd.stdout.readline())
  File "/Users/byron/dev/GitPython/git/cmd.py", line 1021, in _parse_object_header
    raise ValueError("SHA %s could not be resolved, git returned: %r" % (tokens[0], header_line.strip()))
ValueError: SHA b'4edad68336cc60e626c81a106b54bcfc6ab90b41' could not be resolved, git returned: b'4edad68336cc60e626c81a106b54bcfc6ab90b41 missing'

[simisimon]

We face the same problem when we analyze repositories. Do you have any updates on this issue?

[andbue]

Had the same problem, updating git to git-2.30.0-1 solved it for me.

[Yook74]

I was getting this error message when I was trying to get info about a repository which I was not the owner of (even though I was root). I tried doing it manually with git rev-parse HEAD and got
fatal: unsafe repository ('[my repo]' is owned by someone else)

If you're having this problem, see if git rev-parse HEAD works.

I would like to see some better error handling if possible. It seems clear to me that an error was not properly being raised somewhere and a function was silently returning an empty binary string.

[Byron]

I would like to see some better error handling if possible. It seems clear to me that an error was not properly being raised somewhere and a function was silently returning an empty binary string.

This is a known issue as some commands are configured to stream data and only look at stdout. That said, git appears to exit with code 129 making it possible to provide a better error message. We could use an issue to keep track of this or a PR to fix it. Here is a related conversation.

[ramonlins]

Maybe git is running into a different owner than the repository.

Inside repo, you can try this command:
git config --global -add safe.directory <repo_path>

[Yook74]

@ramonlins yes that was the fix for my particular problem

[kapsner]

thanks @ramonlins , that works for me as well.

However, I do not understand what caused this error message to suddenly appear, while it was working before for a long time without any significant changes in the setup lately...

[Yook74]

@kapsner a recent update to Git on the apt repositories added a feature which is causing this problem.

[frozenpandaman]

For more reading: https://stackoverflow.com/a/71904131/3580553

Starting in Git v2.35.3, safe directory checks can be disabled, which will end all the "unsafe repository" errors
This can be done by running:
git config --global --add safe.directory '*'
It will add the following setting to your global .gitconfig file:

[safe]
    directory = *

Before disabling, make sure you understand this security measure, and why it exists. You should not do this if your repositories are stored on a shared drive.
However, if you are the sole user of your machine 100% of the time, and your repositories are stored locally, then disabling this check should, theoretically, pose no increased risk.

[mamiksik]

I am getting this error for https://github.com/nodejs/node. git config --global --add safe.directory '*' did not solve the issue.