Fix GitHub repository cloning #63
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GitHub deprecated use of the unencrypted Git protocol this year per: https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git This results in a quiet failure when cloning GitHub repositories. The clone_url property of a GitHub Repository object contains the HTTPS URL that can be used to clone a repository and will likely continue to be the source for a valid URL to clone a GitHub repository successfully.
Add a check of the returncode of the command executed in scraper.util.execute() and output an error message if it is not zero. Additionally change the logging level from DEBUG to ERROR for failures to process the JSON output from cloc. These combined will make it more clear when failures in core functionality are occurring.
IanLee1521
approved these changes
Aug 19, 2022
|
Thanks @mcdonnnj ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request fixes GitHub repository cloning so that it is functional again.
Per this blog post, and this blog post for background, GitHub has disabled the unencrypted Git protocol for cloning repositories. Since this project was not updated and is still using
git://URLs to clone GitHub repositories it is unable to correctly work with them. Additionally it silently fails on this issue and unless SLOC/labor_hours are checked it runs with no error output.The repository cloning bug is resolved by using the
repository.clone_urlattribute to get an HTTPS URL to use for cloning. Additionally the return code of commands run with thescraper.util.execute()function is checked and an error message is logged if it is not 0. Another logging message was changed fromlogging.debug()tologging.error()to reflect that a failure there impacts core functionality.Running against the
mainbranch:Running against this branch without the repository URL fix:
Running against this branch: