pip: Dependency Hashing does not work for Private Python Registries

[bpandola]

Dependabot relies onhashin to get Python package hashes:

dependabot-core/python/helpers/lib/hasher.py

Lines 7 to 14 in 08ac25e

	def get_dependency_hash(dependency_name, dependency_version, algorithm):
	hashes = hashin.get_package_hashes(
	dependency_name,
	version=dependency_version,
	algorithm=algorithm
	)
	return json.dumps({"result": hashes["hashes"]})

Dependabot does not supply the index_url argument, so the hashin.get_package_hashes function assumes the package is on PyPI ("https://pypi.org/"):
https://github.com/peterbe/hashin/blob/628bfe1d1dac565d743df76c9090f4534b625f23/hashin.py#L49

As a result, Dependabot errors out when hashin tries to look up the package metadata on pypi.org:

updater | 2023/08/25 22:40:42 INFO <job_714033322> Updating private-python-package from 3.3.0 to 3.3.1
  proxy | 2023/08/25 22:40:46 [196] GET https://pypi.org:443/pypi/private-python-package/json
  proxy | 2023/08/25 22:40:46 [196] 404 https://pypi.org:443/pypi/private-python-package/json

Full Dependabot Error Log

updater | 2023/08/25 22:40:39 INFO <job_714033322> Checking if private-python-package 3.3.0 needs updating
  proxy | 2023/08/25 22:40:40 [168] GET https://pypi.org:443/simple/private-python-package/
  proxy | 2023/08/25 22:40:40 [168] 404 https://pypi.org:443/simple/private-python-package/
  proxy | 2023/08/25 22:40:40 [170] GET https://private.pypi.server.com:443/pypi/private-python-package/
  proxy | 2023/08/25 22:40:40 [170] * authenticating python index request (host: private.pypi.server.com)
  proxy | 2023/08/25 22:40:40 [170] 200 https://private.pypi.server.com:443/pypi/private-python-package/
updater | 2023/08/25 22:40:40 INFO <job_714033322> Latest version is 3.3.1
  proxy | 2023/08/25 22:40:41 [176] GET https://pypi.org:443/simple/private-python-package/
  proxy | 2023/08/25 22:40:41 [176] 404 https://pypi.org:443/simple/private-python-package/
  proxy | 2023/08/25 22:40:41 [178] GET https://private.pypi.server.com:443/pypi/private-python-package/
  proxy | 2023/08/25 22:40:41 [178] * authenticating python index request (host: private.pypi.server.com)
  proxy | 2023/08/25 22:40:41 [178] 200 https://private.pypi.server.com:443/pypi/private-python-package/
updater | 2023/08/25 22:40:41 INFO <job_714033322> Requirements to unlock own
updater | 2023/08/25 22:40:42 INFO <job_714033322> Requirements update strategy bump_versions
updater | 2023/08/25 22:40:42 INFO <job_714033322> Updating private-python-package from 3.3.0 to 3.3.1
  proxy | 2023/08/25 22:40:46 [196] GET https://pypi.org:443/pypi/private-python-package/json
  proxy | 2023/08/25 22:40:46 [196] 404 https://pypi.org:443/pypi/private-python-package/json
updater | 2023/08/25 22:40:46 INFO <job_714033322> Sending event 7e655d5933ee4fafa17b2194e15ff8e8 to Sentry
  proxy | 2023/08/25 22:40:46 [198] POST https://sentry.io:443/api/1451818/store/
  proxy | 2023/08/25 22:40:46 [198] 200 https://sentry.io:443/api/1451818/store/
updater | 2023/08/25 22:40:46 ERROR <job_714033322> Error processing private-python-package (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | 2023/08/25 22:40:46 ERROR <job_714033322> 
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/common/lib/dependabot/shared_helpers.rb:138:in `rescue in run_helper_subprocess'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/common/lib/dependabot/shared_helpers.rb:84:in `run_helper_subprocess'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_replacer.rb:136:in `package_hashes_for'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_replacer.rb:78:in `updated_dependency_declaration_string'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_replacer.rb:30:in `block in updated_content'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_replacer.rb:25:in `gsub'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_replacer.rb:25:in `updated_content'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_file_updater.rb:58:in `updated_requirement_or_setup_file_content'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_file_updater.rb:41:in `block in fetch_updated_dependency_files'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_file_updater.rb:36:in `each'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_file_updater.rb:36:in `filter_map'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_file_updater.rb:36:in `fetch_updated_dependency_files'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater/requirement_file_updater.rb:23:in `updated_dependency_files'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater.rb:115:in `updated_requirement_based_files'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/python/lib/dependabot/python/file_updater.rb:35:in `updated_dependency_files'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/dependency_change_builder.rb:86:in `generate_dependency_files'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/dependency_change_builder.rb:36:in `run'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/dependency_change_builder.rb:25:in `create_from'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:122:in `check_and_create_pull_request'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:59:in `check_and_create_pr_with_error_handling'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `block in perform'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `each'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `perform'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:63:in `run'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/update_files_command.rb:38:in `perform_job'
updater | 2023/08/25 22:40:46 ERROR <job_714033322> /home/dependabot/dependabot-updater/lib/dependabot/base_command.rb:52:in `run'

[jurre]

Hmm, hashin's interface requires us to know which index specifically the package lives on, but I don't think that this is information that we have in Dependabot right now.

We can likely mimic the interface that pip uses (add multiple indexes via --extra-index-url and then figure it out), and pass all the index URLs we have and try each one. We have them ordered by likelihood here.

So we'd need to:

• update the native helper to accept the index url
• try all of them in order until we find the hashes
• return a better error that we can handle if none can be found

I already did that last part and added a small test to reproduce in this draft PR