PEP440 direct references pip strings broken in 2.10

[mpcusack-color]

Describe the bug
We have a few 3rdparty deps we've been defining in 2.9 as described in https://www.pantsbuild.org/v2.10/docs/python-third-party-dependencies#version-control-and-local-requirements:

python_requirement(
    name="foo",
    requirements=["foo @ https://github.com/color/foo/archive/v1.0.0.tar.gz"],
)

This has been working but breaks when switching to 2.10:

$ ./pants update-build-files --fix-python-macros
...
01:57:47.40 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.update_build_files.update_build_files
  in pants.backend.python.macros.deprecation_fixers.maybe_update_macros_references (3rdparty/BUILD.pants)
Traceback (most recent call last):
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
    res = func.send(arg)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/backend/python/macros/deprecation_fixers.py", line 303, in maybe_update_macros_references
    tuple(maybe_update(request.lines)),
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/backend/python/macros/deprecation_fixers.py", line 283, in maybe_update
    addr = maybe_address(val, renames, relative_to=os.path.dirname(request.path))
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/backend/python/macros/deprecation_fixers.py", line 172, in maybe_address
    addr = AddressInput.parse(val, relative_to=relative_to).dir_to_address()
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/build_graph/address.py", line 141, in parse
    ) = native_engine.address_parse(spec)
native_engine.AddressParseException: Failed to parse Address `foo@ https://github.com/color/foo/archive/v1.0.0.tar.gz`: error at 1:15: expected one or more key=value pairs to follow a `@`.

Fatal Python error: PyGILState_Release: thread state 0x7fc8d20a2670 must be current when releasing
Python runtime state: finalizing (tstate=0x16d69e0)

Thread 0x00007fc8f5f9f740 (most recent call first):
<no Python frame>
Aborted (core dumped)

Pants version
2.10.rc2

OS
linux

[Eric-Arellano]

Thank you so much for reporting this! I will look into this tomorrow and hopefully get a bug fix soon.

[mpcusack-color]

I also tried:

python_requirement(
    name="foo",
    modules=["foo"],
    requirements=["https://github.com/color/foo/archive/v1.0.0.tar.gz"],
)

which gives this error:

Traceback (most recent call last):
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
    res = func.send(arg)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 163, in resolve_target
    target = target_type(target_adaptor.kwargs, address, union_membership)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/util/meta.py", line 130, in new_init
    prev_init(self, *args, **kwargs)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/engine/target.py", line 339, in __init__
    self.field_values = self._calculate_field_values(unhydrated_values, address)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/engine/target.py", line 361, in _calculate_field_values
    field_values[field_type] = field_type(value, address)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/util/meta.py", line 130, in new_init
    prev_init(self, *args, **kwargs)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/engine/target.py", line 133, in __init__
    self.value: Optional[ImmutableValue] = self.compute_value(raw_value, address)
  File "/home/mpcusack/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0rc2_py39/lib/python3.9/site-packages/pants/backend/python/target_types.py", line 883, in compute_value
    raise InvalidFieldException(
pants.engine.target.InvalidFieldException: Invalid requirement 'https://github.com/color/foo/archive/v1.0.0.tar.gz' in the 'requirements' field for the target 3rdparty/internal:foo: Parse error at "'://githu'": Expected stringEnd

FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown
Aborted (core dumped)

[Eric-Arellano]

If you want to be temporarily unblocked, hold off on using ./pants update-build-files --fix-python-macros and explicitly set use_deprecated_python_macros = true. While we do recommend upgrading, this is a bug specifically in our "fixer" to automate the upgrade. Everything else should still work with foo@ https://github.com/color/foo/archive/v1.0.0.tar.gz, and it's fine to keep using the deprecated macros in the meantime

[mpcusack-color]

Oh that's good to know, thanks. So far I'm just getting a branch ready for the 2.10 release, so this isn't actually blocking anything else.

[stuhood]

Oof... and I see that you're still experiencing the heisenbug which #14253 was meant to address. I'll get that picked to 2.10.x.

[Eric-Arellano]

Oops, looks like this was fixed in main but we forgot to cherry-pick: #14696

Thank you for the report! We will do another 2.10 rc today or tomorow.