improve performance for merging markers from overrides

[radoering]

Pull Request Check List

Related to: #9956

• Added tests for changed code.
• Updated documentation for changed code.

This is far from fixing the regression reported in #9956. It "just" improves the performance (compared to Poetry 2.0) significantly.

Using the simplified repro from #9956 (comment) (thanks @dimbleby 🙏), Poetry 1.8 can solve it in 0.2 s, no matter if there is only one extra or 14 extras. Poetry 2 takes longer the more extras are defined:

[tool.poetry.dependencies]
python = ">=3.8,<3.11"
opencv-python = {version = "^4.6.0.66", optional = true}
ale-py = { version = "0.8.1", optional = true }

[tool.poetry.extras]
extra01 = ["ale-py"]
extra02 = ["ale-py"]
extra03 = ["ale-py"]
# ...

number of extras	poetry 2.0	PR
1	6.5 s	0.5 s
2	>100 s	1.7 s
3	>100 s	6.4 s
4	>100 s	19 s
5	>100 s	57 s

Summary by Sourcery

Tests:

• Updated tests for the solver internals to reflect the performance improvements.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request improves the performance of merging markers from overrides by changing the merge_packages_from_override function to intersect all override markers at once, rather than iteratively. This change significantly reduces the time complexity of the operation, resulting in substantial performance gains, especially when dealing with multiple extras.

Sequence diagram for optimized marker merging

sequenceDiagram
    participant S as Solver
    participant M as MarkerMerger
    participant P as Package

    S->>M: merge_override_packages(override_packages)
    activate M
    M->>M: Group packages by identity
    M->>M: Check if markers are same across overrides
    alt Markers are same
        M->>M: Apply performance shortcut
        M->>M: Union override markers once
    else Markers differ
        M->>M: Apply general algorithm
        M->>M: Iteratively merge markers
    end
    M->>P: Update package dependencies
    M-->>S: Return merged packages
    deactivate M

Loading

Class diagram showing the refactored marker merging structure

classDiagram
    class Package {
        +requires: List[Dependency]
        +add_dependency(dep)
    }

    class TransitivePackageInfo {
        +depth: int
        +groups: Set
        +markers: Dict
    }

    class BaseMarker {
        +intersect(other)
        +union(other)
        +without_extras()
    }

    class Solver {
        -_provider: Provider
        +_solve_in_compatibility_mode(overrides)
        +_solve()
    }

    Package -- TransitivePackageInfo
    TransitivePackageInfo -- BaseMarker
    Solver -- Package

    note for TransitivePackageInfo "Stores package metadata and markers"
    note for BaseMarker "Handles marker operations with improved performance"

Loading

Flow diagram of improved marker merging process

graph TD
    A[Start] --> B[Collect all packages and overrides]
    B --> C{Are markers same\nfor all overrides?}
    C -->|Yes| D[Use performance shortcut]
    C -->|No| E[Use general algorithm]
    D --> F[Union override markers]
    F --> G[Intersect with package markers]
    E --> H[Iteratively merge markers]
    H --> I[Union with existing markers]
    G --> J[Update package dependencies]
    I --> J
    J --> K[End]

Loading

File-Level Changes

Change	Details	Files
Refactored the merge_packages_from_override function to merge_override_packages, changing its logic to intersect all override markers at once.	Changed the merge_packages_from_override function to intersect all override markers in a single operation. Updated the tests to reflect the changes in the merge_override_packages function. Renamed the function from merge_packages_from_override to merge_override_packages. Renamed the test function from test_merge_packages_from_override to test_merge_override_packages.	src/poetry/puzzle/solver.py tests/puzzle/test_solver_internals.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @radoering - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}