Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(fill): improve performance when xdist is enabled #1118

Merged
merged 6 commits into from
Jan 28, 2025
Merged

feat(fill): improve performance when xdist is enabled #1118

merged 6 commits into from
Jan 28, 2025

Conversation

marioevz
Copy link
Member

@marioevz marioevz commented Jan 23, 2025 •
edited by danceratopz
Loading

🗒️ Description

Improves xdist workload distribution method by reverting --dist flag to its default value, load, and also make the fixture collector multi-process safe by creating a lock file per fixture file generated, and locking->reading->writing the file instead of overwriting it.

🔗 Related Issues

Description of xdist scoping here #159 (comment)

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz marioevz added scope:pytest Scope: Changes EEST's pytest plugins type:chore Type: Chore labels Jan 23, 2025
@danceratopz danceratopz changed the title feat(fixtures): Improve xdist fixture filling feat(fill): improve xdist fixture filling Jan 27, 2025
@danceratopz danceratopz changed the title feat(fill): improve xdist fixture filling feat(fill): improve performance when xdist is enabled Jan 27, 2025
@danceratopz danceratopz added type:feat type: Feature scope:fill Scope: fill command and removed type:chore Type: Chore labels Jan 27, 2025
@danceratopz danceratopz mentioned this pull request Jan 27, 2025
Merged
danceratopz
danceratopz reviewed Jan 27, 2025
View reviewed changes
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks! Few comments and suggestions in #1130.

@danceratopz
refactor,test(fill): suggestions for x-dist-mode-fixture-collector #…
787b910 
…1118 (#1130)

* chore(fill): remove `--index` flag; it has no effect

The `--index` flag is of `store_true` type with default True; it's redundant.

* test(fill): remove unnecessary test parameter

* refactor(fill): group variable definitions for readability
danceratopz
danceratopz approved these changes Jan 28, 2025
View reviewed changes
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMMMMM!

@danceratopz danceratopz merged commit 136e15f into main Jan 28, 2025
21 checks passed
@danceratopz danceratopz deleted the x-dist-mode-fixture-collector branch January 28, 2025 09:41
marioevz added a commit to marioevz/execution-spec-tests that referenced this pull request Jan 30, 2025
@marioevz @danceratopz
feat(fill): improve performance when xdist is enabled (ethereum#1118)
ee8e9fb 
Co-authored-by: danceratopz <danceratopz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danceratopz danceratopz danceratopz approved these changes

Assignees
No one assigned
Labels
scope:fill Scope: fill command scope:pytest Scope: Changes EEST's pytest plugins type:feat type: Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@marioevz @danceratopz