Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude pycache when copying packet inputs and outputs. #45

Merged
merged 2 commits into from
Apr 19, 2024
Merged

Conversation

plietar
Copy link
Member

@plietar plietar commented Apr 18, 2024

Python tends to produce a lot of __pycache__ folders. These can be found both in a report's source directory, if the user ran the entrypoint script outside of an orderly context, and in the packet's working directory after executing the packet.

There is never a good reason to include these in packets and we can exclude them unconditionally, both when copying files in and out of a packet's working directory.

As a side-effect of now using the same function to copy files in and out of a packet, this commit also fixes the use of directory in a report's source directory. Previously we were attempting to call shutil.copy2 on every entry, which would fail on directories.

@plietar plietar requested a review from richfitz April 18, 2024 17:53
Copy link
Member

@richfitz richfitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - do we need to exclude .pyc files or are these never found out of the cache directory any more?

@plietar plietar changed the base branch from improve-tests to main April 19, 2024 15:06
Python tends to produce a lot of `__pycache__` folders. These can be
found both in a report's source directory, if the user ran the
entrypoint script outside of an orderly context, and in the packet's
working directory after executing the packet.

There is never a good reason to include these in packets and we can
exclude them unconditionally, both when copying files in and out of a
packet's working directory.

As a side-effect of now using the same function to copy files in and out
of a packet, this commit also fixes the use of directory in a report's
source directory. Previously we were attempting to call `shutil.copy2`
on every entry, which would fail on directories.
@plietar
Copy link
Member Author

plietar commented Apr 19, 2024

do we need to exclude .pyc files or are these never found out of the cache directory any more?

Seems like as of Python 3.2 they are always in the pycache directory, so this should be enough.

https://docs.python.org/dev/whatsnew/3.2.html#pep-3147-pyc-repository-directories

to prevent all of these new files from cluttering source directories, the pyc files are now collected in a __pycache__ directory stored under the package directory.

@plietar plietar merged commit 6ec900c into main Apr 19, 2024
7 checks passed
@plietar plietar deleted the mrc-5262 branch May 13, 2024 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants