Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] python --output_folder doesn't have what filesystems are supported #1303

Closed
Tracked by #1304
tgravescs opened this issue Aug 20, 2024 · 1 comment · Fixed by #1308
Closed
Tracked by #1304

[BUG] python --output_folder doesn't have what filesystems are supported #1303

tgravescs opened this issue Aug 20, 2024 · 1 comment · Fixed by #1308
Assignees
Labels
bug Something isn't working user_tools Scope the wrapper module running CSP, QualX, and reports (python)

Comments

@tgravescs
Copy link
Collaborator

tgravescs commented Aug 20, 2024

Describe the bug
The --output_folder option in the python spark_rapids tool should specify what type of filesystems are supports. s3, hdfs, local, etc.

ie --output_folder file:/// This breaks

Right now specifying a filesystem doesn't work as it seems to append the local dir.

@tgravescs
Copy link
Collaborator Author

I think this is actually:

  • Document that only local file system is supported for --output_folder options
  • Check if a uri scheme is passed into the --output_folder option, if its something other then file: it should fail, if its file: make sure its handled properly
  • Make sure the java qualification tool is always passed the file:// uri scheme to make sure that java tools don't write to a distributed file system because it would break the abililty of the python to read from it.

@amahussein amahussein added user_tools Scope the wrapper module running CSP, QualX, and reports (python) and removed ? - Needs Triage labels Aug 20, 2024
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Aug 22, 2024
Signed-off-by: Ahmed Hussein <ahussein@nvidia.com>

Fixes NVIDIA#1303
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants