-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-11518] [Deploy, Windows] Handle spaces in Windows command scripts #10789
Conversation
Can one of the admins verify this patch? |
Seems reasonable to me on the surface, but I'm not a frequent Windows user so it would be good to have someone else look, too. @tsudukim, would you be interested in looking this over, since you've made a lot of modifications to these scripts in the past? |
@JoshRosen Thank you for your involvement. It seems a good fix, but it doesn't work for my environment because we should fix more files to handle spaces properly.
As this is just a example, there are many other codes that should be double-quoted other than this. |
@tritab can you have a look at other similar instances in other |
Aside: has someone written something like shellcheck for Windows shell scripts? That'd be a really quick way to identify all of the places that need quoting, provided such a tool exists. |
Thanks for the feedback. I'll dig in a bit deeper. |
I made a number of updates to all of the .cmd files, @tsudukim please review to see how it works in your environment. It appears to handle a spark install under "Program Files" now. However, I have a couple of outstanding concerns that could use some work.
Any feedback is welcome. @JoshRosen I looked briefly, but wasn't able to find a tool similar to shellcheck for Windows. That certainly would be helpful though. |
Thank you @tritab . The 1st is that as @tritab already mentioned, if we ctrl-c, our terminal might be left in the SPARK_HOME folder. And the 2nd is, if we change current directory, I'm worried that some command which specifies relative path doesn't work properly. For example, when we execute spark-submit on yarn, we specify application JAR file like this:
If we change current directory, the relative path seems not to work. Did you face some problems to use double quotations like
instead of using |
I think just adding the quotation is good to solve this problem. |
Thanks for the feedback, I'll rework with quotes and push another commit soon. |
@tsudukim I reworked pushd with quoting as you suggested. This tests successfully on my Windows 10 machine for spark-shell, pyspark, sparkR and run-example. I would appreciate it if you could review as well. |
The changes are mostly quoting and those look fine to me. There are a few cases where you removed %dp0 and I'm not entirely familiar with this, but I think you know what you're doing. The SPARK_HOME change looks good. I'd love to get a +1 from @tsudukim before merging |
@srowen Thank you for your review. The%~dp0 changes were to get the current directory working properly. I should mention I tested with spark-submit as well. I did not find any issues with the latest commit. I would also like someone else to test on Windows before merging. Perhaps @andrewor14 would be willing to review this? |
Merged to master. This looks well thought out and correct to my knowledge |
No description provided.