[SPARK-4730][YARN] Warn against deprecated YARN settings #3590
Conversation
|
Test build #24113 has started for PR 3590 at commit
|
|
Test build #24113 has finished for PR 3590 at commit
|
|
Test PASSed. |
| "SPARK_MASTER_MEMORY" -> "SPARK_DRIVER_MEMORY or --driver-memory through spark-submit", | ||
| "SPARK_WORKER_INSTANCES" -> "SPARK_WORKER_INSTANCES or --num-executors through spark-submit", | ||
| "SPARK_WORKER_MEMORY" -> "SPARK_EXECUTOR_MEMORY or --executor-memory through spark-submit", | ||
| "SPARK_WORKER_CORES" -> "SPARK_EXECUTOR_CORES or --executor-cores through spark-submit") |
There was a problem hiding this comment.
aren't essentially all of the env variables deprecated? I know we have warnings on some throughout the code but haven't checked for all of them.
There was a problem hiding this comment.
We actually still document the rest of them in spark-env.sh
|
Test build #24287 has started for PR 3590 at commit
|
|
Test build #24287 has finished for PR 3590 at commit
|
|
Test PASSed. |
|
Any other comments @tgravescs? |
|
+1 looks good. |
|
LGTM, too, so I'm going to merge this into |
See https://issues.apache.org/jira/browse/SPARK-4730. Author: Andrew Or <andrew@databricks.com> Closes #3590 from andrewor14/yarn-settings and squashes the following commits: 36e0753 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-settings dcd1316 [Andrew Or] Warn against deprecated YARN settings (cherry picked from commit 27c5399) Signed-off-by: Josh Rosen <joshrosen@databricks.com>
| "SPARK_WORKER_MEMORY" -> "SPARK_EXECUTOR_MEMORY or --executor-memory through spark-submit", | ||
| "SPARK_WORKER_CORES" -> "SPARK_EXECUTOR_CORES or --executor-cores through spark-submit") | ||
| // Do the same for deprecated properties: property -> suggestion | ||
| val deprecatedProps = Map("spark.master.memory" -> "--driver-memory through spark-submit") |
There was a problem hiding this comment.
SPARK_MASTER_MEMORY and spark.master.memory are not applicable in yarn-client mode
should be removed, please refer SPARK-1953
There was a problem hiding this comment.
@zuxqoj the "master" here doesn't actually refer to the ApplicationMaster, but the driver, so SPARK-1953 is actually unrelated. It's really the result of bad naming that we inherited from a long time ago, and this is why we're deprecating it. I agree we should just remove it for 1.4.
There was a problem hiding this comment.
@andrewor14 In SPARK-1953 properties related to driver memory were removed form yarn-client mode. SPARK_MASTER_MEMORY and spark.master.memory are related to driver memory, so they should also be removed. I have created jira for it https://issues.apache.org/jira/browse/SPARK-5951
There was a problem hiding this comment.
SPARK-1953 did not remove any driver memory setting from yarn-client mode; it just introduced new ones for the AM, so it's unrelated to the discussion here. I do agree that we should remove these configs for 1.4, but the reason has nothing to do with the ApplicationMaster. Thanks for filing that JIRA.
See https://issues.apache.org/jira/browse/SPARK-4730.