[SPARK-49034][CORE] Support server-side sparkProperties replacement in REST Submission API

[dongjoon-hyun]

What changes were proposed in this pull request?

Like SPARK-49033, this PR aims to support server-side sparkProperties replacement in REST Submission API.

• For example, ephemeral Spark clusters with server-side environment variables can provide backend-resource and information without touching client-side applications and configurations.

• The place holder pattern is {{SERVER_ENVIRONMENT_VARIABLE_NAME}} style like the following.

spark/docs/configuration.md

Line 694 in 163e512

<code>-verbose:gc -Xloggc:/tmp/{{APP_ID}}-{{EXECUTOR_ID}}.gc</code>

spark/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala

Lines 233 to 234 in 163e512

	"org.apache.spark.deploy.worker.DriverWrapper",
	Seq("{{WORKER_URL}}", "{{USER_JAR}}", mainClass) ++ appArgs, // args to the DriverWrapper

Why are the changes needed?

A user can submits an environment variable holder like {{AWS_ENDPOINT_URL}} in order to use server-wide environment variables of Spark Master.

$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" \
  AWS_ENDPOINT_URL=ENDPOINT_FOR_THIS_CLUSTER \
  sbin/start-master.sh

$ sbin/start-worker.sh spark://$(hostname):7077

curl -s -k -XPOST http://localhost:6066/v1/submissions/create \
  --header "Content-Type:application/json;charset=UTF-8" \
  --data '{
          "appResource": "",
          "sparkProperties": {
            "spark.master": "spark://localhost:7077",
            "spark.app.name": "",
            "spark.submit.deployMode": "cluster",
            "spark.hadoop.fs.s3a.endpoint": "{{AWS_ENDPOINT_URL}}",
            "spark.jars": "/Users/dongjoon/APACHE/spark-merge/examples/target/scala-2.13/jars/spark-examples_2.13-4.0.0-SNAPSHOT.jar"
          },
          "clientSparkVersion": "",
          "mainClass": "org.apache.spark.examples.SparkPi",
          "environmentVariables": {},
          "action": "CreateSubmissionRequest",
          "appArgs": [ "10000" ]
  }'

• http://localhost:4040/environment/

Does this PR introduce any user-facing change?

No. This is a new feature and disabled by default via spark.master.rest.enabled (default: false)

How was this patch tested?

Pass the CIs with newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this PR too, @viirya and @yaooqinn ?

[dongjoon-hyun]

Thank you, @viirya !

[dongjoon-hyun]

Thank you, @yaooqinn .

[dongjoon-hyun]

Merged to master for Apache Spark 4.0.0-preview2.