[SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_COMPRESS only once

[rxin]

What changes were proposed in this pull request?

Previously SPARK-24519 created a modifiable config SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_COMPRESS. However, the config is being parsed for every creation of MapStatus, which could be very expensive. Another problem with the previous approach is that it created the illusion that this can be changed dynamically at runtime, which was not true. This PR changes it so the config is computed only once.

How was this patch tested?

Removed a test case that's no longer valid.

[SparkQA]

Test build #96449 has finished for PR 22521 at commit f6f9658.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #96495 has finished for PR 22521 at commit 4af98e7.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Jenkins, retest this please.

[SparkQA]

Test build #4350 has started for PR 22521 at commit 4af98e7.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

@rxin . Could you fill the PR description, too?

[rxin]

yup; just did

[dongjoon-hyun]

Thank you.
+1, LGTM.

[dongjoon-hyun]

nit. Could you add [CORE] to the PR title, too?

[SparkQA]

Test build #96534 has finished for PR 22521 at commit 4af98e7.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dilipbiswal]

retest this please

[SparkQA]

Test build #96538 has finished for PR 22521 at commit 4af98e7.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #96554 has finished for PR 22521 at commit 4af98e7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

seems like our tests are really flaky

[cloud-fan]

retest this please

[dongjoon-hyun]

Ya. It's amazing to see 5 different failures in the same PR. :(

1. org.apache.spark.scheduler.MapStatusSuite.SPARK-24519: HighlyCompressedMapStatus has configurable threshold
2. This patch fails SparkR unit tests.
3. This patch fails due to an unknown error code, -9.
4. This patch fails PySpark unit tests.
5. org.apache.spark.sql.streaming.continuous.ContinuousSuite.query without test harness

[SparkQA]

Test build #96571 has finished for PR 22521 at commit 4af98e7.

• This patch fails from timeout after a configured wait of 400m.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #96585 has finished for PR 22521 at commit 4af98e7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #96584 has finished for PR 22521 at commit 4af98e7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4352 has finished for PR 22521 at commit 4af98e7.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #96612 has finished for PR 22521 at commit 4af98e7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4353 has finished for PR 22521 at commit 4af98e7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Merged to master/branch-2.4.