[SPARK-42204][CORE] Add option to disable redundant logging of TaskMetrics internal accumulators in event logs

[JoshRosen]

What changes were proposed in this pull request?

This PR adds an off-by-default option to JsonProtocol to have it exclude certain redundant accumulator information from Spark event logs in order to save space and processing time.

Several event logs types contain both TaskMetrics and Accumulables, but there is redundancy in how the TaskMetrics data is stored:

• TaskMetrics are stored in a map called "Task Metrics" which maps from metric names to metric values.
• An "Accumulables" field contains information on accumulator updates from the task, but this field includes updates from the TaskMetrics internal accumulators (both the value from the task, plus a running "sum-so-far" from all of the tasks completed up to that point).

The redundant task metrics accumulables are not actually used by the Spark History Server: I verified this by reading AppStatusListener and SQLAppStatusListener.

I believe that this redundancy was introduced back in SPARK-10620 when Spark 1.x's separate TaskMetrics implementation was replaced by the current accumulator-based version.

In this PR, I add logic to exclude TaskMetrics internal accumulators when writing this field (if a new flag is enabled).

The new spark.eventLog.includeTaskMetricsAccumulators configuration (default false, meaning "keep the redundant information") can be set to true to exclude these redundant internal accumulator updates.

For now, I am merging this off-by-default, but in a followup PR for Spark 4.0.0 we might consider a change-of-default to true (in which case the flag would serve as an "escape-hatch" for users who want to restore the old behavior. Although I think it's somewhat unlikely that third-party non-Spark consumers of the event logs would be relying on this redundant information, this is changing a longstanding user-facing data format and thus needs a flag.

Why are the changes needed?

This change reduces the size of Spark event logs, especially for logs from applications that run many tasks. It should also have slight benefits on event log read and write speed (although I haven't tried to quantify this).

Does this PR introduce any user-facing change?

No user-facing changes in Spark History Server.

This flag's effects could be considered a user-facing change from the perspective of third-party code which does its own direct processing of Spark event logs, hence the config. However, in this PR (by itself) the flag is off-by-default. Out-of-the-box user-facing changes will be discussed / proposed in a separate flag-flip PR.

How was this patch tested?

New unit tests in JsonProtocolSuite.

Manual tests of event log size in spark-shell with a job that runs spark.parallelize(1 to 1000, 1000).count(). For this toy query, this PR's change shrunk the uncompressed event log size by ~15%. The relative size reduction will be even greater once other issues like https://issues.apache.org/jira/browse/SPARK-42206 or https://issues.apache.org/jira/browse/SPARK-42203 are fixed. The relative reduction will be smaller for tasks with many SQL metrics because those accumulables cannot be excluded.

[JoshRosen]

This is marked as "WIP" because I'm still deciding whether this merits a note in a migration guide / release note.

It's possible that I'm being overly cautious regarding backwards-compatibility for theoretical non-Spark consumers of these event logs, but I'm trying to avoid a situation where an impacted user has no straightforward short-term mitigation.

[JoshRosen]

This pattern is a bit weird, so I'd like to explain it in greater detail here:

To date, JsonProtocol has just been a big object with no configuration options of its own. All other event logging configurations take effect at higher levels, such as EventLoggingListener. We cannot implement this particular change at those levels, though: we shouldn't mutate the mutable TaskInfo or StageInfo objects referenced by the listener events and adding copy() methods to them would be expensive.

JsonProtocol is currently implemented as a global object, so there's no straightforward way to make it easily configurable and testable. If I did something like "get the active SparkContext's configuration and read the configuration value" on each call then this would destroy performance because configuration reading is not cheap.

One approach would be to refactor this so that JsonProtocol is no longer a singleton: we could give it a proper constructor which accepts a SparkConf. That's a much larger refactoring than I want to undertake right now, though, and it would involve significant updates in a bunch of test code.

The approach taken here is to thread through an JsonProtocolOptions class which holds the configuration values. Right now it only holds a single value, but this is done as future-proofing in case we add additional configurability. The lone caller in EventLoggingListener constructs the JsonProtocolOptions using SparkConf, so the configuration is only read from the conf once (avoiding conf reader performance overheads).

This is a more C-style approach than the usual OOP approach that we generally take, but I think it's okay in this limited internal context.

[mridulm]

Sounds good to me.

[JoshRosen]

See https://issues.apache.org/jira/browse/SPARK-42205 for more details. Once that issue is fixed, we can drop the testcases for TaskStart, JobStart, and StageSubmitted here.

[JoshRosen]

Method using default configurations is retained for the purposes of avoiding changes in a bunch of test code across several suites.

[mridulm]

Just couple of minor comments, looks good to me.

Thanks for fixing this @JoshRosen !

[mridulm]

Add a note for this sparkEventToJsonString that it is only for tests ?

[mridulm]

This is marked as "WIP" because I'm still deciding whether this merits a note in a migration guide / release note.
It's possible that I'm being overly cautious regarding backwards-compatibility for theoretical non-Spark consumers of these event logs, but I'm trying to avoid a situation where an impacted user has no straightforward short-term mitigation.

I have seen usages of this unfortunately - so having the flag would help.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[mridulm]

We should remove the [WIP] before merging :-)

[JoshRosen]

We should remove the [WIP] before merging :-)

I will; since it's been a while since I looked at this code, I want to do another self-review and another round of manual testing before I merge. I re-opened the PR to fix merge conflicts and let CI run the tests.

I'm considering whether we might want to change this default for Spark 4.0; if so, I would propose to do that in a separate followup PR to also update docs + migration guide.

[rednaxelafx]

This looks good to me as well (non-binding). Thanks a lot for reviving this improvement!
Agreed that the historical Spark UI itself wouldn't be affected. The REST API will get some exposure to the change though. Hopefully there's no client out in the wild that relies on this redundant information.

The JsonProtocolOptions part looks pragmatic and I'm okay with that. If people are really annoyed by that, there's always the path of creating a JsonProtocol class that wraps the JsonProtocolOptions and delegates all operations down to the existing companion object. I don't think we have to do there here.

[JoshRosen]

I pushed an empty commit to retest this. The only failing test is org.apache.spark.storage.BlockManagerDecommissionIntegrationSuite which was preexistingly flaky for reasons unrelated to this change.

Since I've been so delayed in actually merging this, I took another look to see if any intervening changes might have introduced logical conflicts but I didn't spot any potentially problematic changes.

Especially given that this is flagged-off-by-default opt-in functionality, I feel that this is a low-risk change.

Therefore I'm going to merge this now (finally!). In separate followups I may end up building on the configuration-plumbing infrastructure introduced here to add other opt-in size reduction optimizations.