[FEA] Implement Shim inclusion / activation after dist assembly is published

[gerashegalov]

Is your feature request related to a problem? Please describe.

Our dist assembly is complex, maybe ready to be simplified which will be subject of another issue.

At this time, the set of shims participating in the assembly determines whether binary-dedupe.sh may move class files into a shared location or not.

It may change whether a class from sql-plugin-api passes the bitwise-identity test and if not the build fails.

The current practice of configuring the dist differently for the nightly and the release CI pipelines results in this kind of issue. It will allow us to avoid last minute pre-release issues related to the fact that the release version is not the same layout as has been tested for weeks with nightly pipelines.

Describe the solution you'd like
This issue proposes to implement configs to the tune of

sparkXYZ.ShimServiceProvider.enabled

that is honored at run time.

This should allow to make sure that the dist assembly stays constant throughout the entire lifecycle of a Shim including its introduction as a SNAPSHOT shim first before the release.

Describe alternatives you've considered
continue the old way

Additional context

#11744

[jlowe]

This seems to be advocating to always include snapshot shims in the artifact but have them be disabled so we can test with the jar layout we will ship. I'm all for testing something closer to what we ship, but I'm not a fan of the proposed approach for the following reasons:

• Snapshot builds can break at arbitrary points because Spark signatures could change. We don't want the nightly builds breaking and preventing an artifact from being published due to this
• The release artifact will be bigger than necessary. We already have a large jar artifact.

Instead I would propose a separate artifact for snapshot shims. The main artifact would always be built without snapshot shims. The snapshot shim artifact would be built only with snapshot shims. Regular CI pipelines use the main artifact as usual. CI pipelines for snapshot Spark versions would use the snapshot artifact. No runtime configs need to be added.

[gerashegalov]

The issue is primarily motivated by the shims for DBR like 14.3 and released versions of Spark that are not fully finished. However the user might not care about the features that are still in the works and still interested in trying it out.