Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for shading deploy jars #13259

Closed
stuhood opened this issue Oct 14, 2021 · 2 comments · Fixed by #17251
Closed

Support for shading deploy jars #13259

stuhood opened this issue Oct 14, 2021 · 2 comments · Fixed by #17251
Labels
backend: JVM JVM backend-related issues

Comments

@stuhood
Copy link
Member

stuhood commented Oct 14, 2021

#12796 added support for building deploy ("fat") jars. Most usecases of deploy jars also need the ability to shade the contents of the jar (in order to deploy it to Hadoop, Spark, or other environments without classloader isolation).

Most likely this will involve using 9and likely resurrecting) our fork of jarjar... but it's also possible that modern alternatives have emerged, so we should look at what other tools are doing first.

@jsirois
Copy link
Contributor

jsirois commented Oct 14, 2021

Most usecases of deploy jars also need the ability to shade the contents of the jar (in order to deploy it to Hadoop, Spark, or other environments without classloader isolation).

I'll point out that (I think) shading for user binaries was never implemented in Pants V1 although it would be useful as you suggest. The driving need then was for jvm tools that ran using a mixture of their required classpath and user classpath items.

@stuhood stuhood added the backend: JVM JVM backend-related issues label Oct 14, 2021
@stuhood
Copy link
Member Author

stuhood commented Oct 14, 2021

Most usecases of deploy jars also need the ability to shade the contents of the jar (in order to deploy it to Hadoop, Spark, or other environments without classloader isolation).

I'll point out that (I think) shading for user binaries was never implemented in Pants V1 although it would be useful as you suggest. The driving need then was for jvm tools that ran using a mixture of their required classpath and user classpath items.

It was: https://v1.pantsbuild.org/jvm_projects.html#omitting-or-shading-the-contents-of-a-binary

@stuhood stuhood removed the 2022-idea label Aug 19, 2022
stuhood pushed a commit that referenced this issue Nov 4, 2022
This is a Request For Comments with a PoC with a minimal implementation for supporting shading rules in JVM targets, i.e.: the `deploy_jar` target.

The implementation is straightforwardly based on JarJar and it adds three aliases to the build files to help defining each of the given rules.

For example, when shading a `deploy_jar`, a `BUILD` file would look something like:

```python
deploy_jar(
    name="my_deploy_jar",
    shading_rules=[
      shading_rename(pattern="com.fasterxml.jackson.**", replacement="mycompany.jackson.@1"),
      shading_zap(pattern="javax.servlet.**"),
      shading_keep(pattern="org.apache.spark.**"),
    ]
)
```

The implementation doesn't solve the potential problems that may show up when merging files from different JARs that are at the same relative location in each of those files. That is being left out-of-scope of this implementation.

Fixes #13259.

[ci skip-build-wheels]
[ci skip-rust]
@stuhood stuhood moved this to Done in 2022 Priorities Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend: JVM JVM backend-related issues
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants