Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include JDKs in Dists #1510

Merged
merged 19 commits into from
Aug 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions changelog/@unreleased/pr-1510.v2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type: feature
feature:
description: Add `jdks` property to `distribution` extension to help include JDKs
into dists.
links:
- https://github.com/palantir/sls-packaging/pull/1510
1 change: 1 addition & 0 deletions gradle-sls-packaging/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ dependencies {
testImplementation 'com.netflix.nebula:nebula-test'
testImplementation 'org.awaitility:awaitility'
testImplementation 'org.junit.jupiter:junit-jupiter'
testImplementation 'org.rauschig:jarchivelib'
testRuntimeOnly 'org.junit.vintage:junit-vintage-engine'
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ public class SlsBaseDistPlugin implements Plugin<Project> {

public static final String SLS_DIST_USAGE = "sls-dist";

public static final GradleVersion MINIMUM_GRADLE = GradleVersion.version("6.1");
public static final GradleVersion MINIMUM_GRADLE = GradleVersion.version("7.6");
Copy link
Contributor Author

@CRogers CRogers Aug 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provider#zip is introduced in 6.6. However, it seems that gradle introduced plugin bytecode rewriting sometime between 6.1 and 6.6. This means 6.6 now falls over on the jackson 2.15.2 dep from baseline, so we need to go all the way to 7.6

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't we need 7.6.2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

happy to merge if 7.6 doesn't break and have a lower min version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I wonder why it works...


@Override
public final void apply(Project project) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@

package com.palantir.gradle.dist.service;

import java.util.Arrays;
import java.util.concurrent.Callable;
import org.gradle.api.JavaVersion;
import org.gradle.api.Project;
import org.gradle.api.file.DuplicatesStrategy;
import org.gradle.api.provider.Provider;
Expand Down Expand Up @@ -53,6 +55,21 @@ static void configure(
t.setFileMode(0755);
});

// We do this trick of iterating through every java version and making a from with a lazy value to be lazy
// enough to handle the case where another plugin has set the value of the jdks property based on the result
// of resolving a configuration. Unfortunately, lots of our internal plugins/build.gradle force the value
// of the distTar task at configuration time, so this would cause a Configuration to resolved at
// configuration time (which is disallowed) with the naive getting the value from the property and looping
// over it. Reading the code below, you might be concerned that it would create empty directories for unset
// java versions, but Gradle does not appear to do this for empty file collections.
Arrays.stream(JavaVersion.values()).forEach(javaVersion -> {
root.from(
distributionExtension.getJdks().getting(javaVersion).orElse(project.provider(project::files)),
t -> {
t.into(distributionExtension.jdkPathInDist(javaVersion));
});
});

root.into("service/lib", t -> {
t.from(jarTask);
t.from(project.getConfigurations().named("runtimeClasspath"));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import groovy.lang.DelegatesTo;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import javax.annotation.Nullable;
import javax.inject.Inject;
import org.gradle.api.Action;
Expand All @@ -48,6 +49,7 @@ public class JavaServiceDistributionExtension extends BaseDistributionExtension
private final ListProperty<String> defaultJvmOpts;
private final ListProperty<String> excludeFromVar;
private final MapProperty<String, String> env;
private final MapProperty<JavaVersion, Object> jdks;

private final ObjectFactory objectFactory;

Expand All @@ -63,7 +65,16 @@ public JavaServiceDistributionExtension(Project project) {
.getTargetCompatibility()));
mainClass = objectFactory.property(String.class);

jdks = objectFactory.mapProperty(JavaVersion.class, Object.class).empty();

javaHome = objectFactory.property(String.class).value(javaVersion.map(javaVersionValue -> {
Optional<Object> possibleIncludedJdk =
Optional.ofNullable(jdks.getting(javaVersionValue).getOrNull());

if (possibleIncludedJdk.isPresent()) {
return jdkPathInDist(javaVersionValue);
}

boolean javaVersionLessThanOrEqualTo8 = javaVersionValue.compareTo(JavaVersion.VERSION_1_8) <= 0;
if (javaVersionLessThanOrEqualTo8) {
return "";
Expand All @@ -85,7 +96,7 @@ public JavaServiceDistributionExtension(Project project) {
excludeFromVar = objectFactory.listProperty(String.class);
excludeFromVar.addAll("log", "run");

env = objectFactory.mapProperty(String.class, String.class).empty();
env = objectFactory.mapProperty(String.class, String.class);
CRogers marked this conversation as resolved.
Show resolved Hide resolved
setProductType(ProductType.SERVICE_V1);
}

Expand Down Expand Up @@ -213,6 +224,10 @@ public final void setEnv(Map<String, String> env) {
this.env.set(env);
}

public final MapProperty<JavaVersion, Object> getJdks() {
return jdks;
}

public final Provider<GcProfile> getGc() {
return gc;
}
Expand Down Expand Up @@ -249,4 +264,20 @@ private static GcProfile getDefaultGcProfile(JavaVersion javaVersion) {
}
return new GcProfile.Throughput();
}

final String jdkPathInDist(JavaVersion javaVersionValue) {
// We put the JDK in a directory that contains the name and version of service. This is because in our cloud
// environments (and some customer environments), there is a third party security scanning tool that will report
// vulnerabilities in the JDK by printing a path, but does not display symlinks. This means it's hard to tell
// from a scan report which service is actually vulnerable, as our internal deployment infra uses symlinks,
// and you end up with a report like so:
// Path: /opt/palantir/services/.24710105/service/jdk17
// rather than more useful:
// Path: /opt/palantir/services/.24710105/service/multipass-2.1.3-jdks/jdk17
// which is implemented below.

return String.format(
"service/%s-%s-jdks/jdk%s",
getDistributionServiceName().get(), project.getVersion(), javaVersionValue.getMajorVersion());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
import java.nio.file.Path;
import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.stream.Collectors;
import org.gradle.api.Action;
import org.gradle.api.InvalidUserCodeException;
Expand Down Expand Up @@ -222,7 +224,7 @@ public void execute(Task _task) {
task.getAddJava8GcLogging().set(distributionExtension.getAddJava8GcLogging());
task.getJavaHome().set(distributionExtension.getJavaHome());
task.getJavaVersion().set(distributionExtension.getJavaVersion());
task.getEnv().set(distributionExtension.getEnv());
task.getEnv().set(userConfiguredEnvWithJdkEnvVars(distributionExtension));
});

TaskProvider<CreateInitScriptTask> initScript = project.getTasks()
Expand Down Expand Up @@ -323,6 +325,22 @@ public void execute(Task _task) {
project.getArtifacts().add(SlsBaseDistPlugin.SLS_CONFIGURATION_NAME, distTar);
}

private static Provider<Map<String, String>> userConfiguredEnvWithJdkEnvVars(
JavaServiceDistributionExtension distributionExtension) {

return distributionExtension.getEnv().zip(distributionExtension.getJdks(), (userConfiguredEnv, jdks) -> {
Map<String, String> actualEnv = new LinkedHashMap<>(userConfiguredEnv);

jdks.keySet().stream().sorted().forEach(javaVersion -> {
actualEnv.put(
"JAVA_" + javaVersion.getMajorVersion() + "_HOME",
distributionExtension.jdkPathInDist(javaVersion));
});

return Collections.unmodifiableMap(actualEnv);
});
}

private static void replaceManifestClasspath(Path windowsScript, String manifestClassPathArchiveFileName)
throws IOException {
// Replace standard classpath with pathing jar in order to circumnavigate length limits:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

public final class Versions {

public static final String GRADLE_CONSISTENT_VERSIONS = "1.27.0";
public static final String GRADLE_CONSISTENT_VERSIONS = "2.15.0";

private Versions() {}
}
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deprecation warning with new higher version of gradle

mavenCentral()
}

Expand Down Expand Up @@ -219,7 +218,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}

Expand Down Expand Up @@ -599,7 +597,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}
distribution {
Expand Down Expand Up @@ -633,7 +630,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}
version '0.0.1'
Expand Down Expand Up @@ -678,7 +674,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}
version '0.0.1'
Expand Down Expand Up @@ -797,7 +792,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
args "hello"
}
repositories {
jcenter()
mavenCentral()
}
dependencies {
Expand All @@ -813,7 +807,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
id 'java-library'
}
repositories {
jcenter()
mavenCentral()
}
dependencies {
Expand Down Expand Up @@ -932,7 +925,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
enableManifestClasspath true
}
repositories {
jcenter()
mavenCentral()
}
dependencies {
Expand All @@ -948,7 +940,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
id 'java-library'
}
repositories {
jcenter()
mavenCentral()
}
dependencies {
Expand Down Expand Up @@ -1031,7 +1022,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}

Expand Down Expand Up @@ -1066,7 +1056,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}

Expand Down Expand Up @@ -1101,7 +1090,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
}

repositories {
jcenter()
mavenCentral()
}

Expand Down Expand Up @@ -1385,7 +1373,6 @@ class JavaServiceDistributionPluginTests extends GradleIntegrationSpec {
project.group = 'service-group'

repositories {
jcenter()
mavenCentral()
}

Expand Down
Loading