Deflater not found on spark with gatk-launch

[jamesemery]

When I tried to run the latest build of the spark gatk I got the following error message on both dataproc and the onprem cluster.
File Not Found: [/Users/emeryj/IdeaProjects/gatk/build/libIntelDeflater.so]

Making the following changes to gatk-launch seems to fix it.

@@ -23,14 +23,13 @@ BUILD_LOCATION = script +"/build/install/" + projectName + "/bin/"
 GATK_RUN_SCRIPT = BUILD_LOCATION + projectName
 BIN_PATH = script + "/build/libs"

-EXTRA_JAVA_OPTIONS="-Dsamjdk.intel_deflater_so_path=libIntelDeflater.so -Dsamjdk.compression_level=1 -DGATK_STACKTRACE_ON_USER_EXCEPTION=true "
+EXTRA_JAVA_OPTIONS="-DGATK_STACKTRACE_ON_USER_EXCEPTION=true "

 DEFAULT_SPARK_ARGS = ["--conf", "spark.kryoserializer.buffer.max=512m",
 "--conf", "spark.driver.maxResultSize=0",
 "--conf", "spark.driver.userClassPathFirst=true",
 "--conf", "spark.io.compression.codec=lzf",
 "--conf", "spark.yarn.executor.memoryOverhead=600",
-"--conf", "spark.yarn.dist.files=" + script + "/build/libIntelDeflater.so",
 "--conf", "spark.driver.extraJavaOptions=" + EXTRA_JAVA_OPTIONS,
 "--conf", "spark.executor.extraJavaOptions=" + EXTRA_JAVA_OPTIONS]

[jamesemery]

@akiezun can you look at this?

[akiezun]

the way this will be solved is going to change when we move to using intel gkl. assigning to @gspowley because he's already done most of the work here.

[droazen]

@gspowley Is this one done?

[droazen]

Also, how does this relate to #1969

[gspowley]

This File Not Found issue should be resolved with #1964. However, IntelDeflater will not be used on Spark, which is captured in #1969.

[droazen]

@gspowley Should this be closed as resolved?

[gspowley]

Yes, this should be resolved now.

[droazen]

Closing -- this is fixed