You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For the context regarding the classloader. It is hard to reproduce but we have seen cases triggering
24/01/09 19:57:21 ERROR RapidsExecutorPlugin: Exception in the executor plugin, shutting down!
java.lang.IllegalStateException: Cannot initialize the RAPIDS Shuffle Manager
at org.apache.spark.sql.rapids.GpuShuffleEnv$.initShuffleManager(GpuShuffleEnv.scala:111)
at com.nvidia.spark.rapids.RapidsExecutorPlugin.init(Plugin.scala:397)
at org.apache.spark.internal.plugin.ExecutorPluginContainer.$anonfun$executorPlugins$1(PluginContainer.scala:125)
at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
at scala.collection.immutable.ArraySeq.flatMap(ArraySeq.scala:35)
at org.apache.spark.internal.plugin.ExecutorPluginContainer.<init>(PluginContainer.scala:113)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:211)
although the ShuffleManager class was correctly specified. In those cases rapidsShuffleManager is not instance of RapidsShuffleManagerLike. Thus. presumably classOf[RapidsShuffleManagerLike] is different than RapidsShuffleManagerLike reachable via org.apache.spark.SparkEnv.get.shuffleManager.getClass.getSuperclass.getInterfaces
Is your feature request related to a problem? Please describe.
Instead of just reporting "Cannot initialize ..."
spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuShuffleEnv.scala
Lines 106 to 113 in 47047a9
The text was updated successfully, but these errors were encountered: