[3.0.0] UpgradeReadyMonitor[G=true,W=true] failed, will retry after 60000 (NoClassDefFoundError)

[daniel-humla]

It looks like the migration from 2.x to 3.0.0 issn't working properly.

I get a lot of these logs

UpgradeReadyMonitor[G=true,W=true] failed, will retry after 60000
stack_trace:

java.lang.NoClassDefFoundError: org/apache/commons/collections4/MapUtils
	at software.amazon.kinesis.coordinator.CoordinatorStateDAO.updateCoordinatorStateWithExpectation(CoordinatorStateDAO.java:253)
	at software.amazon.kinesis.coordinator.migration.MigrationClientVersionUpgradeFrom2xState.updateDynamoStateForTransition(MigrationClientVersionUpgradeFrom2xState.java:227)
	at software.amazon.kinesis.coordinator.migration.MigrationClientVersionUpgradeFrom2xState.onMigrationReady(MigrationClientVersionUpgradeFrom2xState.java:139)
	at software.amazon.kinesis.coordinator.migration.MigrationReadyMonitor$MonitorTriggerStabilizer.call(MigrationReadyMonitor.java:322)
	at software.amazon.kinesis.coordinator.migration.MigrationReadyMonitor.run(MigrationReadyMonitor.java:149)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.collections4.MapUtils
	... 11 common frames omitted

[akidambisrinivasan]

Hello @daniel-humla Thank you for reporting the issue. Just confirming you ran into NoClassDefFoundError and not NPE, correct ? Because your title says NPE but the logs show NoClassDefFoundError.

Regarding the NoClassDefFoundError, KCL migration code uses commons-collection4 dependency which seems to be met transitively. Can you please clarify how you are building your application, maybe that excludes the transitive dependencies. The work-around would be to add the commons-collection4 dependency manually. Please let me know if you are able to try that and get it working.

[daniel-humla]

Hello @daniel-humla Thank you for reporting the issue. Just confirming you ran into NoClassDefFoundError and not NPE, correct ? Because your title says NPE but the logs show NoClassDefFoundError.

Regarding the NoClassDefFoundError, KCL migration code uses commons-collection4 dependency which seems to be met transitively. Can you please clarify how you are building your application, maybe that excludes the transitive dependencies. The work-around would be to add the commons-collection4 dependency manually. Please let me know if you are able to try that and get it working.

That's right, my bad. I'll update the title. I was a bit rushed yesterday.

We're building the application using gradle and it looks like I'm only getting org.apache.commons:commons-lang3:3.14.0 from org.apache.commons. I'll add it manually for now and try to figure out whats wrong.

./gradlew dependencyInsight --configuration runtimeClasspath --dependency org.apache.commons
> Task :dependencyInsight
org.apache.commons:commons-lang3:3.14.0
  Variant runtime:
    | Attribute Name                     | Provided     | Requested    |
    |------------------------------------|--------------|--------------|
    | org.gradle.status                  | release      |              |
    | org.gradle.category                | library      | library      |
    | org.gradle.libraryelements         | jar          | jar          |
    | org.gradle.usage                   | java-runtime | java-runtime |
    | org.gradle.dependency.bundling     |              | external     |
    | org.gradle.jvm.environment         |              | standard-jvm |
    | org.gradle.jvm.version             |              | 21           |
    | org.jetbrains.kotlin.platform.type |              | jvm          |

org.apache.commons:commons-lang3:3.14.0
\--- software.amazon.kinesis:amazon-kinesis-client:3.0.0
     \--- project :libs:event-source (requested software.amazon.kinesis:amazon-kinesis-client:{strictly 3.0.0})
          \--- project :libs:event-listener
               \--- runtimeClasspath (requested libs:event-listener)

[akidambisrinivasan]

Got it, thank you for the clarification. I have merged the fix and it will be part of next release. Meanwhile please let me know if you are able to get it working by manually adding the dependency.

[daniel-humla]

Got it, thank you for the clarification. I have merged the fix and it will be part of next release. Meanwhile please let me know if you are able to get it working by manually adding the dependency.

Hi! Sorry for not responding earlier. I got it working. Although version 3 ended up crashing our production workload, but that another story :)

[minuhong-aws]

@daniel-humla Can you reach out to us to further discuss on the issue you're having? Please email us via kinesis-kcl-feedback@amazon.com. We'd like to hear about what you experienced and provide support.