Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NU-1979] Flink MiniCluster: job recovery on start #7710

Merged
merged 13 commits into from
Mar 25, 2025

Conversation

arkadius
Copy link
Member

@arkadius arkadius commented Mar 21, 2025

Describe your changes

This change adapts the solution that worked for embedded DM to work with every DM (including the flink mini-cluster). Now, it is part of the reconciliation process done on the designer's side instead of the DM's side.

Checklist before merge

  • Related issue ID is placed at the beginning of PR title in [brackets] (can be GH issue or Nu Jira issue)
  • Code is cleaned from temporary changes and commented out lines
  • Parts of the code that are not easy to understand are documented in the code
  • Changes are covered by automated tests
  • Showcase in dev-application.conf added to demonstrate the feature
  • Documentation added or updated
  • Added entry in Changelog.md describing the change from the perspective of a public distribution user
  • Added MigrationGuide.md entry in the appropriate subcategory if introducing a breaking change
  • Verify that PR will be squashed during merge

@arkadius arkadius force-pushed the preview/minicluster-job-recovery branch from 8548a91 to a0d2bf5 Compare March 21, 2025 13:59
@arkadius arkadius changed the title Preview/minicluster job recovery [NU-1979] Flink MiniCluster: job recovery on start Mar 21, 2025
Copy link
Contributor

updated: #7718
⚠️ Be careful! Snapshot changes are not necessarily the cause of the error. Check the logs.

@arkadius arkadius marked this pull request as ready for review March 21, 2025 14:49
@@ -72,84 +72,6 @@ class StreamingEmbeddedDeploymentManagerTest
manager.getScenarioDeploymentsStatuses(name).futureValue.value shouldBe List.empty
}

test("Run persisted scenario deployments") {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not so easy to test embedded DM after removing ProcessingTypeDeployedScenariosProvider. To do this, we should probably add some separated test for this DM inside designer module. I think that it is not a priority now, so I only tested if it works manually.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe should we leave todo somewhere?

@arkadius arkadius requested a review from coutoPL March 21, 2025 15:03
Copy link
Contributor

@lukasz-bigorajski lukasz-bigorajski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arkadius arkadius merged commit 1dfe3ad into staging Mar 25, 2025
9 checks passed
@arkadius arkadius deleted the preview/minicluster-job-recovery branch March 25, 2025 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants