Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize creating cluster snapshot #7630

Merged
merged 1 commit into from
Jan 3, 2025
Merged

Parallelize creating cluster snapshot #7630

merged 1 commit into from
Jan 3, 2025

Conversation

macsko
Copy link
Member

@macsko macsko commented Dec 19, 2024 •
edited
Loading

What type of PR is this?

/kind feature

What this PR does / why we need it:

Creating cluster snapshots using SetClusterState can take a long time, especially in large clusters with plenty of pods. Fortunately, we can parallelize the work done per each node. This PR does it by calling workqueue.ParallelizeUntil capped at 16 workers.

I verified the performance and it gives me near 7 times faster execution for 1k nodes, 40k pods scenario, but it depends on the machine's CPU.

In the future, basic store used for scale down can be also changed similarly.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Adding pods when creating a cluster snapshot for scale up is now parallelized, improving performance. Parallelization can be adjusted or disabled by setting the cluster-snapshot-parallelization flag.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. area/cluster-autoscaler labels Dec 19, 2024
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 19, 2024
@macsko
Copy link
Member Author

macsko commented Dec 19, 2024

/cc @yaroslava-serdiuk @aleksandra-malinowska

@yaroslava-serdiuk
Copy link
Contributor

Wow, cool feature!
Let's make in backwards compatible by introducing a flag. We can default it to 16, but in this case let's add release note since we change the behavior. In case this feature will brake someone (seems unlikely, but still we need to consider all scenarious) it should be easy to disable it (set flag to 1).

@yaroslava-serdiuk
Copy link
Contributor

cc: @x13n

@macsko macsko force-pushed the parallelize_cluster_snapshot_creation branch from 7368ae8 to f621876 Compare December 19, 2024 11:04
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 19, 2024
@macsko macsko force-pushed the parallelize_cluster_snapshot_creation branch from f621876 to 76eb858 Compare December 19, 2024 11:09
@macsko
Copy link
Member Author

macsko commented Dec 19, 2024

Let's make in backwards compatible by introducing a flag.

Added. If the flag is > 1, the snapshot is created using a new way.

@yaroslava-serdiuk
Copy link
Contributor

Overall LGTM
Thanks! Could you also mention how did you measure performance? I suppose it's a CA loop duration, right?

@x13n Could you take a look?

x13n
x13n requested changes Dec 20, 2024
View reviewed changes
Copy link
Member

@x13n x13n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Out of curiosity, did you test it only on the Go benchmarks or in an actual cluster? I wonder how much it improves in practice.

cluster-autoscaler/config/autoscaling_options.go Outdated Show resolved Hide resolved
cluster-autoscaler/main.go Outdated Show resolved Hide resolved
cluster-autoscaler/simulator/clustersnapshot/store/delta.go Outdated Show resolved Hide resolved
cluster-autoscaler/simulator/clustersnapshot/store/delta.go Show resolved Hide resolved
@macsko macsko force-pushed the parallelize_cluster_snapshot_creation branch 2 times, most recently from 60477f7 to b53877b Compare December 20, 2024 15:17
@macsko
Copy link
Member Author

macsko commented Dec 20, 2024

I wonder how much it improves in practice.

Yes, I tested it, but forgot to write. Before the change, in real cluster, for 1k nodes and 40k pods scale, a single SetClusterState took 300-500ms. With this improvement it takes 40-100ms.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 20, 2024
@macsko macsko force-pushed the parallelize_cluster_snapshot_creation branch from b53877b to 4de0f60 Compare December 23, 2024 09:20
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 23, 2024
@macsko macsko requested a review from x13n December 23, 2024 09:20
x13n
x13n reviewed Dec 31, 2024
View reviewed changes
Copy link
Member

@x13n x13n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good, just one last comment, maybe this could be simplified a bit?

cluster-autoscaler/simulator/clustersnapshot/store/delta.go
@@ -428,24 +455,69 @@ func (snapshot *DeltaSnapshotStore) AddSchedulerNodeInfo(nodeInfo *schedulerfram
return nil
}

// setClusterStatePodsSequential sets the pods in cluster state in a sequential way.
func (snapshot *DeltaSnapshotStore) setClusterStatePodsSequential(nodeInfos []*schedulerframework.NodeInfo, nodeNameToIdx map[string]int, scheduledPods []*apiv1.Pod) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need a separate sequential implementation? It behaves the same as ...Parallelized one with parallelism == 1, right?

Copy link
Member Author

@macsko macsko Dec 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to leave parallelism == 1 with previous implementation to make the change potentially revertible (see the discussion above). Sequential implementation could be eventually removed and use workqueue.ParallelizedUntil always, when we are sure the behavior is correct.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, it is just that code cleanups are easy to forget. Can you at least leave a TODO here with your github handle to follow up on this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@macsko macsko force-pushed the parallelize_cluster_snapshot_creation branch from 4de0f60 to 3988255 Compare January 3, 2025 10:35
@x13n
Copy link
Member

x13n commented Jan 3, 2025

Thanks!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 3, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: macsko, x13n

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 3, 2025
@k8s-ci-robot k8s-ci-robot merged commit e8a3df6 into kubernetes:master Jan 3, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@x13n x13n x13n requested changes

@BigDarkClown BigDarkClown Awaiting requested review from BigDarkClown

@vadasambar vadasambar Awaiting requested review from vadasambar

@aleksandra-malinowska aleksandra-malinowska Awaiting requested review from aleksandra-malinowska

@yaroslava-serdiuk yaroslava-serdiuk Awaiting requested review from yaroslava-serdiuk

Assignees

@x13n x13n

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/cluster-autoscaler cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@macsko @yaroslava-serdiuk @x13n @k8s-ci-robot