Allow live-migration for pod network in bridge mode #7768

kvaps · 2022-05-18T12:02:45Z

This PR enables the live-migration feaure for pod networking in any mode.

The feature protected by feature gate NetworkAwareLiveMigration

The functionality has been added:

If MAC address changed: detach / attach interface after live migration to have correct MAC address set
If MAC address is not changed: link down / link up interface to force VM request new IP address and routes from DHCP

Full proposal is described here kubevirt/community#182

fixes #7238

What this PR does / why we need it:

This feature would allow to live-migrate the VMs with pod network in bridge and macvtap modes.

We're at Deckhouse developing virtualization solution based on KubeVirt and Cilium.
We'd like to make live-migration working for less latency modes (macvtap and bridge)

Which issue(s) this PR fixes

Special notes for your reviewer:

Slack thread: https://kubernetes.slack.com/archives/C8ED7RKFE/p1652296916563589?thread_ts=1650371688.593359&cid=C8ED7RKFE

Release note:

New feature gate NetworkAwareLiveMigration which allows live-migration for pod network connected in bridge mode

kubevirt-bot · 2022-05-18T12:02:57Z

Hi @kvaps. Thanks for your PR.

I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/virt-launcher/virtwrap/manager.go

maiqueb

Some initial comments.

Let's check out your new e2e test
/test pull-kubevirt-e2e-k8s-1.21-sig-network

pkg/virt-handler/vm.go

pkg/virt-launcher/virtwrap/api/schema.go

maiqueb · 2022-05-18T14:52:40Z

Some more questions:

for which binding type would this apply to ? bridge ?...
did you consider IPAM (when bridge binding is used) in this scenario ? Like, inside the VMI, we will have the same IP address, but what about the pod interface ? Unless the CNI features some sort of sticky IP, once the lease expires, it will get a new IP - which might break the applications inside the VM. And this means that if the CNI features a sticky IP, you'll have 2 live pods with the same IP. This, iiuc, breaks the kubernetes networking model, where each pod can reach any other pod - i.e. which one would it reach ?...

kvaps · 2022-05-18T16:20:14Z

for which binding type would this apply to ? bridge ?...

I tested this with bridge and macvtap

did you consider IPAM (when bridge binding is used) in this scenario ? Like, inside the VMI, we will have the same IP address, but what about the pod interface ? Unless the CNI features some sort of sticky IP, once the lease expires, it will get a new IP - which might break the applications inside the VM. And this means that if the CNI features a sticky IP, you'll have 2 live pods with the same IP. This, iiuc, breaks the kubernetes networking model, where each pod can reach any other pod - i.e. which one would it reach ?...

Yeah, both scenarios are supported:

If target pod has the same IP, then only routes will be updated.
If target pod has different IP, then a new IP will be assigned.

Both are done by setting link down / link up to force DHCP-client renew a lease.

I tested this with cilium/cilium#19789

EdDev · 2022-05-28T05:20:30Z

@kvaps , this is a very interesting proposal.
The topic itself was raised multiple times and got stuck due to the corner cases which k8s brings in.

I think it will very useful to push this idea forward through a formal proposal.

I am interested to see the use cases where such an option fits better than the existing masquerade, a discussion on when and how to enable migrations in such setups and the technical details of how such a migration will work from the pod, vm and guest perspective.

At the last point regarding the tech details, I remember the challenge of having the mac address at the source and target pods active for the migration period.

Generally speaking, if this can fit a specific configuration (e.g. specific CNI, guests with/without dhcp, acceptable link-flickering on the vnic, etc), I see no reason why not to have it in as an option. The main risk and limitation is that future features/changes may break it (i.e. if activating a future enhancement will break this migration option, it may still be accepted in and you will not be able to use it).

I hope this can be discussed through a proposal.

EdDev · 2022-05-28T05:23:50Z

/sig network
/cc @AlonaKaplan
/cc @phoracek

maiqueb

Some code related nits, and requests for unit / e2e tests.

I agree with @EdDev - this would move forward faster if you presented a design proposal of this work.

tests/migration_test.go

pkg/virt-launcher/virtwrap/manager.go

pkg/virt-handler/vm.go

AlonaKaplan · 2022-05-31T07:36:36Z

Some code related nits, and requests for unit / e2e tests.

I agree with @EdDev - this would move forward faster if you presented a design proposal of this work.

@kvaps thanks for the PR. I agree with Miguel and Eddy that it would be much easier to continue the discussion over a proposal.
Previously @maiqueb suggested two different solutions to the same issue-

Make the DHCP server advertise very short leases, thus causing the client to re-acquire the lease.
Using FORCERENEW dhcp server option to force the client to move to the RENEW state.

It would be nice to explain why unlinking/linking the interface is better than one of those options.

Another question I would love to discuss over the proposal-

Why the migration is not allowed if the MAC address is bot persistent, but allowed in case the IP address is not persistent? What is the difference?

kvaps · 2022-06-16T13:04:14Z

Sorry for long answer

Make the DHCP server advertise very short leases, thus causing the client to re-acquire the lease.

This is not working in every distribution, IIRC some old ubuntu and debians setting IP lifetime for infinity despite the fact DHCP provided lease just from shot amount of time.

Using FORCERENEW dhcp server option to force the client to move to the RENEW state.

This is still RFC and is not supported on some distributions and MS Windows for example

The link down/up method works perfectly in both cases

Why the migration is not allowed if the MAC address is bot persistent, but allowed in case the IP address is not persistent? What is the difference?

Well, we can force renewing IP configuration for the VM (this is not only about the IP address, but also routes).
But there is no easy way to force renewing MAC address inside the VM.

However I was also thinking about this, and I think we have two options:

We could use detach/attach interface
We could generate cloud-init config with mac address, and push it into VM somehow. But in this way it will also lead to link down/up inside the VM

I think detaching link for working virtual machine can be destructive action if application is binded on specific interface, not just 0.0.0.0, so we can provide this opportunity only by enabling specific feature flag

kubevirt-bot · 2023-03-01T14:32:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dhiller for approval by writing /assign @dhiller in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kvaps · 2023-03-01T14:34:07Z

PR rebased on v0.59

old versions:

v0.57 - 562c8bc
v0.58 - 1a161d3

kubevirt-bot · 2023-03-01T19:41:25Z

@kvaps: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kubevirt-bot · 2023-06-13T10:27:38Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

rthallisey · 2023-06-13T14:42:59Z

/remove-lifecycle stale

kvaps · 2023-07-26T11:52:37Z

I migrated my fork from kvaps/kubevirt to deckhouse/3p-kubevirt, now it will be supported by deckhouse project
fyi

kubevirt-bot · 2023-10-24T12:21:21Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

remram44 · 2023-10-24T12:29:23Z

Still wanted

rthallisey · 2023-10-24T12:49:56Z

/remove-lifecycle stale

xpivarc

Never mind I see the problem

kubevirt-bot · 2024-01-22T15:07:00Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kubevirt-bot · 2024-02-21T15:36:52Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

remram44 · 2024-02-21T17:55:20Z

I am interested in this feature

/remove-lifecycle stale

kubevirt-bot · 2024-03-22T18:09:14Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

kubevirt-bot · 2024-03-22T18:09:17Z

@kubevirt-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-bot requested review from EdDev and kbidarkar May 18, 2022 12:03

kvaps mentioned this pull request May 18, 2022

[virtualization] Workflow - phase 2 deckhouse/deckhouse#602

Open

89 tasks

vasiliy-ul reviewed May 18, 2022

View reviewed changes

pkg/virt-launcher/virtwrap/manager.go Show resolved Hide resolved

maiqueb reviewed May 18, 2022

View reviewed changes

pkg/virt-handler/vm.go Outdated Show resolved Hide resolved

pkg/virt-launcher/virtwrap/api/schema.go Show resolved Hide resolved

kvaps mentioned this pull request May 18, 2022

Add kvaps to org members kubevirt/project-infra#2113

Merged

7 tasks

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 28, 2022

kubevirt-bot requested review from AlonaKaplan and phoracek May 28, 2022 05:23

kubevirt-bot added the sig/network label May 28, 2022

maiqueb reviewed May 30, 2022

View reviewed changes

kubevirt-bot added dco-signoff: no Indicates the PR's author has not DCO signed all their commits. and removed dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jun 17, 2022

kvaps marked this pull request as draft June 17, 2022 15:30

kubevirt-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 17, 2022

kvaps force-pushed the allow-live-migration branch 4 times, most recently from 723e19e to cdbd9d9 Compare June 18, 2022 08:23

kvaps changed the base branch from release-0.58 to release-0.59 March 1, 2023 14:31

kubevirt-bot added size/XXL and removed size/XL labels Mar 1, 2023

kvaps force-pushed the allow-live-migration branch from 1a161d3 to d8fdb00 Compare March 1, 2023 14:31

kubevirt-bot added size/XL and removed size/XXL labels Mar 1, 2023

kvaps mentioned this pull request Mar 15, 2023

cannot migrate VMI which does not use masquerade to connect to the pod network #7238

Closed

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 13, 2023

kubevirt-bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 13, 2023

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 24, 2023

kubevirt-bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 24, 2023

xpivarc reviewed Oct 24, 2023

View reviewed changes

kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 22, 2024

kubevirt-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 21, 2024

kubevirt-bot closed this Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow live-migration for pod network in bridge mode #7768

Allow live-migration for pod network in bridge mode #7768

kvaps commented May 18, 2022 •

edited

Loading

kubevirt-bot commented May 18, 2022

maiqueb left a comment

maiqueb commented May 18, 2022

kvaps commented May 18, 2022 •

edited

Loading

EdDev commented May 28, 2022

EdDev commented May 28, 2022

maiqueb left a comment

AlonaKaplan commented May 31, 2022

kvaps commented Jun 16, 2022 •

edited

Loading

kubevirt-bot commented Mar 1, 2023

kvaps commented Mar 1, 2023

kubevirt-bot commented Mar 1, 2023

kubevirt-bot commented Jun 13, 2023

rthallisey commented Jun 13, 2023

kvaps commented Jul 26, 2023 •

edited

Loading

kubevirt-bot commented Oct 24, 2023

remram44 commented Oct 24, 2023

rthallisey commented Oct 24, 2023

xpivarc left a comment •

edited

Loading

kubevirt-bot commented Jan 22, 2024

kubevirt-bot commented Feb 21, 2024

remram44 commented Feb 21, 2024

kubevirt-bot commented Mar 22, 2024

kubevirt-bot commented Mar 22, 2024

Allow live-migration for pod network in bridge mode #7768

Allow live-migration for pod network in bridge mode #7768

Conversation

kvaps commented May 18, 2022 • edited Loading

kubevirt-bot commented May 18, 2022

maiqueb left a comment

Choose a reason for hiding this comment

maiqueb commented May 18, 2022

kvaps commented May 18, 2022 • edited Loading

EdDev commented May 28, 2022

EdDev commented May 28, 2022

maiqueb left a comment

Choose a reason for hiding this comment

AlonaKaplan commented May 31, 2022

kvaps commented Jun 16, 2022 • edited Loading

kubevirt-bot commented Mar 1, 2023

kvaps commented Mar 1, 2023

kubevirt-bot commented Mar 1, 2023

kubevirt-bot commented Jun 13, 2023

rthallisey commented Jun 13, 2023

kvaps commented Jul 26, 2023 • edited Loading

kubevirt-bot commented Oct 24, 2023

remram44 commented Oct 24, 2023

rthallisey commented Oct 24, 2023

xpivarc left a comment • edited Loading

Choose a reason for hiding this comment

kubevirt-bot commented Jan 22, 2024

kubevirt-bot commented Feb 21, 2024

remram44 commented Feb 21, 2024

kubevirt-bot commented Mar 22, 2024

kubevirt-bot commented Mar 22, 2024

kvaps commented May 18, 2022 •

edited

Loading

kvaps commented May 18, 2022 •

edited

Loading

kvaps commented Jun 16, 2022 •

edited

Loading

kvaps commented Jul 26, 2023 •

edited

Loading

xpivarc left a comment •

edited

Loading