DHCP leases file not getting updated #18
Comments
|
any idea @AkihiroSuda what might cause this? we see this error as a Flake on our mac Os sometimes, but on Github Action it happens all the time. so I think finding the root cause would help a lot of users |
|
Could you make sure that you are NOT using the bridged mode?
Bridged mode is known not to work on Github Action |
|
I'm just running Which according to the comments in the Makefile |
|
@AkihiroSuda but here is the last part of the log |
|
When an instance is terminated it logs the following: Not sure if that's expected or not. |
|
@AkihiroSuda Is there anything I could do (debug commands, etc.) to help figure out this issue? I'm still experiencing this issue. |
|
Cc @jandubois could you take a look? |
|
I just uninstalled, rebooted, and installed with master and it works now, not sure if recent commits fixed this |
|
@spowelljr I assume you built from source? I ask because there is a brew formula for |
|
Correct, from source |
|
Thanks. Would be good to know if it still works after surviving a reboot. Regardless, I'll perform the exact steps you did and update everyone. |
|
TLDR; DHCP address allocation works when not in bridged mode and directly building When compiling for bridged mode with... sudo make PREFIX=/opt/socket_vmnet install BRIDGED=en0...and then starting minikube with.... minikube start \
--network socket_vmnet \
--socket-vmnet-client-path="/opt/socket_vmnet/bin/socket_vmnet_client" \
--socket-vmnet-path="/var/run/socket_vmnet.bridged.en0"...results in the following failure: 😿 Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 76:b3:20:a2:9b:74
❌ Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 76:b3:20:a2:9b:74Of course What's interesting is I can confirm a new DHCP address is being allocated since my firewall notifies me of new device activity. However, I don't see any new addresses in I've also followed this "Known Issues" workaround to no avail. UPDATE: As another interesting data point, I am able to start Lima using the same network:
- lima: bridged |
That is expected, as bridged IP addresses are assigned by the DHCP on your local network, and are not local to your host. That is the whole point of using a bridged network: to get an externally routable IP address, so other machines can connect to your VM. Maybe this helps:
|
|
@jandubois Yes. That helps quite a bit. Thanks! |
|
@jandubois I've attached the logs.txt from my latest attempt with bridged mode. Given the logs are Minikube, if folks prefer, I can move this over to the linked Minikube issue 15300. |
|
@mprimeaux I'm not sure what you are trying to achieve. You will not be able to get the IP address of the bridged network from the host machine. You will have to run e.g. It looks like minikube only supports looking in the local leases log, so it does not support bridged networks at the moment. Somebody will have to add it. If you don't have a slirp interface to use in addition to the bridged network, then you will have to add something to the VM setup to log the address in a place that can be inspected externally. I tried to see if there is a way to determine the address via There is probably a way to force the VM to announce itself, e.g. by incorrectly broadcasting that the MAC address belongs to the host, but then giving up when the VM replies "well, actually, that is my MAC address". But that feels rather hacky (and I haven't tried if it actually works, as the bridged interface technically does belong to the host). In all, querying the IP address via a local slirp connection seems to be the way to do this. |
I think this is just missing functionality in minikube and don't see what could be done in the |
|
@jandubois In short, what I'm trying to accomplish is to have the Kubernetes control plane (minikube) obtain a routable address on my host network. |
|
I still have this problem, I confirm that uninstalling and rebooting and re-installing from source fixes it (but after a few days it goes bad again) so the root of the problem still exists on the code on HEAD. @AkihiroSuda is there any reason a restart would fix this ? is there a process other than "/opt/socket_vmnet/bin/socket_vmnet" that we could check for, that maybe is not killed or stoppped after installation ? that only a restart kills it ? or is there a "lock" on a file or something like that ? I just need to find out what exactly "restart" does that uninstalling/reinstalling without restart wont do the trick.... that way we could tell our users or make minikube just do that when it happens automatically |
|
@medyagh It's coincidental that you mention this as this happened to me just today. I ended up rebooting to fix. Could this be due to network going to sleep or an other power option in macOS? Just a hypothesis. |
|
My assumption is that this is an issue of Apple's bootpd (https://github.com/apple-oss-distributions/bootp) or com.apple.NetworkSharing , but not 100% sure. If |
|
thanks for giving somethings to try out, they dont seem to fix it but i apperciate that you try here are the logs just in case: |
|
@AkihiroSuda I wonder if could this be related to Crop VPNs ? |
|
It seems like this problem comes back after a few days of working without any issue. At first, rebooting and re-installing from the source was working but now it's not. |
@onatm I am curious are you on a Corp Network or behind a VPN ? |
I'm not sure what "the issue" is in this discussion anymore. As I explained above, the DHCP lease information will never be found on the local machine because it comes from the local network and not the builtin DHCP. You have to determine that IP address from inside the VM, e.g. by also having a shared or host-only interface, whose IP address you can get from the local leases file. I don't think there is anything left to discuss about that. If this discussion is about socket_vmnet hanging, then I think there should be a separate Github issue for that, and this issue should be closed. |
|
@jandubois I agree that a different issue should be opened. Sorry for the tangent. Speaking for my desire (to have Minikube + QEMU driver work in bridged mode), a minikube enhancement request might be the best course of action. |
@jandubois are you saying even at the times that socket_vmnet with minikube works well with minikube, even at those times the DHCP were not found in minikube either ? I dont think thats has been the case in my experience. when it works, minikube finds the lease on before the 6th try ... but when it doesnt work it wont find it and times out |
|
I'm sorry, I was only thinking about But I just realized that macOS also stores client leases under |
This is the traditional (i.e. VirtualBox) approach, later copied to other drivers such as libvirt |
|
Question here and maybe this is not related, but based on tests I have been running I think this might be - and indicating a change or issue with See - lima-vm/lima#1259 The issue I have seen is that on Mac OS 13.3 M1 is that when I try to use Colima / Lima / Minikube with sock_vmnet and qemu is that the DHCP address is not being allocated. If I stop Now with Minikube it seem to look for specific format of the dhcpd_leases and it must be Basic setup of the DHCP: Example output I see which shows while minikube is trying to start that it is getting to the DHCP server: /opt/homebrew/etc/dhcpd-minikube.conf contents |
|
This might work?
Originally posted by @ AravindGopala in lima-vm/lima#1259 (comment) |
I tried on my personal machine and GitHub Actions and both returned |
|
I also can confirm the same experience as @spowelljr on my macOS ARM machines. ❯ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
The application is not part of the firewallSame message whether my firewall is enabled or not. |
|
Did you add it to the firewall first? $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Password:
The application is not part of the firewall
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
Application at path ( /usr/libexec/bootpd ) added to firewall
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Incoming connection to the application is permitted |
|
Thanks, @jandubois. I did not but do it now and test. |
|
@jandubois That worked for my personal machine, trying on GitHub Actions now, thanks. |
|
@jandubois This worked for my machines, also. Thanks! |
Thank you so much for finding this out ! this helps a lot of us, btw this does not on github action, do you have any clue/guess how github action MacOs could be different ? |
|
This issue is solved on physical macbooks, but still persists on github action macos I suggest we rename this issue, and it might be helpful to investigate with Github Action Environments to see if they know if any special quirks of gh action envs ? |
|
Unfortunately, even with sudo, we cannot run the above workaround commands on Macs enrolled in Mobile Device Management (MDM) I imagine this would be a common occurrence for developers working for a medium-large company and where difficult to persuade their company's IT department to allow firewall exceptions, though based on my limited knowledges seems doable - https://developer.apple.com/documentation/devicemanagement/firewall/applicationsitem Is it possible this is actually a bug on Apple's end and if so has anyone reported it https://developer.apple.com/bug-reporting/? We only see this bug only after rebooting where socket_vmnet was enabled on some previous boot via From https://developer.apple.com/documentation/vmnet#1681413
I have verified using wireshark that QEMU VM starts/boots and sends through a Presumably |
@tmoschou - unfortunately I think the firewall MDM profile attributes only allows specifying apps via a BundleID and I don't think Microsoft InTune docs for creating macOS details a way to get the BundleID of an app but it returns an empty value for ~ osascript -e 'id of app "/usr/libexec/bootpd"'
Compared to other things: ~ osascript -e 'id of app "Notes"'
com.apple.Notes
~ osascript -e 'id of app "/usr/libexec/not_a_real_thing"'
0:2: execution error: Can’t get application "/usr/libexec/not_a_real_thing". (-1728)
|
Ah maybe |
|
in my case I do not have I assume there's no need to enlist it, it would be great if someone could confirm this |
@tmoschou This works on my managed mac, it probably depends on your organization. If the issue is with bootpd blocked by the firewall, it may work in bridged mode since it does not depend on bootpd. Please open a new issue for this, this old issue was resolved long time ago by #18 (comment). |
@medyagh lets open a new issue for this use case? |
|
@jandubois No replies so far for comments, I think it is safe to close this. |
We've been getting issues on our personal machines where our DHCP leases file is not being updated after starting an instance with socket_vmnet. The fix so far has been uninstall socket_vmnet, reboot machine, reinstall, which resolves the issue. However I'm trying to integrate GitHub Action tests for QEMU/socket_vmnet and I'm getting the same thing (logs below).
Is there a command we could run that would resolve this issue? I see in the README there's a command to
Reload the DHCP daemonI'm not sure if that should resolve the issue, however trying to run it in GitHub Action resulted in
Could not find service "com.apple.bootpd" in domain for systemBelow are to logs from the GitHub Action machine with the main error being
StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directoryWe experienced the same thing on our personal machines, we tried manually creating the file just incase the there were some permission errors but that didn't resolve the issue, plus I'm sure it's more than capable of creating the file itself.
Any help would be appreciated, thanks!
The text was updated successfully, but these errors were encountered: