Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial rpmbuld failure - test_disable_ip_forwarding #2717

Closed
phillxnet opened this issue Oct 19, 2023 · 5 comments · Fixed by #2718
Closed

Initial rpmbuld failure - test_disable_ip_forwarding #2717

phillxnet opened this issue Oct 19, 2023 · 5 comments · Fixed by #2718
Assignees

Comments

@phillxnet
Copy link
Member

phillxnet commented Oct 19, 2023

During the initial review of #2712 (comment #2712 (review)) there was an as yet unexplained fail in a new test regarding ip forwarding:

======================================================================
FAIL: test_disable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
test proper call of os.remove() and final sysctl command
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/worker/Poetry-Build-on-Leap15-4/rpmbuild/rockstor-core-5.0.5-0/src/rockstor/system/tests/test_system_network.py", line 795, in test_disable_ip_forwarding
    self.mock_os_remove.assert_called_once_with("/etc/sysctl.d/99-tailscale.conf")
  File "/usr/lib64/python3.9/unittest/mock.py", line 918, in assert_called_once_with
    raise AssertionError(msg)
AssertionError: Expected 'remove' to be called once. Called 0 times.
----------------------------------------------------------------------
Ran 279 tests in 32.169s
FAILED (failures=1)

On further fresh rpmbuild attempts (other OS/arch targets) this has been seen again on all systems that have not previously done an rpmbuild with this test included. We likely have some inadvertent system sensitivity/interaction here.

@FroggyFlox
Copy link
Member

Apologies for failing with these tests... It seems I missed some mocking somewhere. Here's what I noticed:

I ran the test once first and it all passed, even after stopping, uninstall tailscale and its repo (and after a reboot):

buildvm155:/opt/rockstor/src/rockstor # poetry run django-admin test -v 2 -p test_system_network.py
Skipping setup of unused database(s): default, smart_manager.
System check identified no issues (0 silenced).
test_disable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
test proper call of os.remove() and final sysctl command ... ok
test_enable_ip_forwarding_syctl (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() triggers sysctl config reload ... ok
test_enable_ip_forwarding_write (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() calls write with the correct contents ... ok
test_get_con_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli connection config by get_con_config(), ... ok
test_get_con_config_con_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if connection is not found / vanished. ... ok
test_get_con_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if nmcli returns error code != 10. ... ok
test_get_dev_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli device config by get_dev_config(), ... ok
test_get_dev_config_dev_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if device is not found / vanished. ... ok
test_get_dev_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if nmcli returns error code != 10. ... ok

----------------------------------------------------------------------
Ran 9 tests in 0.014s

OK

I noticed, however, that I had the file /etc/sysctl.d/99-tailscale.conf physically present on the system, hence why it worked. Indeed, that failing test_disable_ip_forwarding() test checks if that file exists and tries to verify os.remove() was called on it.

buildvm155:/opt/rockstor/src/rockstor # ls -lah /etc/sysctl.d/
total 8.0K
drwxr-xr-x 1 root root   58 Oct 17 18:51 .
drwxr-xr-x 1 root root 3.6K Oct 19 15:11 ..
-rw-r--r-- 1 root root   35 Aug 11 10:49 70-yast.conf
-rw-r--r-- 1 root root   57 Oct 19 15:13 99-tailscale.conf

Now, if I manually delete that file and run the tests again, I get the same failure you have:

buildvm155:/opt/rockstor/src/rockstor # ls -lah /etc/sysctl.d/
total 4.0K
drwxr-xr-x 1 root root   24 Oct 19 15:15 .
drwxr-xr-x 1 root root 3.6K Oct 19 15:11 ..
-rw-r--r-- 1 root root   35 Aug 11 10:49 70-yast.conf


buildvm155:/opt/rockstor/src/rockstor # poetry run django-admin test -v 2 -p test_system_network.py
Skipping setup of unused database(s): default, smart_manager.
System check identified no issues (0 silenced).
test_disable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
test proper call of os.remove() and final sysctl command ... FAIL
test_enable_ip_forwarding_syctl (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() triggers sysctl config reload ... ok
test_enable_ip_forwarding_write (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() calls write with the correct contents ... ok
test_get_con_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli connection config by get_con_config(), ... ok
test_get_con_config_con_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if connection is not found / vanished. ... ok
test_get_con_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if nmcli returns error code != 10. ... ok
test_get_dev_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli device config by get_dev_config(), ... ok
test_get_dev_config_dev_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if device is not found / vanished. ... ok
test_get_dev_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if nmcli returns error code != 10. ... ok

======================================================================
FAIL: test_disable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
test proper call of os.remove() and final sysctl command
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/system/tests/test_system_network.py", line 795, in test_disable_ip_forwarding
    self.mock_os_remove.assert_called_once_with("/etc/sysctl.d/99-tailscale.conf")
  File "/usr/lib64/python3.9/unittest/mock.py", line 918, in assert_called_once_with
    raise AssertionError(msg)
AssertionError: Expected 'remove' to be called once. Called 0 times.

----------------------------------------------------------------------
Ran 9 tests in 0.014s

FAILED (failures=1)

The devil is in the details here: the order in which the tests are run: test_disable_ip_forwarding() is run before test_enable_ip_forwarding_write() and test_enable_ip_forwarding_sysctl(). The latter two indeed are testing if that 99-tailscale.conf file is written and then sysctl called to refresh its config from that file. It seems I failed to properly mock that writing as the file is actually written to disk after this round of tests with disable_ip_forwarding() that failed:

buildvm155:/opt/rockstor/src/rockstor # ls -lah /etc/sysctl.d/
total 8.0K
drwxr-xr-x 1 root root   58 Oct 19 15:15 .
drwxr-xr-x 1 root root 3.6K Oct 19 15:11 ..
-rw-r--r-- 1 root root   35 Aug 11 10:49 70-yast.conf
-rw-r--r-- 1 root root   57 Oct 19 15:15 99-tailscale.conf

If we run again these tests, they all pass now given the file is present:

buildvm155:/opt/rockstor/src/rockstor # poetry run django-admin test -v 2 -p test_system_network.py
Skipping setup of unused database(s): default, smart_manager.
System check identified no issues (0 silenced).
test_disable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
test proper call of os.remove() and final sysctl command ... ok
test_enable_ip_forwarding_syctl (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() triggers sysctl config reload ... ok
test_enable_ip_forwarding_write (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() calls write with the correct contents ... ok
test_get_con_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli connection config by get_con_config(), ... ok
test_get_con_config_con_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if connection is not found / vanished. ... ok
test_get_con_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if nmcli returns error code != 10. ... ok
test_get_dev_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli device config by get_dev_config(), ... ok
test_get_dev_config_dev_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if device is not found / vanished. ... ok
test_get_dev_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if nmcli returns error code != 10. ... ok

----------------------------------------------------------------------
Ran 9 tests in 0.013s

OK

@phillxnet
Copy link
Member Author

@FroggyFlox Thanks for clearing that one up. And my apologies for missing this initially in my review. Bit by bit.

@FroggyFlox
Copy link
Member

To prevent actually writing 99-tailscale.conf to disk, we can combine the previously-split test_enable_ip_forwarding_write() and test_enable_ip_forwarding_sysctl() tests into one:

    def test_enable_ip_forwarding(self):
        """enable_ip_forwarding() calls write with the correct contents
        Enable_ip_forwarding should write to /etc/sysctl.d/99-tailscale.conf
        with the following lines
          - net.ipv4.ip_forward = 1
          - net.ipv6.conf.all.forwarding = 1
        """
        priority = 99
        name = "tailscale"
        file_path = f"{SYSCTL_CONFD_PATH}{priority}-{name}.conf"
        m = mock_open()
        with patch("system.network.open", m):
            enable_ip_forwarding(name=name, priority=priority)

            # Test that the file was opened in 'w' mode
            m.assert_called_once_with(file_path, "w")

            # Test that 'write' was called with the correct content
            calls = [
                call("net.ipv4.ip_forward = 1\n"),
                call("net.ipv6.conf.all.forwarding = 1\n"),
            ]
            m().write.assert_has_calls(calls, any_order=False)

        # Test that run_command was called as expected
        self.mock_run_command.assert_called_once_with(
            [SYSCTL, "-p", file_path], log=True
        )

With regards to testing disable_ip_forwarding(), however, things are a bit more complicated due to the nature of the function itself. It indeed uses os.scandir() and that one is complex to mock... I'll keep looking into how to do that but I'm not that optimistic. We may end up having to slightly refactor that function as there are multiple ways to do what it accomplishes.

@FroggyFlox
Copy link
Member

Of course, it is right after I write this that I finally understood how it works. Following some sparse online answers on how to do that, it appears we can manually re-create the DirEntry object that the real os.scandir() would return. For instance, create the following class:

class PseudoDirEntry:
    def __init__(self, name, path, is_dir, stat):
        self.name = name
        self.path = path
        self._is_dir = is_dir
        self._stat = stat

    def is_dir(self):
        return self._is_dir

    def stat(self):
        return self._stat

... and use it as a return value for our os.scandir mock (modified to account for the fact that it is within a context manager):

    def test_disable_ip_forwarding(self):
        """test proper call of os.remove() and final sysctl command"""
        # mock os.scandir()
        self.patch_os_scandir = patch("system.network.os.scandir")
        self.mock_os_scandir = self.patch_os_scandir.start()
        self.mock_os_scandir.return_value.__enter__.return_value = [
            MockDirEntry(name="99-tailscale.conf", path="/etc/sysctl.d/99-tailscale.conf", is_file=True),
            MockDirEntry(name="70-yast.conf", path="/etc/sysctl.d/70-yast.conf", is_file=True),
        ]

        # mock os.remove()
        self.patch_os_remove = patch("system.network.os.remove")
        self.mock_os_remove = self.patch_os_remove.start()
        # mock run_command()
        self.mock_run_command.return_value = [""], [""], 0
        disable_ip_forwarding(name="tailscale")
        # test os.remove() was called
        self.mock_os_remove.assert_called_once_with("/etc/sysctl.d/99-tailscale.conf")

With this, all tests pass even without 99-tailscale.conf being physically present on the disk:

buildvm155:/opt/rockstor/src/rockstor # ls -lah /etc/sysctl.d/
total 4.0K
drwxr-xr-x 1 root root   24 Oct 19 15:25 .
drwxr-xr-x 1 root root 3.6K Oct 19 15:11 ..
-rw-r--r-- 1 root root   35 Aug 11 10:49 70-yast.conf


buildvm155:/opt/rockstor/src/rockstor # poetry run django-admin test -v 2 -p test_system_network.py
Skipping setup of unused database(s): default, smart_manager.
System check identified no issues (0 silenced).
test_disable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
test proper call of os.remove() and final sysctl command ... ok
test_enable_ip_forwarding (rockstor.system.tests.test_system_network.SystemNetworkTests)
enable_ip_forwarding() calls write with the correct contents ... ok
test_get_con_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli connection config by get_con_config(), ... ok
test_get_con_config_con_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if connection is not found / vanished. ... ok
test_get_con_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_con_config() if nmcli returns error code != 10. ... ok
test_get_dev_config (rockstor.system.tests.test_system_network.SystemNetworkTests)
This tests for correct parsing of nmcli device config by get_dev_config(), ... ok
test_get_dev_config_dev_not_found (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if device is not found / vanished. ... ok
test_get_dev_config_exception (rockstor.system.tests.test_system_network.SystemNetworkTests)
Test get_dev_config() if nmcli returns error code != 10. ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.014s

OK

FroggyFlox added a commit to FroggyFlox/rockstor-core that referenced this issue Oct 19, 2023
We had two mocking deficiencies:
 - one that resulted in one test actually writing a file to disk
 - one failing to properly mock os.scandir (we were thus depending on
   the real system state)

This commit fixes the former by combining 2 previously split tests,
thereby sharing the same mock (we're not writing to disk anymore).
To fix the latter, we instantiate a new class used to mock os.scandir
return values that we can now properly control.
FroggyFlox added a commit to FroggyFlox/rockstor-core that referenced this issue Oct 19, 2023
We had two mocking deficiencies:
 - one that resulted in one test actually writing a file to disk
 - one failing to properly mock os.scandir (we were thus depending on
   the real system state)

This commit fixes the former by combining 2 previously split tests,
thereby sharing the same mock (we're not writing to disk anymore).
To fix the latter, we instantiate a new class used to mock os.scandir
return values that we can now properly control.
@FroggyFlox FroggyFlox linked a pull request Oct 19, 2023 that will close this issue
@FroggyFlox FroggyFlox self-assigned this Oct 20, 2023
phillxnet added a commit that referenced this issue Oct 20, 2023
…e---test_disable_ip_forwarding

Fix mocking insufficiencies in system.network.py #2717
@FroggyFlox
Copy link
Member

Closing as fixed by #2718

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants