Test failure with "shutil.rmtree()" in isolated build environments (1.5.3.rc1 in Fedora 41)

[hannes101]

I was trying to build backintime for fedora 41. I am getting the following test failures. I could of course drop these tests, but I thought it might be worth reporting during this time of the development cycle.

build.log

The build can be found here.
https://koji.fedoraproject.org/koji/taskinfo?taskID=125095920

After I removed the the above tests, it fails at a later one:
https://kojipkgs.fedoraproject.org//work/tasks/6177/125096177/build.log
Uploading build_2.log…

[buhtz]

Hello Johannes,

thank you very much for trying that RC and reporting the problem.

Problem 1 shutil.rmtree()

Mhm.... Here is an example of one failed test:

__________________________ General.test_ctor_defaults __________________________
self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
    def test_ctor_defaults(self):
        """Default values in constructor."""
>       with TemporaryDirectory(prefix='bit.') as temp_name:
test/test_uniquenessset.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.13/tempfile.py:946: in __exit__
    self.cleanup()
/usr/lib64/python3.13/tempfile.py:950: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
/usr/lib64/python3.13/tempfile.py:930: in _rmtree
    _shutil.rmtree(name, onexc=onexc)
/usr/lib64/python3.13/shutil.py:763: in rmtree
    _rmtree_safe_fd(stack, onexc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
stack = []
onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at 0xffffb39bc860>
    def _rmtree_safe_fd(stack, onexc):
        # Each stack item has four elements:
        # * func: The first operation to perform: os.lstat, os.close or os.rmdir.
        #   Walking a directory starts with an os.lstat() to detect symlinks; in
        #   this case, func is updated before subsequent operations and passed to
        #   onexc() if an error occurs.
        # * dirfd: Open file descriptor, or None if we're processing the top-level
        #   directory given to rmtree() and the user didn't supply dir_fd.
        # * path: Path of file to operate upon. This is passed to onexc() if an
        #   error occurs.
        # * orig_entry: os.DirEntry, or None if we're processing the top-level
        #   directory given to rmtree(). We used the cached stat() of the entry to
        #   save a call to os.lstat() when walking subdirectories.
        func, dirfd, path, orig_entry = stack.pop()
        name = path if orig_entry is None else orig_entry.name
        try:
            if func is os.close:
                os.close(dirfd)
                return
            if func is os.rmdir:
                os.rmdir(name, dir_fd=dirfd)
                return
    
            # Note: To guard against symlink races, we use the standard
            # lstat()/open()/fstat() trick.
>           assert func is os.lstat
E           AssertionError
/usr/lib64/python3.13/shutil.py:663: AssertionError

It looks like that there is a problem when cleaning up the temporary directory. shutil.rmtree() raise the error. But I don't understand the details.

Might it be the case that this is related to some "special attributes" of your build environment (chroot magic or something like that stuff I don't understand)?

Can you run the test in a regular fedora setup?

EDIT: I asked at python-llist and at the mock upstream project about the assert-error from shutil.rmtree().

Problem 2: User not in second usergroup

Moved to #1912

[hannes101]

Yes, you are right that it's a mock environment. I am not really sure how to properly debug something in a mock environment.
But checking the root.log of a build it seems that the user mockbuild gets created and is only added to one group.
root.log

DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['userdel', '-f', 'mockbuild', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:608: Child return code was: 0
DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['groupdel', 'mock', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:608: Child return code was: 0
DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['groupadd', 'mock', '-g', '135', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:608: Child return code was: 0
DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['useradd', 'mockbuild', '-o', '-u', '1001', '-g', '135', '-N', '-d', '/builddir', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:459: useradd: warning: the home directory /builddir already exists.
DEBUG util.py:459: useradd: Not copying any file from skel directory into it.
DEBUG util.py:459: Creating mailbox file: File exists
DEBUG util.py:608: Child return code was: 0

[buhtz]

About Problem 1: I asked at python-list and at the mock upstream project about the problem.

About Problem 2: Are you experienced enough with mock to configure it that way that you can add the user to a second usergroup? Do you think this would be an appropriate solution?

[xsuchy]

During the build in Mock the build is started as:

<mock-chroot> sh-5.2# id
uid=0(root) gid=0(root) groups=0(root)

and then dropped privs to:

<mock-chroot> sh-5.2$ id
uid=1000(mockbuild) gid=135(mock) groups=135(mock)

The environment is very minimalistic on purpose. I recommend to skip the test if the len(GROUPS) < 2.

[buhtz]

Problem about usergroup separated into its own issue #1912.

[buhtz]

FYI: The shutil problem might also be related to a specific Python version. It is reported with Python 3.13. I was not able to find the code location in the shutil.py file of Python 3.11.2.

This is the commit where this assert line was introduce in Python.

python/cpython@53b1981

The related PR

python/cpython#119808

Not sure if it is appropriate to contact the committing person about the problem? Maybe he can give us a hint what is going on?

EDIT: Python 3.13 on TravisCI (Ubuntu Jammy) pass all BIT tests.

[buhtz]

The folks on the python-list started the discussion.

It would help them with diagnosis if you could edit the file /usr/lib64/python3.13/shutil.py on your system.
In there before this line

assert func is os.lstat

Add a debug print like this.

print(f'{os.lstat=}')

And let us know the output please.

Thanks in advance,

[hannes101]

This is the outcome:
build_local_pychanged.log

[buhtz]

Thank you very much for your output. I realized that the relevant test is using PyFakeFS, to create and use a surrogate file-system in the RAM.

Might you be able doing some more editing work? Otherwise I also could provide you with a PR, if this is easier for you. Let me know. Just for diagnosis I would like to eliminate the PyFakeFS.

In common/test/test_uniquenessset.py you will find this lines:

logger.DEBUG = True


class General(pyfakefs_ut.TestCase):
    # TODO: add test for follow_symlink
    def setUp(self):
        """Setup a fake filesystem."""
        self.setUpPyfakefs(allow_root_user=False)

    def _create_unique_file_pairs(self, pairs):

Please modify the parent class and remove the "setUp()" method.

logger.DEBUG = True
import unittest


class General(unittest.TestCase):
    def _create_unique_file_pairs(self, pairs):

[buhtz]

FYI: Asked pyfakefs upstream for assistance and ideas.

[hannes101]

Thanks a lot for your efforts, sorry I was working. The removal of the setUp function worked and the local build was successful.
build_local_withoutpyfakefs.log

[buhtz]

Thank you for testing. Based on the feedback from the upstream maintainer of pyfakefs and my own tests on TravisCI it seems to me that the reason for this issue is that the version of pyfakefs you are using, in combination with Python 3.13, is outdated.

pytest-dev/pyfakefs#1084 (comment)

PyFakeFS does add support for python 3.13 in version 5.6.
Based on my own tests on TravisCI I can confirm that the tests fails with PyFakeFS 5.2.4 in Python 3.13 and 3.12 but not Python 3.11 or older.

Fedora 41 PyFakeFS is at version 5.2.4.
But Python in Fedora 41 is at 3.13.

Reading Wikipedia tells me that Fedora 41 is not yet released. Am I right? So this looks like a RC bug.

By the way:
This can also be interpreted as a "bug" in BIT because BIT does not force a specific version of its dependencies. We do have #1575 open for this issue.

[hannes101]

I'll get that updated then in fedora, thanks. I opened a pull-request and I guess we can close this now, since it's not an upstream issue.