sanity check fails with dnf5

[nirik]

Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]: Traceback (most recentcall last):                                                                                     
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib/python3.13/site-packages/bodhi/server/tasks/composer.py", line 411, in work                           
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     self._compose_updates()                                                                                            
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     ~~~~~~~~~~~~~~~~~~~
~~^^                                                                                            
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib/python3.13/site-packages/bodhi/server/tasks/composer.py", line 967, in _compose_updates               
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     self._sanity_check_repo()                                                                                          
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     ~~~~~~~~~~~~~~~~~~~
~~~~^^                                                                                          
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib/python3.13/site-packages/bodhi/server/tasks/composer.py", line 1167, in _sanity_check_repo            
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     sanity_check_repodata(repodata, repo_type='source', drpms=False)                                                   
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     ~~~~~~~~~~~~~~~~~~~
~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib/python3.13/site-packages/bodhi/server/util.py", line 373, in sanity_check_repodata                    
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     output = sanity_check_repodata_dnf(testdir, myurl, *dnfargs)                                                       
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib/python3.13/site-packages/bodhi/server/util.py", line 406, in sanity_check_repodata_dnf                
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     return subprocess.check_output(cmd, encoding='utf-8', stderr=subprocess.STDOUT)                                    
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:            ~~~~~~~~~~~~
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                    
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib64/python3.13/subprocess.py", line 472, in check_output                                                
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,                                                   
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:            ~~~^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:                **kwargs
).stdout                                                                                        
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:                ^^^^^^^^
^                                                                                               
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:   File "/usr/lib64/python3.13/subprocess.py", line 577, in run                                                         
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:     raise CalledProcessError(retcode, process.args,                                                                    
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]:                              output=stdout, stderr=stderr)                                                             
Dec 07 04:31:03 bodhi-backend01.iad2.fedoraproject.org celery-3[181529]: subprocess.CalledProcessError: Command '['dnf', '--disablerepo=*', '--repofrompath=testrepo,/mnt/koji/compose/updates/Fedora-Epel-8-updates-testing-20241207.1/compose/Everything/source/tree/', '--enablerepo=testrepo', '--setopt=skip_if_unavailable=0', '--setopt=testrepo.skip_if_unavailable=0', '--refresh', '--nogpgcheck', 'list', '--available']' returned non-zero exit status 1.   

This is with:
dnf5-5.2.7.0-1.fc41.x86_64

Switchking back to dnf-3 works.

[mattiaverga]

So, F40 worked with 5.1.x, F41 with 5.2 doesn't...
Neither unit tests and integration tests showed anything wrong here, but it happens on the real bodhi backend. Fun.

[nirik]

Oddly, it works when I manually run it too:

[root@bodhi-backend01 ~][PROD-IAD2]# dnf5 --disablerepo=* --repofrompath=testrepo,/mnt/koji/comp
ose/updates/Fedora-Epel-8-updates-testing-20241207.1/compose/Everything/source/tree/ --enablerep
o=testrepo --setopt=skip_if_unavailable=0 --setopt=testrepo.skip_if_unavailable=0 --refresh --no
gpgcheck list --available                                                                       
Updating and loading repositories:                                                              
 testrepo                                               100% | 691.8 KiB/s |   3.5 KiB |  00m00s
Repositories loaded.                            
Available packages
CGAL.src                        5.6.2-1.el8          testrepo
apptainer.src                   1.3.6-1.el8          testrepo
chromium.src                    131.0.6778.108-1.el8 testrepo
cucumber-messages.src           27.0.2-2.el8         testrepo
distribution-gpg-keys.src       1.106-1.el8          testrepo
dkms.src                        3.1.3-1.el8          testrepo
efifs.src                       1.11-1.el8           testrepo
fedora-license-data.src         1.63-1.el8           testrepo
google-benchmark.src            1.5.6-2.el8          testrepo
iaito.src                       5.9.9-2.el8          testrepo
imagej.src                      1.54m-1.el8          testrepo
lxi-tools.src                   2.8-1.el8            testrepo
memtester.src                   4.7.0-3.el8          testrepo
perl-OLE-Storage_Lite.src       0.20-1.el8           testrepo
radare2.src                     5.9.8-5.el8          testrepo
radsecproxy.src                 1.11.1-1.el8         testrepo
rpm-local-generator-support.src 1-6.el8              testrepo
tio.src                         3.8-1.el8            testrepo
xrootd.src                      1:5.7.2-1.el8        testrepo
zabbix6.0.src                   6.0.36-1.el8         testrepo
[root@bodhi-backend01 ~][PROD-IAD2]# echo $?
0
[root@bodhi-backend01 ~][PROD-IAD2]# dnf --disablerepo=* --repofrompath=testrepo,/mnt/koji/compo
se/updates/Fedora-Epel-8-updates-testing-20241207.1/compose/Everything/source/tree/ --enablerepo
=testrepo --setopt=skip_if_unavailable=0 --setopt=testrepo.skip_if_unavailable=0 --refresh --nog
pgcheck list --available                         
Added testrepo repo from /mnt/koji/compose/updates/Fedora-Epel-8-updates-testing-20241207.1/comp
ose/Everything/source/tree/
testrepo                                                        3.4 MB/s | 3.5 kB     00:00    
Available Packages
CGAL.src                                         5.6.2-1.el8                            testrepo
apptainer.src                                    1.3.6-1.el8                            testrepo
chromium.src                                     131.0.6778.108-1.el8                   testrepo
cucumber-messages.src                            27.0.2-2.el8                           testrepo
distribution-gpg-keys.src                        1.106-1.el8                            testrepo
dkms.src                                         3.1.3-1.el8                            testrepo
efifs.src                                        1.11-1.el8                             testrepo
fedora-license-data.src                          1.63-1.el8                             testrepo
google-benchmark.src                             1.5.6-2.el8                            testrepo
iaito.src                                        5.9.9-2.el8                            testrepo
imagej.src                                       1.54m-1.el8                            testrepo
lxi-tools.src                                    2.8-1.el8                              testrepo
memtester.src                                    4.7.0-3.el8                            testrepo
perl-OLE-Storage_Lite.src                        0.20-1.el8                             testrepo
radare2.src                                      5.9.8-5.el8                            testrepo
radsecproxy.src                                  1.11.1-1.el8                           testrepo
rpm-local-generator-support.src                  1-6.el8                                testrepo
tio.src                                          3.8-1.el8                              testrepo
xrootd.src                                       1:5.7.2-1.el8                          testrepo
zabbix6.0.src                                    6.0.36-1.el8                           testrepo
[root@bodhi-backend01 ~][PROD-IAD2]# echo $?
0

[mattiaverga]

It just came to my mind that on F40 dnf4 was still used by default, wasn't it? So the problem is not between dnf5 5.1 and 5.2.

The command syntax seems right, even if dnf5 could use different options (see #5821 ) it maintain backward compatibility. I've also tested that by cloning the above repository locally and ran the command both in my F41 workstation and in the bodhi development environment through bodhi-shell by using the bodhi.server.util.sanity_check_repodata_dnf method.

The only way I can reproduce a failure on that part of code is by enabling the relevant tests during a real Koji build (I cannot see it in a local mockbuild). There I get some more details:

stdout = 'Updating and loading repositories:\nFailed to download metadata (baseurl: "/tmp/tmp6wwj7ippbodhi/") for repository "testrepo"\n filesystem error: cannot create directories: Permission denied [/var/cache/yum/testrepo-494d39131e28f167]\n'

But I'm not sure it can be relevant.

[mattiaverga]

I have updated #5821 to catch the error thrown by subprocess and print it out, so, if you want, we can try to patch bodhi.server.util on bodhi-backend, reinstate dnf5 and fire a compose to better see what the error is.

[mattiaverga]

I've also set up #5822 which is a little more radical... instead of trying to load the repository by a dnf CLI call through subprocess, we can load it through libdnf5 python bindings.
All we need (what bodhi does currently) is to look if the repository loads without errors. I've also tested that the new way also fixes the failing tests under Koji builds.

But I'd like a review by someone with more knowledge on dnf python bindings on the method I wrote to load the repository. I have the feeling it is not quite right about the settings changed on the base config, despite I've followed the official tutorial.

In the end, I think the problem is that we run dnf5 in a subprocess started by a user without bash login. Using python bindings would solve that.

[nirik]

Yeah, that seems a fine approach. I am not sure who knows those bindings and can review tho.