Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement reposync plugin #1903

Merged
merged 4 commits into from
Dec 5, 2024
Merged

Implement reposync plugin #1903

merged 4 commits into from
Dec 5, 2024

Conversation

m-blaha
Copy link
Member

@m-blaha m-blaha commented Nov 26, 2024

Implements command dnf5 reposync for creating local copies of remote
repositories.
The syntax is mostly compatible with the dnf4 version of the command.

Resolves: #931

Comment on lines +170 to +172
:ref:`reposync <reposync_plugin_ref-label>`
| Synchronize packages and metadata of a remote DNF repository to a local directory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency we could also add a link to the See Also section at the bottom of this file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.


if (norepopath_option->get_value() && repos_query.size() > 1) {
throw libdnf5::cli::ArgumentParserConflictingArgumentsError(
M_("Can't use --norepopath with multiple repositories enabled"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs the --safe-write-path can also be used only with a single repository should we also check it here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed.

if (urls.empty()) {
std::cerr << libdnf5::utils::sformat(_("Failed to get mirror for package: \"{}\""), pkg.get_name())
<< std::endl;
continue;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The continue doesn't seem to be needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

if (!optional_metadata_option.get_value().contains(libdnf5::METADATA_TYPE_ALL)) {
optional_metadata_option.set(libdnf5::METADATA_TYPE_ALL);
}
repo.download_metadata(repo_path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly we download the same metadata for the second time here. All enabled repositories were expired in the configure step so we could just copy those and download only the missing parts.

I think this would be a nice optimization but I wouldn't block this PR on that.

void run() override;

private:
using download_list_type = std::vector<std::pair<std::filesystem::path, libdnf5::rpm::Package>>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be missing something but would std::map<std::filesystem::path, libdnf5::rpm::Package> be a better fit here?

The paths from the vector are converted to an unordered_set twice in the code, using the map we could avoid that.

Though the order would be different.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! The use of vector here is most likely the result of gradual development. Let me check whether the std::map would be a better fit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've replaced the std::vector with std::map, thanks again.

@kontura
Copy link
Contributor

kontura commented Dec 2, 2024

I have also noticed in the tests that there is an exit code change: https://github.com/rpm-software-management/ci-dnf-stack/pull/1599/files#diff-ea74dda041309ef66140b8a3856ddc3f3ace60e6e93733efcdc762410324fc39R390

Is this on purpose?
To be honest I think I would prefer the dnf4 behavior. The way I see it if any package from the repository failed to download the reposync command did fail.

@m-blaha
Copy link
Member Author

m-blaha commented Dec 4, 2024

I have also noticed in the tests that there is an exit code change: https://github.com/rpm-software-management/ci-dnf-stack/pull/1599/files#diff-ea74dda041309ef66140b8a3856ddc3f3ace60e6e93733efcdc762410324fc39R390

Is this on purpose? To be honest I think I would prefer the dnf4 behavior. The way I see it if any package from the repository failed to download the reposync command did fail.

Thanks for noticing! I did not take a note and forgot about it. The thing is, that currently PackageDownloader::download() does not indicate recoverable errors related to individual targets. It also looks like download() method does not honor retries config option. I'm thinking about creating a separate issue for this.

Edit: filed as #1926 and added TODO to the reposync code

Implements command `dnf5 reposync` for creating local copies of remote
repositories.
The syntax is mostly compatible with the dnf4 version of the command.
To unify the DNF behavior across different commands use the same option
to operate on source packages everywhere.
Using std::map<path, Package> instead of std::vector<std::pair<path, Package>>
eliminates the need to construct auxiliary sets.
Copy link
Contributor

@kontura kontura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@kontura kontura added this pull request to the merge queue Dec 5, 2024
Merged via the queue into main with commit fad4308 Dec 5, 2024
15 of 17 checks passed
@kontura kontura deleted the mblaha/reposync branch December 5, 2024 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DNF reposync Plugin
2 participants